MySQL/MariaDB: Use strict SQL mode instead of just ANSI_QUOTES #624

lippserd · 2023-07-31T14:28:54Z

This PR

introduces Quoter that provides utility functions for quoting table names and columns, where the quotation mark depends on the database driver used
and replaces the ANSI_QUOTES SQL mode with TRADITIONAL which enables strict mode.

Previously, setting ANSI_QUOTES overrode the server defaults, which were most likely stricter. Also Vitess, which will be supported in the near future, does not support ANSI_QUOTES.

refs #606

Quoter provides utility functions for quoting table names and columns, where the quotation mark depends on the database driver used. This is in preparation for using strict SQL mode instead of just ANSI_QUOTES. Also Vitess, which will be supported in the near future, does not support ANSI_QUOTES.

julianbrost · 2023-07-31T15:16:59Z

pkg/config/database.go

+		// Set strict SQL mode, i.e. trigger an error if an incorrect value is inserted into a column.
+		config.Params = map[string]string{"sql_mode": "TRADITIONAL"}


What does this do differently compared to not setting sql_mode here?

I want to explicitly enable strict mode instead of letting something else decide it.

MariaDB's default SQL_MODE depends on the MariaDB version and, even with current MariaDB versions, does not equal TRADITIONAL.

MariaDB [icingadb]> SELECT @@SQL_MODE, @@GLOBAL.SQL_MODE\G *************************** 1. row *************************** @@SQL_MODE: STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION @@GLOBAL.SQL_MODE: STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION 1 row in set (0.000 sec) MariaDB [icingadb]> SET sql_mode = 'TRADITIONAL'; Query OK, 0 rows affected (0.001 sec) MariaDB [icingadb]> SELECT @@SQL_MODE, @@GLOBAL.SQL_MODE\G *************************** 1. row *************************** @@SQL_MODE: STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION @@GLOBAL.SQL_MODE: STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION 1 row in set (0.000 sec)

Furthermore, MySQL's default SQL_MODE is also something else.

Fortunately, both MySQL's as well as MariaDB's TRADITIONAL mode maps to the same mode list.
Thus, explicitly setting the SQL_MODE to TRADITIONAL sounds like a good idea.

julianbrost · 2023-07-31T15:22:23Z

pkg/icingadb/db.go

@@ -32,6 +32,7 @@ type DB struct {
 	logger            *logging.Logger
 	tableSemaphores   map[string]*semaphore.Weighted
 	tableSemaphoresMu sync.Mutex
+	quoter            *Quoter


I'm wondering: why hide this inside a member instead of having the functions directly available on DB (maybe by embedding the struct/pointer)? Would make the calls a bit nicer and the functions wouldn't look out of place there.

Didn't think of that tbh.

julianbrost · 2023-07-31T15:31:44Z

pkg/icingadb/quoter.go

+// QuoteColumnList quotes the given columns into a single comma concatenated string
+// so that they can be safely used as a column list for SELECT and INSERT statements.
+func (q *Quoter) QuoteColumnList(columns []string) string {
+	return fmt.Sprintf("%[1]s%s%[1]s", q.quoteCharacter, strings.Join(columns, q.quoteCharacter+", "+q.quoteCharacter))


Mixing numbered and unnumbered placeholders looks strange. At least I can't tell without double-checking which argument an unnumbered placeholder would refer to after some numbered ones.

So maybe at least go with %[1]s%[2]s%[1]s here.

But it's probably easier to not use printf here at all, q.quoteCharacter + strings.Join(columns, q.quoteCharacter+", "+q.quoteCharacter)) + q.quoteCharacter is shorter, even though it repeats the name quoteCharacter yet another time.

Sure, I can do that.

julianbrost · 2023-07-31T15:34:36Z

pkg/icingadb/quoter.go

Adding a few tests for this file should be easy by initializing with &Quoter{"`"}. Testing NewQuoter() would probably be annoying without a database connection.

Al2Klimov

Using MySQL quotes on both Vitess and, while on it, on MySQL is absolutely legit. For 1.2.0.

Al2Klimov · 2023-08-01T08:22:27Z

pkg/config/database.go

@@ -61,7 +61,8 @@ func (d *Database) Open(logger *logging.Logger) (*icingadb.DB, error) {

 		config.DBName = d.Database
 		config.Timeout = time.Minute
-		config.Params = map[string]string{"sql_mode": "ANSI_QUOTES"}
+		// Set strict SQL mode, i.e. trigger an error if an incorrect value is inserted into a column.
+		config.Params = map[string]string{"sql_mode": "TRADITIONAL"}


For 1.1.1 we should just set TRADITIONAL,ANSI_QUOTES here. This is a bugfix release. PoC:

MariaDB [idb]> SET SQL_MODE='TRADITIONAL'; Query OK, 0 rows affected (0,003 sec) MariaDB [idb]> SET SQL_MODE='TRADITIONAL,ANSI_QUOTES'; Query OK, 0 rows affected (0,003 sec) MariaDB [idb]> select @@SQL_MODE\G *************************** 1. row *************************** @@SQL_MODE: ANSI_QUOTES,STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION 1 row in set (0,003 sec) MariaDB [idb]> SET SQL_MODE='TRADITIONAL,ANSI_QUOTES,LOLCAT'; ERROR 1231 (42000): Variable 'sql_mode' can't be set to the value of 'LOLCAT' MariaDB [idb]>

julianbrost · 2023-08-01T13:25:32Z

I mean the missing of strict mode is not really bug in itself. Enabling it, at best, changes nothing, at worst, it exposes other bugs. In doubt, exposing would mean crashing the Icinga DB process. So better just do this as a whole in 1.2.0.

oxzi

I have successfully tested this PR and a rebased version on the main branch.

Should the SQL_MODE also be reflected in the schema.sql file?

Otherwise, there is nothing I would add to the already written comments.

oxzi · 2024-03-07T12:41:03Z

pkg/config/database.go

+		// Set strict SQL mode, i.e. trigger an error if an incorrect value is inserted into a column.
+		config.Params = map[string]string{"sql_mode": "TRADITIONAL"}


MariaDB's default SQL_MODE depends on the MariaDB version and, even with current MariaDB versions, does not equal TRADITIONAL.

MariaDB [icingadb]> SELECT @@SQL_MODE, @@GLOBAL.SQL_MODE\G *************************** 1. row *************************** @@SQL_MODE: STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION @@GLOBAL.SQL_MODE: STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION 1 row in set (0.000 sec) MariaDB [icingadb]> SET sql_mode = 'TRADITIONAL'; Query OK, 0 rows affected (0.001 sec) MariaDB [icingadb]> SELECT @@SQL_MODE, @@GLOBAL.SQL_MODE\G *************************** 1. row *************************** @@SQL_MODE: STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION @@GLOBAL.SQL_MODE: STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION 1 row in set (0.000 sec)

Furthermore, MySQL's default SQL_MODE is also something else.

Fortunately, both MySQL's as well as MariaDB's TRADITIONAL mode maps to the same mode list.
Thus, explicitly setting the SQL_MODE to TRADITIONAL sounds like a good idea.

lippserd · 2024-03-18T09:02:58Z

#699 sets TRADITIONAL in addition to ANSI_QUOTES to enable strict mode, while this PR eliminates the latter by introducing driver-specific quoting. I've removed this PR from the next release and will leave it open until we decide whether to support Vitess, which would be the only reason to change quoting.

lippserd added the area/connection label Jul 31, 2023

lippserd added this to the 1.1.1 milestone Jul 31, 2023

lippserd requested a review from julianbrost July 31, 2023 14:28

cla-bot bot added the cla/signed label Jul 31, 2023

MySQL/MariaDB: Use strict SQL mode instead of just ANSI_QUOTES

f050b7c

lippserd force-pushed the quoting branch from 9e7f109 to f050b7c Compare July 31, 2023 14:29

lippserd changed the title ~~MySQL: Use strict SQL mode instead of just ANSI_QUOTES~~ MySQL/MariaDB: Use strict SQL mode instead of just ANSI_QUOTES Jul 31, 2023

julianbrost reviewed Jul 31, 2023

View reviewed changes

Al2Klimov reviewed Aug 1, 2023

View reviewed changes

julianbrost modified the milestones: 1.1.1, 1.2.0 Aug 1, 2023

julianbrost requested a review from oxzi March 7, 2024 09:10

oxzi reviewed Mar 7, 2024

View reviewed changes

lippserd removed this from the 1.1.2 milestone Mar 18, 2024

lippserd mentioned this pull request Mar 18, 2024

MySQL/MariaDB: Use strict SQL mode #699

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MySQL/MariaDB: Use strict SQL mode instead of just ANSI_QUOTES #624

MySQL/MariaDB: Use strict SQL mode instead of just ANSI_QUOTES #624

lippserd commented Jul 31, 2023 •

edited

julianbrost Jul 31, 2023

lippserd Jul 31, 2023

oxzi Mar 7, 2024

julianbrost Jul 31, 2023

lippserd Jul 31, 2023

julianbrost Jul 31, 2023

lippserd Jul 31, 2023

julianbrost Jul 31, 2023

Al2Klimov left a comment

Al2Klimov Aug 1, 2023

julianbrost commented Aug 1, 2023

oxzi left a comment

oxzi Mar 7, 2024

lippserd commented Mar 18, 2024 •

edited

		// Set strict SQL mode, i.e. trigger an error if an incorrect value is inserted into a column.
		config.Params = map[string]string{"sql_mode": "TRADITIONAL"}

MySQL/MariaDB: Use strict SQL mode instead of just ANSI_QUOTES #624

Are you sure you want to change the base?

MySQL/MariaDB: Use strict SQL mode instead of just ANSI_QUOTES #624

Conversation

lippserd commented Jul 31, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Al2Klimov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

julianbrost commented Aug 1, 2023

oxzi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lippserd commented Mar 18, 2024 • edited

lippserd commented Jul 31, 2023 •

edited

lippserd commented Mar 18, 2024 •

edited