skip SPACE_TYPE column for MariaDB >=10.5 #724

businessbean · 2023-04-15T18:47:31Z

this PR skips the SPACE_TYPE column for MariaDB server instances with version 10.5 or newer.

to be able to scrape the information_schema.innodb_sys_tablespaces tables
MariaDB has cleaned up the INFORMATION_SCHEMA.INNODB_ tables from version 10.5 on
unit tests added
fixes
- SPACE_TYPE Error while querying innodb_tablespaces #528
- MariaDB 10.6.* - 10.8.* - Unknown column 'SPACE_TYPE' #662

build environment

macOS 13.3.1 Intel
golang 1.20.2
make build

unit test results

go test -timeout 30s -run ^TestScrapeInfoSchemaInnodbTablespaces$ github.com/prometheus/mysqld_exporter/collector
ok  	github.com/prometheus/mysqld_exporter/collector	0.391s

go test -timeout 30s -run ^TestScrapeInfoSchemaInnodbTablespacesWithoutSpaceType$ github.com/prometheus/mysqld_exporter/collector
ok  	github.com/prometheus/mysqld_exporter/collector	0.202s

go test -timeout 30s -run ^TestSemanticVersionCheck$ github.com/prometheus/mysqld_exporter/collector
ok  	github.com/prometheus/mysqld_exporter/collector	0.174s

go test -timeout 30s github.com/prometheus/mysqld_exporter/collector -short
ok  	github.com/prometheus/mysqld_exporter/collector	0.468s

MariaDB test results

10.4.28

# TYPE mysql_info_schema_innodb_tablespace_allocated_size_bytes gauge
mysql_info_schema_innodb_tablespace_allocated_size_bytes{tablespace_name="innodb_temporary"} 1.2582912e+07
mysql_info_schema_innodb_tablespace_allocated_size_bytes{tablespace_name="mysql/gtid_slave_pos"} 65536
mysql_info_schema_innodb_tablespace_allocated_size_bytes{tablespace_name="mysql/innodb_index_stats"} 65536
mysql_info_schema_innodb_tablespace_allocated_size_bytes{tablespace_name="mysql/innodb_table_stats"} 65536
mysql_info_schema_innodb_tablespace_allocated_size_bytes{tablespace_name="mysql/transaction_registry"} 114688
mysql_info_schema_innodb_tablespace_allocated_size_bytes{tablespace_name="mysql/wsrep_cluster"} 65536
mysql_info_schema_innodb_tablespace_allocated_size_bytes{tablespace_name="mysql/wsrep_cluster_members"} 65536
mysql_info_schema_innodb_tablespace_allocated_size_bytes{tablespace_name="mysql/wsrep_streaming_log"} 65536
# HELP mysql_info_schema_innodb_tablespace_file_size_bytes The apparent size of the file, which represents the maximum size of the file, uncompressed.
# TYPE mysql_info_schema_innodb_tablespace_file_size_bytes gauge
mysql_info_schema_innodb_tablespace_file_size_bytes{tablespace_name="innodb_temporary"} 1.2582912e+07
mysql_info_schema_innodb_tablespace_file_size_bytes{tablespace_name="mysql/gtid_slave_pos"} 65536
mysql_info_schema_innodb_tablespace_file_size_bytes{tablespace_name="mysql/innodb_index_stats"} 65536
mysql_info_schema_innodb_tablespace_file_size_bytes{tablespace_name="mysql/innodb_table_stats"} 65536
mysql_info_schema_innodb_tablespace_file_size_bytes{tablespace_name="mysql/transaction_registry"} 114688
mysql_info_schema_innodb_tablespace_file_size_bytes{tablespace_name="mysql/wsrep_cluster"} 65536
mysql_info_schema_innodb_tablespace_file_size_bytes{tablespace_name="mysql/wsrep_cluster_members"} 65536
mysql_info_schema_innodb_tablespace_file_size_bytes{tablespace_name="mysql/wsrep_streaming_log"} 65536
# HELP mysql_info_schema_innodb_tablespace_space_info The Tablespace information and Space ID.
# TYPE mysql_info_schema_innodb_tablespace_space_info gauge
mysql_info_schema_innodb_tablespace_space_info{file_format="NONE",row_format="Dynamic",space_type="Single",tablespace_name="mysql/gtid_slave_pos"} 4
mysql_info_schema_innodb_tablespace_space_info{file_format="NONE",row_format="Dynamic",space_type="Single",tablespace_name="mysql/innodb_index_stats"} 2
mysql_info_schema_innodb_tablespace_space_info{file_format="NONE",row_format="Dynamic",space_type="Single",tablespace_name="mysql/innodb_table_stats"} 1
mysql_info_schema_innodb_tablespace_space_info{file_format="NONE",row_format="Dynamic",space_type="Single",tablespace_name="mysql/transaction_registry"} 3
mysql_info_schema_innodb_tablespace_space_info{file_format="NONE",row_format="Dynamic",space_type="Single",tablespace_name="mysql/wsrep_cluster"} 5
mysql_info_schema_innodb_tablespace_space_info{file_format="NONE",row_format="Dynamic",space_type="Single",tablespace_name="mysql/wsrep_cluster_members"} 6
mysql_info_schema_innodb_tablespace_space_info{file_format="NONE",row_format="Dynamic",space_type="Single",tablespace_name="mysql/wsrep_streaming_log"} 7
mysql_info_schema_innodb_tablespace_space_info{file_format="NONE",row_format="NONE",space_type="System",tablespace_name="innodb_temporary"} 4.294967294e+09

10.5.18

# TYPE mysql_info_schema_innodb_tablespace_allocated_size_bytes gauge
mysql_info_schema_innodb_tablespace_allocated_size_bytes{tablespace_name="innodb_temporary"} 1.2582912e+07
mysql_info_schema_innodb_tablespace_allocated_size_bytes{tablespace_name="mysql/gtid_slave_pos"} 65536
mysql_info_schema_innodb_tablespace_allocated_size_bytes{tablespace_name="mysql/innodb_index_stats"} 65536
mysql_info_schema_innodb_tablespace_allocated_size_bytes{tablespace_name="mysql/innodb_table_stats"} 65536
mysql_info_schema_innodb_tablespace_allocated_size_bytes{tablespace_name="mysql/transaction_registry"} 114688
mysql_info_schema_innodb_tablespace_allocated_size_bytes{tablespace_name="mysql/wsrep_cluster"} 65536
mysql_info_schema_innodb_tablespace_allocated_size_bytes{tablespace_name="mysql/wsrep_cluster_members"} 65536
mysql_info_schema_innodb_tablespace_allocated_size_bytes{tablespace_name="mysql/wsrep_streaming_log"} 65536
# HELP mysql_info_schema_innodb_tablespace_file_size_bytes The apparent size of the file, which represents the maximum size of the file, uncompressed.
# TYPE mysql_info_schema_innodb_tablespace_file_size_bytes gauge
mysql_info_schema_innodb_tablespace_file_size_bytes{tablespace_name="innodb_temporary"} 1.2582912e+07
mysql_info_schema_innodb_tablespace_file_size_bytes{tablespace_name="mysql/gtid_slave_pos"} 65536
mysql_info_schema_innodb_tablespace_file_size_bytes{tablespace_name="mysql/innodb_index_stats"} 65536
mysql_info_schema_innodb_tablespace_file_size_bytes{tablespace_name="mysql/innodb_table_stats"} 65536
mysql_info_schema_innodb_tablespace_file_size_bytes{tablespace_name="mysql/transaction_registry"} 114688
mysql_info_schema_innodb_tablespace_file_size_bytes{tablespace_name="mysql/wsrep_cluster"} 65536
mysql_info_schema_innodb_tablespace_file_size_bytes{tablespace_name="mysql/wsrep_cluster_members"} 65536
mysql_info_schema_innodb_tablespace_file_size_bytes{tablespace_name="mysql/wsrep_streaming_log"} 65536
# HELP mysql_info_schema_innodb_tablespace_space_info The Tablespace information and Space ID.
# TYPE mysql_info_schema_innodb_tablespace_space_info gauge
mysql_info_schema_innodb_tablespace_space_info{file_format="NONE",row_format="NONE",space_type="",tablespace_name="innodb_temporary"} 4.294967294e+09
mysql_info_schema_innodb_tablespace_space_info{file_format="NONE",row_format="NONE",space_type="",tablespace_name="mysql/gtid_slave_pos"} 4
mysql_info_schema_innodb_tablespace_space_info{file_format="NONE",row_format="NONE",space_type="",tablespace_name="mysql/innodb_index_stats"} 2
mysql_info_schema_innodb_tablespace_space_info{file_format="NONE",row_format="NONE",space_type="",tablespace_name="mysql/innodb_table_stats"} 1
mysql_info_schema_innodb_tablespace_space_info{file_format="NONE",row_format="NONE",space_type="",tablespace_name="mysql/transaction_registry"} 3
mysql_info_schema_innodb_tablespace_space_info{file_format="NONE",row_format="NONE",space_type="",tablespace_name="mysql/wsrep_cluster"} 5
mysql_info_schema_innodb_tablespace_space_info{file_format="NONE",row_format="NONE",space_type="",tablespace_name="mysql/wsrep_cluster_members"} 6
mysql_info_schema_innodb_tablespace_space_info{file_format="NONE",row_format="NONE",space_type="",tablespace_name="mysql/wsrep_streaming_log"} 7

- to be able to scrape information_schema.innodb_sys_tablespaces - https://jira.mariadb.org/browse/MDEV-19940 Signed-off-by: Birk Bohne <birk.bohne@sap.com>

- to avoid panic errors with invalid version numbers - unit test added Signed-off-by: Birk Bohne <birk.bohne@sap.com>

- to be able to validate code with and without SPACE_TYPE column - version query skipped if running inside test Signed-off-by: Birk Bohne <birk.bohne@sap.com>

firecow · 2023-07-15T18:45:55Z

@businessbean Can I do anything to help get this into main?
I'm getting annoyed filtering these logs in our Kibana 😄

businessbean · 2023-07-17T08:04:07Z

@businessbean Can I do anything to help get this into main? I'm getting annoyed filtering these logs in our Kibana 😄

Hi @firecow , you should filter these logs already on Logstash level 😄 I will update the PR today...

firecow · 2023-07-17T08:18:25Z

@businessbean Nice, thanks man..

SuperQ · 2023-07-17T08:25:10Z

collector/binlog.go

@@ -72,7 +72,7 @@ func (ScrapeBinlogSize) Version() float64 {
 }

 // Scrape collects data from database connection and sends it over channel as prometheus metric.
-func (ScrapeBinlogSize) Scrape(ctx context.Context, db *sql.DB, ch chan<- prometheus.Metric, logger log.Logger) error {
+func (ScrapeBinlogSize) Scrape(ctx context.Context, db *sql.DB, ch chan<- prometheus.Metric, logger log.Logger, semanticVersionIsNewer bool) error {


This semanticVersionIsNewer is not a good way to handle this. It's too specific to one collector, and hard-codes it as a flag for every collector.

What we probably want is to do something like we did in the postgres_expoter, which is to make the db instance include both a database connection and a semver that each collector can access.

Do you have a link to the related code?

Here's the PR: prometheus-community/postgres_exporter#785

I would prefer to do that refactoring as a separate PR.

I will look into this, but it will take some time...

Signed-off-by: Birk Bohne <birk.bohne@sap.com>

Pebin · 2024-03-13T09:48:53Z

is there any chance to merge it? 🙂 or is it waiting for the refactor? can I help with some testing, rebasing or coding? It's not exactly clear to me what blocked the merge 🙂

Pebin · 2024-06-06T09:46:02Z

hello @SuperQ are you still working on this repo? It would be useful for us to merge it. I can do some changes if needed. We are currently making custom build for this library because of this

businessbean added 2 commits April 15, 2023 20:56

skip SPACE_TYPE colums for MariaDB >=10.5

7681885

- to be able to scrape information_schema.innodb_sys_tablespaces - https://jira.mariadb.org/browse/MDEV-19940 Signed-off-by: Birk Bohne <birk.bohne@sap.com>

go-version instead of go-semver

07aa31d

- to avoid panic errors with invalid version numbers - unit test added Signed-off-by: Birk Bohne <birk.bohne@sap.com>

businessbean force-pushed the main branch from b062fca to 07aa31d Compare April 15, 2023 18:56

businessbean changed the title ~~skip SPACE_TYPE colum for MariaDB >=10.5~~ skip SPACE_TYPE column for MariaDB >=10.5 Apr 15, 2023

dedicated test for old and new table structures

057063a

- to be able to validate code with and without SPACE_TYPE column - version query skipped if running inside test Signed-off-by: Birk Bohne <birk.bohne@sap.com>

businessbean force-pushed the main branch from 245ca20 to 057063a Compare April 20, 2023 12:14

SuperQ reviewed Jul 17, 2023

View reviewed changes

businessbean added 2 commits July 17, 2023 12:06

Merge remote-tracking branch 'upstream/main'

3a0d12c

go-version mod info added

705b9dc

Signed-off-by: Birk Bohne <birk.bohne@sap.com>

businessbean force-pushed the main branch from ba68879 to 705b9dc Compare July 17, 2023 10:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

skip SPACE_TYPE column for MariaDB >=10.5 #724

skip SPACE_TYPE column for MariaDB >=10.5 #724

businessbean commented Apr 15, 2023 •

edited

firecow commented Jul 15, 2023 •

edited

businessbean commented Jul 17, 2023

firecow commented Jul 17, 2023

SuperQ Jul 17, 2023

businessbean Jul 17, 2023

SuperQ Jul 17, 2023

SuperQ Jul 17, 2023

businessbean Jul 17, 2023

Pebin commented Mar 13, 2024

Pebin commented Jun 6, 2024

skip SPACE_TYPE column for MariaDB >=10.5 #724

Are you sure you want to change the base?

skip SPACE_TYPE column for MariaDB >=10.5 #724

Conversation

businessbean commented Apr 15, 2023 • edited

build environment

unit test results

MariaDB test results

10.4.28

10.5.18

firecow commented Jul 15, 2023 • edited

businessbean commented Jul 17, 2023

firecow commented Jul 17, 2023

SuperQ Jul 17, 2023

Choose a reason for hiding this comment

businessbean Jul 17, 2023

Choose a reason for hiding this comment

SuperQ Jul 17, 2023

Choose a reason for hiding this comment

SuperQ Jul 17, 2023

Choose a reason for hiding this comment

businessbean Jul 17, 2023

Choose a reason for hiding this comment

Pebin commented Mar 13, 2024

Pebin commented Jun 6, 2024

businessbean commented Apr 15, 2023 •

edited

firecow commented Jul 15, 2023 •

edited