converting driver.Value type []uint8 to a int64: value out of range #9391

lionelee · 2023-07-17T05:32:10Z

What did you do?

start dm task

What did you expect to see?

task runs well

What did you see instead?

[lightning.go:465]["failed to runlightning"]
[code=34019:class=load-unit:scope=internal:level=high],
RawCause: fetch table columns failed: sql: Scan error on column index 3,
name \"NEXT_GLOBAL_ROW_ID\": converting driver.Value type []uint8 (\"10942694589135710587\") to a int64: value out of range
github.com/pingcap/tiflow/dm/pkg/terror.(*Error).Delegate
    github.com/pingcap/tiflow/dm/pkg/terror/terror.go:337
github.com/pingcap/tiflow/dm/loader.convertLightningError
    github.com/pingcap/tiflow/dm/loader/lightning.go:306
github.com/pingcap/tiflow/dm/loader.(*LightningLoader).runLightning
    github.com/pingcap/tiflow/dm/loader/lightning.go:287
github.com/pingcap/tiflow/dm/loader.(*LightningLoader).restore
    github.com/pingcap/tiflow/dm/loader/lightning.go:456
github.com/pingcap/tiflow/dm/loader.(*LightningLoader).Process
    github.com/pingcap/tiflow/dm/loader/lightning.go:524
runtime.goexit
    runtime/asm_amd64.s:1598

Versions of the cluster

DM version (run dmctl -V or dm-worker -V or dm-master -V):

v7.1.0

Upstream MySQL/MariaDB server version:

(paste upstream MySQL/MariaDB server version here)

Downstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

How did you deploy DM: tiup or manually?

(leave TiUP or manually here)

Other interesting information (system version, hardware config, etc):

>
>

current status of DM cluster (execute `query-status <task-name>` in dmctl)

(paste current status of DM cluster here)

The text was updated successfully, but these errors were encountered:

lionelee · 2023-07-17T05:35:44Z

It seems already been solved in version 1.6 of the MySQL driver

Support returning uint64 from Valuer in ConvertValue (go-sql-driver/mysql#1143)

lance6716 · 2023-07-18T03:41:27Z

Can you provide the full log? Seems it's a bug of FetchTableAutoIDInfos in lightning library.

lance6716 · 2023-07-18T03:43:56Z

And how to reproduce this problem? Seems 10942694589135710587 is illegal for NEXT_GLOBAL_ROW_ID because it should be int64 indeed

lionelee · 2023-07-18T07:44:44Z

Can you provide the full log? Seems it's a bug of FetchTableAutoIDInfos in lightning library.

[INFO] [lightning.go:491] ["lightning load start"] [task=task01] [unit=lightning-load]
[INFO] [util.go:190] ["put load worker in etcd"] [task=task01] [source=mysql01] [worker=serverless-svc-dm-worker-0]
[INFO] [checkpoint.go:123] ["initial checkpoint record"] [task=task01] [unit=lightning-load] [component="lightning checkpoint database list"] [task=task01] [source=mysql01]
[INFO] [checkpoint.go:143] ["update lightning loader status"] [task=task01] [unit=lightning-load] [component="lightning checkpoint database list"] [task=task01] [source=mysql01] [status=running]
[INFO] [info.go:49] ["Welcome to TiDB-Lightning"] [release-version=v7.1.1] [git-hash=None] [git-branch=None] [go-version=go1.20.5] [utc-build-time=None] [race-enabled=false]
[INFO] [lightning.go:425] [cfg] [task=task01] [unit=lightning-load] [cfg="{\"id\":1689663743031778450,\"lightning\":{\"table-concurrency\":16,\"index-concurrency\":16,\"region-concurrency\":16,\"io-concurrency\":5,\"check-requirements\":true,\"meta-schema-name\":\"\",\"max-error\":{\"type\":0,\"conflict\":9223372036854775807},\"task-info-schema-name\":\"lightning_task_info\"},\"tidb\":{\"status-port\":10080,\"pd-addr\":\"\",\"sql-mode\":\"NO_ENGINE_SUBSTITUTION,NO_AUTO_VALUE_ON_ZERO,ALLOW_INVALID_DATES,IGNORE_SPACE\",\"tls\":\"false\",\"security\":{\"ca-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\",\"redact-info-log\":false},\"max-allowed-packet\":67108864,\"distsql-scan-concurrency\":15,\"build-stats-concurrency\":20,\"index-serial-scan-concurrency\":20,\"checksum-table-concurrency\":2,\"vars\":{\"foreign_key_checks\":\"0\",\"tidb_txn_mode\":\"optimistic\",\"time_zone\":\"+00:00\"}},\"checkpoint\":{\"schema\":\"tidb_lightning_checkpoint\",\"driver\":\"file\",\"enable\":true,\"keep-after-success\":\"remove\"},\"mydumper\":{\"read-block-size\":65536,\"batch-size\":0,\"batch-import-ratio\":0,\"source-id\":\"mysql01\",\"data-source-dir\":\"file:///dumped_data.task01\",\"character-set\":\"auto\",\"csv\":{\"separator\":\",\",\"delimiter\":\"\\\"\",\"terminator\":\"\",\"null\":[\"\\\\N\"],\"header\":true,\"header-schema-match\":true,\"trim-last-separator\":false,\"not-null\":false,\"backslash-escape\":true,\"escaped-by\":\"\\\\\"},\"max-region-size\":268435456,\"filter\":[\"*.*\",\"!mysql.*\",\"!sys.*\",\"!INFORMATION_SCHEMA.*\",\"!PERFORMANCE_SCHEMA.*\",\"!METRICS_SCHEMA.*\",\"!INSPECTION_SCHEMA.*\"],\"files\":null,\"no-schema\":false,\"case-sensitive\":false,\"strict-format\":false,\"default-file-rules\":true,\"ignore-data-columns\":null,\"data-character-set\":\"binary\",\"data-invalid-char-replace\":\"�\"},\"tikv-importer\":{\"addr\":\"\",\"backend\":\"tidb\",\"on-duplicate\":\"replace\",\"max-kv-pairs\":4096,\"send-kv-pairs\":32768,\"compress-kv-pairs\":\"\",\"region-split-size\":0,\"region-split-keys\":0,\"sorted-kv-dir\":\"\",\"disk-quota\":9223372036854775807,\"range-concurrency\":0,\"duplicate-resolution\":\"none\",\"incremental-import\":true,\"keyspace-name\":\"\",\"add-index-by-sql\":false,\"engine-mem-cache-size\":536870912,\"local-writer-mem-cache-size\":134217728,\"store-write-bwlimit\":0},\"post-restore\":{\"checksum\":\"off\",\"analyze\":\"off\",\"level-1-compact\":false,\"post-process-at-last\":true,\"compact\":false},\"cron\":{\"switch-mode\":\"5m0s\",\"log-progress\":\"5m0s\",\"check-disk-quota\":\"1m0s\"},\"routes\":[],\"security\":{\"ca-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\",\"redact-info-log\":false},\"black-white-list\":{\"do-tables\":null,\"do-dbs\":null,\"ignore-tables\":null,\"ignore-dbs\":null}}"]
[INFO] [lightning.go:425] [cfg] [task=task01] [unit=lightning-load] [cfg="{\"id\":1689663743031778450,\"lightning\":{\"table-concurrency\":16,\"index-concurrency\":16,\"region-concurrency\":16,\"io-concurrency\":5,\"check-requirements\":true,\"meta-schema-name\":\"\",\"max-error\":{\"type\":0,\"conflict\":9223372036854775807},\"task-info-schema-name\":\"lightning_task_info\"},\"tidb\":{\"status-port\":10080,\"pd-addr\":\"\",\"sql-mode\":\"NO_ENGINE_SUBSTITUTION,NO_AUTO_VALUE_ON_ZERO,ALLOW_INVALID_DATES,IGNORE_SPACE\",\"tls\":\"false\",\"security\":{\"ca-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\",\"redact-info-log\":false},\"max-allowed-packet\":67108864,\"distsql-scan-concurrency\":15,\"build-stats-concurrency\":20,\"index-serial-scan-concurrency\":20,\"checksum-table-concurrency\":2,\"vars\":{\"foreign_key_checks\":\"0\",\"tidb_txn_mode\":\"optimistic\",\"time_zone\":\"+00:00\"}},\"checkpoint\":{\"schema\":\"tidb_lightning_checkpoint\",\"driver\":\"file\",\"enable\":true,\"keep-after-success\":\"remove\"},\"mydumper\":{\"read-block-size\":65536,\"batch-size\":0,\"batch-import-ratio\":0,\"source-id\":\"mysql01\",\"data-source-dir\":\"file:///dumped_data.task01\",\"character-set\":\"auto\",\"csv\":{\"separator\":\",\",\"delimiter\":\"\\\"\",\"terminator\":\"\",\"null\":[\"\\\\N\"],\"header\":true,\"header-schema-match\":true,\"trim-last-separator\":false,\"not-null\":false,\"backslash-escape\":true,\"escaped-by\":\"\\\\\"},\"max-region-size\":268435456,\"filter\":[\"*.*\",\"!mysql.*\",\"!sys.*\",\"!INFORMATION_SCHEMA.*\",\"!PERFORMANCE_SCHEMA.*\",\"!METRICS_SCHEMA.*\",\"!INSPECTION_SCHEMA.*\"],\"files\":null,\"no-schema\":false,\"case-sensitive\":false,\"strict-format\":false,\"default-file-rules\":true,\"ignore-data-columns\":null,\"data-character-set\":\"binary\",\"data-invalid-char-replace\":\"�\"},\"tikv-importer\":{\"addr\":\"\",\"backend\":\"tidb\",\"on-duplicate\":\"replace\",\"max-kv-pairs\":4096,\"send-kv-pairs\":32768,\"compress-kv-pairs\":\"\",\"region-split-size\":0,\"region-split-keys\":0,\"sorted-kv-dir\":\"\",\"disk-quota\":9223372036854775807,\"range-concurrency\":0,\"duplicate-resolution\":\"none\",\"incremental-import\":true,\"keyspace-name\":\"\",\"add-index-by-sql\":false,\"engine-mem-cache-size\":536870912,\"local-writer-mem-cache-size\":134217728,\"store-write-bwlimit\":0},\"post-restore\":{\"checksum\":\"off\",\"analyze\":\"off\",\"level-1-compact\":false,\"post-process-at-last\":true,\"compact\":false},\"cron\":{\"switch-mode\":\"5m0s\",\"log-progress\":\"5m0s\",\"check-disk-quota\":\"1m0s\"},\"routes\":[],\"security\":{\"ca-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\",\"redact-info-log\":false},\"black-white-list\":{\"do-tables\":null,\"do-dbs\":null,\"ignore-tables\":null,\"ignore-dbs\":null}}"]
[INFO] [lightning.go:519] ["load data source start"] [task=task01] [unit=lightning-load]
[INFO] [loader.go:465] ["[loader] file is filtered by file router"] [task=task01] [unit=lightning-load] [path=metadata]
[INFO] [lightning.go:522] ["load data source completed"] [task=task01] [unit=lightning-load] [takeTime=809.589µs] []
[WARN] [tidb.go:92] ["set session variable failed, will skip this query"] [task=task01] [unit=lightning-load] [query="SET SESSION tidb_opt_write_row_id = '1';"] [error="Error 1227 (42000): Access denied; you need (at least one of) the RESTRICTED_VARIABLES_ADMIN privilege(s) for this operation"]
[INFO] [checkpoints.go:1113] ["open checkpoint file failed, going to create a new one"] [task=task01] [unit=lightning-load] [path=./dumped_data.task01/tidb_lightning_checkpoint.pb] []
[INFO] [import.go:486] ["the whole procedure start"] [task=task01] [unit=lightning-load]
[INFO] [import.go:792] ["restore all schema start"] [task=task01] [unit=lightning-load]
[WARN] [util.go:196] ["run create schema job failed with no retry"] [task=task01] [unit=lightning-load] [query="CREATE DATABASE IF NOT EXISTS `nightly` CHARACTER SET = utf8mb4 COLLATE = utf8mb4_0900_ai_ci ENCRYPTION = 'N';"] [args=null] [retryCnt=0] [error="Error 1273 (HY000): Unsupported collation when new collation is enabled: 'utf8mb4_0900_ai_ci'"]
[INFO] [version.go:429] ["detect server version"] [type=TiDB] [version=6.6.0-serverless]
[WARN] [util.go:196] ["fetch table columns failed with no retry"] [task=task01] [unit=lightning-load] [retryCnt=0] [error="sql: Scan error on column index 3, name \"NEXT_GLOBAL_ROW_ID\": converting driver.Value type []uint8 (\"10942694589135710587\") to a int64: value out of range"]
[INFO] [import.go:811] ["restore all schema completed"] [task=task01] [unit=lightning-load] [takeTime=1.170356567s] []
[INFO] [version.go:429] ["detect server version"] [type=TiDB] [version=6.6.0-serverless]
[WARN] [util.go:196] ["fetch table columns failed with no retry"] [task=task01] [unit=lightning-load] [retryCnt=0] [error="sql: Scan error on column index 3, name \"NEXT_GLOBAL_ROW_ID\": converting driver.Value type []uint8 (\"10942694589135710587\") to a int64: value out of range"]
[ERROR] [import.go:504] ["run failed"] [task=task01] [unit=lightning-load] [step=1] [error="fetch table columns failed: sql: Scan error on column index 3, name \"NEXT_GLOBAL_ROW_ID\": converting driver.Value type []uint8 (\"10942694589135710587\") to a int64: value out of range"]
[ERROR] [import.go:514] ["the whole procedure failed"] [task=task01] [unit=lightning-load] [takeTime=1.706797063s] [error="fetch table columns failed: sql: Scan error on column index 3, name \"NEXT_GLOBAL_ROW_ID\": converting driver.Value type []uint8 (\"10942694589135710587\") to a int64: value out of range"]
[ERROR] [lightning.go:465] ["failed to runlightning"] [task=task01] [unit=lightning-load] [error="[code=34019:class=load-unit:scope=internal:level=high], RawCause: fetch table columns failed: sql: Scan error on column index 3, name \"NEXT_GLOBAL_ROW_ID\": converting driver.Value type []uint8 (\"10942694589135710587\") to a int64: value out of range"] [errorVerbose="[code=34019:class=load-unit:scope=internal:level=high], RawCause: fetch table columns failed: sql: Scan error on column index 3, name \"NEXT_GLOBAL_ROW_ID\": converting driver.Value type []uint8 (\"10942694589135710587\") to a int64: value out of range\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:337\ngithub.com/pingcap/tiflow/dm/loader.convertLightningError\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:306\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).runLightning\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:287\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).restore\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:456\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).Process\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:524\nruntime.goexit\n\truntime/asm_amd64.s:1598"]
[ERROR] [lightning.go:525] ["process error"] [task=task01] [unit=lightning-load] [error="[code=34019:class=load-unit:scope=internal:level=high], RawCause: fetch table columns failed: sql: Scan error on column index 3, name \"NEXT_GLOBAL_ROW_ID\": converting driver.Value type []uint8 (\"10942694589135710587\") to a int64: value out of range"] [errorVerbose="[code=34019:class=load-unit:scope=internal:level=high], RawCause: fetch table columns failed: sql: Scan error on column index 3, name \"NEXT_GLOBAL_ROW_ID\": converting driver.Value type []uint8 (\"10942694589135710587\") to a int64: value out of range\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:337\ngithub.com/pingcap/tiflow/dm/loader.convertLightningError\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:306\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).runLightning\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:287\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).restore\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:456\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).Process\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:524\nruntime.goexit\n\truntime/asm_amd64.s:1598"]
[INFO] [lightning.go:612] ["progress status of lightning"] [task=task01] [unit=lightning-load] [finished_bytes=0] [total_bytes=0] [progress="0.00 %"] ["current speed (bytes / seconds)"=0]
[INFO] [lightning.go:537] ["lightning load end"] [task=task01] [unit=lightning-load] [IsCanceled=false] [finished_bytes=0] [total_bytes=0] [progress="0.00 %"]
[INFO] [lightning.go:612] ["progress status of lightning"] [task=task01] [unit=lightning-load] [finished_bytes=0] [total_bytes=0] [progress="0.00 %"] ["current speed (bytes / seconds)"=0]
[INFO] [subtask.go:333] ["unit process returned"] [subtask=task01] [unit=Load] [stage=Paused] [status="{\"progress\":\"0.00 %\",\"metaBinlog\":\"(mysql-bin-changelog.012917, 1785)\"}"]
[ERROR] [subtask.go:354] ["unit process error"] [subtask=task01] [unit=Load] ["error information"="ErrCode:34019 ErrClass:\"load-unit\" ErrScope:\"internal\" ErrLevel:\"high\" RawCause:\"fetch table columns failed: sql: Scan error on column index 3, name \\\"NEXT_GLOBAL_ROW_ID\\\": converting driver.Value type []uint8 (\\\"10942694589135710587\\\") to a int64: value out of range\" "]

lionelee · 2023-07-18T07:55:54Z

And how to reproduce this problem? Seems 10942694589135710587 is illegal for NEXT_GLOBAL_ROW_ID because it should be int64 indeed

steps to reproduce:

create table in mysql:

CREATE TABLE IF NOT EXISTS `test`.`dummy` (
  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(255) NOT NULL,
  PRIMARY KEY (`id`)
);

insert some value:

INSERT INTO `test`.`dummy` values (10942694589135710585, "test0");
INSERT INTO `test`.`dummy` values (10942694589135710586, "test1");

create source in dm

source-id: "mysql01"
from:
  host: "localhost"
  port: 3306
  user: "root"
  password: "xxxxxx"

start task in dm

name: task01
task-mode: all
ignore-checking-items: ["table_schema"]
mysql-instances:
  - source-id: "mysql01"
    block-allow-list: "rule-1"
target-database:
  host: "xxxxxx"
  port: 4000
  user: "xxxxxx"
  password: "xxxxxx"
  security:
    ssl-ca: "/etc/ssl/cert.pem"
block-allow-list:
  rule-1:
    do-dbs: ["test"]

then stop the task and drop db dm_meta
restart the task and get the error

lance6716 · 2023-07-18T07:59:57Z

Thanks, the problem is INSERT INTO `test`.`dummy` values (10942694589135710585, "test0"); I'll open an issue for lightning.

lance6716 · 2023-11-08T05:58:32Z

closed by pingcap/tidb#45614

lionelee added area/dm Issues or PRs related to DM. type/bug This is a bug. labels Jul 17, 2023

github-actions bot added this to Need Triage in Question and Bug Reports Jul 17, 2023

jebter added the severity/major This is a major bug. label Jul 17, 2023

ti-chi-bot bot added may-affects-5.2 may-affects-5.3 may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 labels Jul 17, 2023

lance6716 mentioned this issue Jul 18, 2023

lightning wrongly use int64 to save NEXT_GLOBAL_ROW_ID pingcap/tidb#45427

Closed

lyzx2001 mentioned this issue Jul 27, 2023

lightning: save NEXT_GLOBAL_ROW_ID as uint64 instead of int64 pingcap/tidb#45614

Merged

12 tasks

ti-chi-bot added the affects-7.5 label Oct 23, 2023

lance6716 closed this as completed Nov 8, 2023

Question and Bug Reports automation moved this from Need Triage to Done Nov 8, 2023

lance6716 removed the affects-7.5 label Nov 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

converting driver.Value type []uint8 to a int64: value out of range #9391

converting driver.Value type []uint8 to a int64: value out of range #9391

lionelee commented Jul 17, 2023

lionelee commented Jul 17, 2023 •

edited

lance6716 commented Jul 18, 2023

lance6716 commented Jul 18, 2023

lionelee commented Jul 18, 2023 •

edited

lionelee commented Jul 18, 2023

lance6716 commented Jul 18, 2023 •

edited

lance6716 commented Nov 8, 2023

converting driver.Value type []uint8 to a int64: value out of range #9391

converting driver.Value type []uint8 to a int64: value out of range #9391

Comments

lionelee commented Jul 17, 2023

What did you do?

What did you expect to see?

What did you see instead?

Versions of the cluster

current status of DM cluster (execute query-status <task-name> in dmctl)

lionelee commented Jul 17, 2023 • edited

lance6716 commented Jul 18, 2023

lance6716 commented Jul 18, 2023

lionelee commented Jul 18, 2023 • edited

lionelee commented Jul 18, 2023

lance6716 commented Jul 18, 2023 • edited

lance6716 commented Nov 8, 2023

current status of DM cluster (execute `query-status <task-name>` in dmctl)

lionelee commented Jul 17, 2023 •

edited

lionelee commented Jul 18, 2023 •

edited

lance6716 commented Jul 18, 2023 •

edited