We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
We are facing error due toconnection reset by peer with TiDB v6.5.3/BR v6.5.2
connection reset by peer
The error was resolved with 3rd attempt. we can see number of logs to be deleted were decreasing each time, so assuming there can be some limitation.
[2023/06/22 06:05:34.262 +00:00] [INFO] [info.go:49] ["Welcome to Backup & Restore (BR)"] [release-version=v6.5.2] [git-hash=29116c0256c52b224da2b34d712c1063d171c0ad] [git-branch=heads/refs/tags/v6.5.2] [go-version=go1.19.8] [utc-build-time="2023-04-19 10:44:07"] [race-enabled=false] .............. [2023/06/22 06:06:01.817 +00:00] [ERROR] [stream.go:513] ["failed to stream"] [command="log truncate"] [error="failed to write back file xxxxx/xxxxxx/.meta: failed to remove the empty meta xxxxx/xxxxxx/xxxx.meta: RequestError: send request failed\ncaused by: Delete \"https://bucketxxxxx.meta\": read tcp xxx.xxx.xxx.xxx:36116-> xxx.xxx.xx:443: read: connection reset by peer"] [errorVerbose="RequestError: send request failed\ncaused by: Delete \"https:/xxxxxx.meta\": read tcp xx.xxx.xxx.xxx:36116->xx.xxx.xxx.xxx:443: read: connection reset by peer\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/juju_adaptor.go:15\ngithub.com/pingcap/tidb/br/pkg/storage.(*S3Storage).DeleteFile\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:533\ngithub.com/pingcap/tidb/br/pkg/restore.(*StreamMetadataSet).doWriteBackForFile\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/stream_metas.go:266\ngithub.com/pingcap/tidb/br/pkg/restore.(*StreamMetadataSet).removeDataFilesAndUpdateMetadata\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/stream_metas.go:254\ngithub.com/pingcap/tidb/br/pkg/restore.(*StreamMetadataSet).RemoveDataFilesAndUpdateMetadataInBatch.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/stream_metas.go:184\ngithub.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\nfailed to remove the empty meta v1/backupmeta/442040825730301953-dc9eecc3-b9be-4da0-8501-5f5c688ad6f9.meta\nfailed to write back file v1/backupmeta/442040825730301953-dc9eecc3-b9be-4da0-8501-5f5c688ad6f9.meta"] [stack="github.com/pingcap/tidb/br/pkg/task.RunStreamCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/stream.go:513\nmain.streamCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/stream.go:231\nmain.newStreamTruncateCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/stream.go:143\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
Seems like BR's retry mechanism can't cover this case.
The text was updated successfully, but these errors were encountered:
This repo is deprecated, and br support is now part of tidb repo in https://github.com/pingcap/tidb/tree/master/br. please open the issue in tidb repo. thx
Sorry, something went wrong.
No branches or pull requests
We are facing error due to
connection reset by peer
with TiDB v6.5.3/BR v6.5.2The error was resolved with 3rd attempt. we can see number of logs to be deleted were decreasing each time, so assuming there can be some limitation.
[2023/06/22 06:05:34.262 +00:00] [INFO] [info.go:49] ["Welcome to Backup & Restore (BR)"] [release-version=v6.5.2] [git-hash=29116c0256c52b224da2b34d712c1063d171c0ad] [git-branch=heads/refs/tags/v6.5.2] [go-version=go1.19.8] [utc-build-time="2023-04-19 10:44:07"] [race-enabled=false] .............. [2023/06/22 06:06:01.817 +00:00] [ERROR] [stream.go:513] ["failed to stream"] [command="log truncate"] [error="failed to write back file xxxxx/xxxxxx/.meta: failed to remove the empty meta xxxxx/xxxxxx/xxxx.meta: RequestError: send request failed\ncaused by: Delete \"https://bucketxxxxx.meta\": read tcp xxx.xxx.xxx.xxx:36116-> xxx.xxx.xx:443: read: connection reset by peer"] [errorVerbose="RequestError: send request failed\ncaused by: Delete \"https:/xxxxxx.meta\": read tcp xx.xxx.xxx.xxx:36116->xx.xxx.xxx.xxx:443: read: connection reset by peer\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/juju_adaptor.go:15\ngithub.com/pingcap/tidb/br/pkg/storage.(*S3Storage).DeleteFile\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:533\ngithub.com/pingcap/tidb/br/pkg/restore.(*StreamMetadataSet).doWriteBackForFile\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/stream_metas.go:266\ngithub.com/pingcap/tidb/br/pkg/restore.(*StreamMetadataSet).removeDataFilesAndUpdateMetadata\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/stream_metas.go:254\ngithub.com/pingcap/tidb/br/pkg/restore.(*StreamMetadataSet).RemoveDataFilesAndUpdateMetadataInBatch.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/stream_metas.go:184\ngithub.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\nfailed to remove the empty meta v1/backupmeta/442040825730301953-dc9eecc3-b9be-4da0-8501-5f5c688ad6f9.meta\nfailed to write back file v1/backupmeta/442040825730301953-dc9eecc3-b9be-4da0-8501-5f5c688ad6f9.meta"] [stack="github.com/pingcap/tidb/br/pkg/task.RunStreamCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/stream.go:513\nmain.streamCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/stream.go:231\nmain.newStreamTruncateCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/stream.go:143\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
Seems like BR's retry mechanism can't cover this case.
The text was updated successfully, but these errors were encountered: