Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed #123452

Closed
cockroach-teamcity opened this issue May 2, 2024 · 3 comments
Closed

ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed #123452

cockroach-teamcity opened this issue May 2, 2024 · 3 comments
Assignees
@msbutler
Labels
branch-release-24.1 Used to mark GA and release blockers and technical advisories for 24.1 C-test-failure Broken test (automatically or manually discovered). GA-blocker O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Projects
Disaster Recovery Backlog
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 2, 2024 •
edited by cockroach-jira-scripts

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on release-24.1 @ 9400571541f006110b28dac7780e91dcec85140d:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingJobRetryReset (14m48s)

Stack:

goroutine 405875 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

* 
* goroutine 14261 [select]:
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSenderCircuitBreakers).probeStallLoop(0xc004e0c008, {0x7f32f98, 0xc00be58210})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_circuit_breaker.go:296 +0x186
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 14246
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
* goroutine 14312 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:243 +0xcb
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 14245
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:238 +0x58
* 
* goroutine 14344 [chan receive, 4 minutes]:
* github.com/cockroachdb/cmux.muxListener.Accept(...)
* 	github.com/cockroachdb/cmux/external/com_github_cockroachdb_cmux/cmux.go:184
* net/http.(*Server).Serve(0xc00b226a50, {0x7f1f1a0, 0xc0090bb590})
* 	GOROOT/src/net/http/server.go:3255 +0x33e
* github.com/cockroachdb/cockroach/pkg/server.startHTTPService.func3({0xc00f7bf008?, 0xc007eb0ff0?})
* 	github.com/cockroachdb/cockroach/pkg/server/server_http.go:333 +0x106
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 14246
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
* goroutine 14321 [select, 4 minutes]:
* github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x7f32f98, 0xc0079b5bc0})
* 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:246 +0xc5
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 14246
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
* goroutine 14260 [select]:
* github.com/cockroachdb/cockroach/pkg/ccl/utilccl.UpdateMetricOnLicenseChange.func2({0x7f32f98, 0xc00c0c5b30})
* 	github.com/cockroachdb/cockroach/pkg/ccl/utilccl/license_check.go:166 +0x12e
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 14246
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
* goroutine 14293 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:243 +0xcb
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 14246
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:238 +0x58
* 
*

Parameters:

  • attempt=1
  • run=6
  • shard=4
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-38358
@cockroach-teamcity cockroach-teamcity added branch-release-24.1 Used to mark GA and release blockers and technical advisories for 24.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery labels May 2, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone May 2, 2024
@cockroach-teamcity cockroach-teamcity mentioned this issue May 2, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed with artifacts on release-24.1 @ 0d6b6fe97ef174d392b89f6fac2a70cb0ad41837:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingJobRetryReset (14m44s)

Stack:

goroutine 291653 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x30c
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x38
Log preceding fatal error 

* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 26023
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c
* 
* goroutine 57272 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:243 +0x9c
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 13896
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:238 +0x64
* 
* goroutine 25951 [semacquire, 3 minutes]:
* sync.runtime_Semacquire(0x4013dc3ba8?)
* 	GOROOT/src/runtime/sema.go:62 +0x2c
* sync.(*WaitGroup).Wait(0x40099c8888)
* 	GOROOT/src/sync/waitgroup.go:116 +0x74
* golang.org/x/sync/errgroup.(*Group).Wait(0x40099c8880)
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:56 +0x2c
* github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0x40099c8880?, {0x7756f78?, 0x4011f1e730?}})
* 	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x48
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans(0x4010689808, {0x7756f78, 0x4011f1e6e0}, {0x40099c8800, 0x1, 0x1}, 0x4013f05860, {0x400e11bd80, 0x2, 0x40099c8760?})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:302 +0x4d0
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0x4010689808, {0x7756f78, 0x4011f1e6e0}, {0x400c7e32c0, 0x1, 0x4?}, {0x7756fb0?, 0xbf6538?}, 0x4013f05860, {0x400e11bd80, ...})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:197 +0x1c4
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x21224f0?, {0x7756f78?, 0x4011f1e6e0?}, {0x400c7e32c0?, 0x400ae27f18?, 0x1039270?}, {0x0?, 0xae27ed8?}, 0x0?, {0x400e11bd80?, ...})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:79 +0x30
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x7756f78?, 0x4011f1e6e0?})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:337 +0x68
* github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x2c
* golang.org/x/sync/errgroup.(*Group).Go.func1()
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x58
* created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 25950
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x98
* 
* goroutine 57273 [sync.Cond.Wait, 2 minutes]:
* sync.runtime_notifyListWait(0x40029800a8, 0xd05)
* 	GOROOT/src/runtime/sema.go:569 +0x154
* sync.(*Cond).Wait(0x4002980098)
* 	GOROOT/src/sync/cond.go:70 +0xcc
* github.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:369
* github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0x4002980000, {0x54dbc00?, 0xb668d80?})
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:643 +0x628
* runtime/pprof.Do({0x7756c18?, 0xb668d80?}, {{0x40014a5ac0?, 0x40172160c0?, 0x4016184f68?}}, 0x4016184fb8)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x80
* github.com/cockroachdb/pebble/record.NewLogWriter.func2()
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:562 +0x60
* created by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 13896
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:561 +0x44c
* 
*

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@msbutler msbutler self-assigned this May 2, 2024
@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on release-24.1 @ d46c47860a78e77a3cb46c422fda99b0f75da449:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingJobRetryReset (14m50s)

Stack:

goroutine 293094 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

* github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x7f375d8?, 0xc0077eaf60?})
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:452 +0x94
* runtime/pprof.Do({0x7f37288?, 0xc303a00?}, {{0xc000e37240?, 0x0?, 0x0?}}, 0xc0024577a8)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x9d
* github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:450 +0x52
* created by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 11260
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:442 +0xe5
* 
* goroutine 11296 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 11260
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f
* 
* goroutine 11289 [select]:
* github.com/cockroachdb/cockroach/pkg/gossip.newInfoStore.func1({0xc0049a6008?, 0x0?})
* 	github.com/cockroachdb/cockroach/pkg/gossip/pkg/gossip/infostore.go:192 +0x74
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 11260
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
* goroutine 11327 [chan receive, 2 minutes]:
* github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x7f375d8?, 0xc00a04a930?})
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:452 +0x94
* runtime/pprof.Do({0x7f37288?, 0xc303a00?}, {{0xc000e37240?, 0x0?, 0x0?}}, 0xc00268dfa8)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x9d
* github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:450 +0x52
* created by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 11260
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:442 +0xe5
* 
* goroutine 53106 [sync.Cond.Wait]:
* sync.runtime_notifyListWait(0xc002a080a8, 0x94)
* 	GOROOT/src/runtime/sema.go:569 +0x159
* sync.(*Cond).Wait(0xc1853ec1ac7132cd?)
* 	GOROOT/src/sync/cond.go:70 +0x85
* github.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:369
* github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0xc002a08000, {0x5cabea0?, 0xc303a00?})
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:643 +0x6fd
* runtime/pprof.Do({0x7f37288?, 0xc303a00?}, {{0xc000e36b40?, 0xc00f7c3680?, 0xc00b36d308?}}, 0xc0069a47c0)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x9d
* github.com/cockroachdb/pebble/record.NewLogWriter.func2()
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:562 +0x56
* created by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 11542
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:561 +0x4b8
* 
*

Parameters:

  • attempt=1
  • run=9
  • shard=4
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue May 3, 2024
Closed
@cockroach-teamcity cockroach-teamcity mentioned this issue May 6, 2024
Closed
@msbutler
Copy link
Collaborator

msbutler commented May 6, 2024

fixed by #123660

@msbutler msbutler closed this as completed May 6, 2024
Disaster Recovery Backlog automation moved this from Backlog to Done May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@msbutler msbutler

Labels
branch-release-24.1 Used to mark GA and release blockers and technical advisories for 24.1 C-test-failure Broken test (automatically or manually discovered). GA-blocker O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Projects
Disaster Recovery Backlog
Done
Milestone
24.1
Development

No branches or pull requests
2 participants
@cockroach-teamcity @msbutler