ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed

[cockroach-teamcity]

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ 379d332c9a4ce49cda4a3565a852e7ddb850ffb5:

Fatal error:

panic: test timed out after 53m17s
running tests:
	TestTenantStreamingJobRetryReset (53m0s)

Stack:

goroutine 224386 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x265
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x45

Log preceding fatal error

github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).startPersistingHLCUpperBound.func3({0xc003595008?, 0x0?})
	github.com/cockroachdb/cockroach/pkg/server/clock_monotonicity.go:271 +0xc6
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10085
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d

goroutine 22034 [chan send, 52 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.sendMetadata(...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:525
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans.func1({0xdcf3fe0, 0xc00cca2050})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:288 +0xcf4
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans.Group.GoCtx.func3()
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x47
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x92
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 22000
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x125

goroutine 10317 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvadmission.(*controllerImpl).SetTenantWeightProvider.func1()
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvadmission/kvadmission.go:512 +0x129
created by github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvadmission.(*controllerImpl).SetTenantWeightProvider in goroutine 10085
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvadmission/kvadmission.go:505 +0x125

goroutine 10042 [select, 52 minutes]:
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Tracer).runPeriodicSnapshotsLoop(0xc009b1a160, 0xc002fc8000, 0xc00d95d9e0, 0xc00c488720, 0x0)
	github.com/cockroachdb/cockroach/pkg/util/tracing/tracer_snapshots.go:284 +0x1b6
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Tracer).PeriodicSnapshotsLoop(0xc009b1a160, 0xc002fc8000, 0xc00c488720)
	github.com/cockroachdb/cockroach/pkg/util/tracing/tracer_snapshots.go:250 +0x105
github.com/cockroachdb/cockroach/pkg/server.newSQLServer.func1({0xc00338e808?, 0x0?})
	github.com/cockroachdb/cockroach/pkg/server/server_sql.go:554 +0xee
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10085
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d

goroutine 13871 [chan receive, 52 minutes]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeTCPServer.func1({0xc009cec808?, 0xc00b0b7c50?})
	github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:126 +0x65
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10085
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d

goroutine 10120 [select]:
github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x1cd
created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 10084
	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x8e

Parameters:

• attempt=1
• race=true
• run=3
• shard=5

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed #123540 ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed [C-test-failure O-robot T-disaster-recovery branch-release-24.1.0-rc release-blocker]
• ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed #123452 ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed [C-test-failure GA-blocker O-robot T-disaster-recovery branch-release-24.1 release-blocker]

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-38458