cli: TestDebugCheckStore failed

[cockroach-teamcity]

cli.TestDebugCheckStore failed on master @ f117eea22dd7be380c7141cdf6cd7aba92dd9c70:

=== RUN   TestDebugCheckStore
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestDebugCheckStore1090274282
    test_log_scope.go:81: use -show-logs to present logs inline
    debug_check_store_test.go:90: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/cli/debug_check_store_test.go:90
        	Error:      	Received unexpected error:
        	            	check-store found problems
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/cli.init
        	            	  | 	github.com/cockroachdb/cockroach/pkg/cli/debug_check_store.go:55
        	            	  | runtime.doInit1
        	            	  | 	GOROOT/src/runtime/proc.go:7206
        	            	  | runtime.doInit
        	            	  | 	GOROOT/src/runtime/proc.go:7173
        	            	  | runtime.main
        	            	  | 	GOROOT/src/runtime/proc.go:253
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) check-store found problems
        	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
        	Test:       	TestDebugCheckStore
        	Messages:   	dir=/var/lib/engflow/worker/work/2/exec/_tmp/10a3dedc16573dd48c5adba69877df01/TestDebugCheckStore1258554494/s1
        	            	out=r18:/Table/1{5-6} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: stats inconsistency:
        	            	- stored:
        	            	{ContainsEstimates:0 LastUpdateNanos:1712989308216988618 LockAge:0 GCBytesAge:89851 LiveBytes:2897 LiveCount:45 KeyBytes:2601 KeyCount:55 ValBytes:2323 ValCount:90 IntentBytes:0 IntentCount:0 LockBytes:0 LockCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:1466 SysCount:8 AbortSpanBytes:68}
        	            	- recomputed:
        	            	{ContainsEstimates:0 LastUpdateNanos:1712989308216988618 LockAge:0 GCBytesAge:89851 LiveBytes:2897 LiveCount:45 KeyBytes:2601 KeyCount:55 ValBytes:2323 ValCount:90 IntentBytes:0 IntentCount:0 LockBytes:0 LockCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:1319 SysCount:8 AbortSpanBytes:68}
        	            	- diff:
        	            	SysBytes: 1466 != 1319
        	            	scanned 68 ranges (1 with estimates), total stats contains_estimates:0 last_update_nanos:1712989308687825374 lock_age:0 gc_bytes_age:1492250 live_bytes:1783393 live_count:9169 key_bytes:503798 key_count:9463 val_bytes:1314297 val_count:9854 intent_bytes:68 intent_count:1 lock_bytes:0 lock_count:1 range_key_count:0 range_key_bytes:0 range_val_count:0 range_val_bytes:0 sys_bytes:37944 sys_count:416 abort_span_bytes:68 
        	            	
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestDebugCheckStore1090274282
--- FAIL: TestDebugCheckStore (61.52s)

Parameters:

• attempt=1
• run=22
• shard=6

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-37803

[miraradeva]

This is the same issue as #112219, which we haven't gotten around to doing.

I'm removing the release-blocker label and assigning P3 to match the original issue.

[msbutler]

flaked again here https://mesolite.cluster.engflow.com/invocations/default/090b7489-3aae-44d1-be20-2b36a31149fb

[cockroach-teamcity]

cli.TestDebugCheckStore failed on master @ d8073b58467ddb4c5b2ebe4a485fc42fb98f7200:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestDebugCheckStore (14m57s)

Stack:

goroutine 10955309 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d

Log preceding fatal error

=== RUN   TestDebugCheckStore
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestDebugCheckStore3141802188
    test_log_scope.go:81: use -show-logs to present logs inline

Parameters:

• attempt=1
• run=8
• shard=6

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• cli: TestDebugCheckStore failed #123451 cli: TestDebugCheckStore failed [C-bug C-test-failure O-robot P-3 T-kv]

This test on roachdash | Improve this report!