cli: TestDebugCheckStore failed

[cockroach-teamcity]

cli.TestDebugCheckStore failed on release-24.1 @ 9400571541f006110b28dac7780e91dcec85140d:

E240502 06:25:26.645703 3299576 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r32/1:/{Table/29-NamespaceTab…}] 1787  [n1,s1,r32/1:/{Table/29-NamespaceTab…}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r32:/{Table/29-NamespaceTable/30} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630873.745492719,0 (2024-05-02 06:21:13); raft status: {"id":"1","term":10,"vote":"1","commit":42,"lead":"0","raftState":"StatePreCandidate","applied":42,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/29]
E240502 06:25:26.646083 3299577 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r17/1:/Table/1{4-5}] 1788  [n1,s1,r17/1:/Table/1{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r17:/Table/1{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630857.424347936,0 (2024-05-02 06:20:57); raft status: {"id":"1","term":9,"vote":"1","commit":36,"lead":"0","raftState":"StatePreCandidate","applied":36,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/14]
E240502 06:25:26.646522 3299588 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r52/1:/Table/5{0-1}] 1789  [n1,s1,r52/1:/Table/5{0-1}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r52:/Table/5{0-1} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630868.464415262,0 (2024-05-02 06:21:08); raft status: {"id":"1","term":8,"vote":"1","commit":41,"lead":"0","raftState":"StatePreCandidate","applied":41,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Table/50]
E240502 06:25:26.646900 3299589 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r24/1:/Table/2{1-2}] 1790  [n1,s1,r24/1:/Table/2{1-2}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r24:/Table/2{1-2} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630875.315373110,0 (2024-05-02 06:21:15); raft status: {"id":"1","term":7,"vote":"0","commit":46,"lead":"0","raftState":"StatePreCandidate","applied":46,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/21]
E240502 06:25:26.647255 3299590 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r56/1:/Table/5{4-5}] 1791  [n1,s1,r56/1:/Table/5{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r56:/Table/5{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630857.424347936,0 (2024-05-02 06:20:57); raft status: {"id":"1","term":9,"vote":"1","commit":108,"lead":"0","raftState":"StatePreCandidate","applied":108,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/54]
E240502 06:25:26.647867 3299601 kv/kvserver/queue.go:1202  [T1,Vsystem,n3,replicate,s3,r56/2:/Table/5{4-5}] 1792  [n3,s3,r56/2:/Table/5{4-5}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r56:/Table/5{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630857.424347936,0 (2024-05-02 06:20:57); raft status: {"id":"2","term":9,"vote":"1","commit":108,"lead":"0","raftState":"StatePreCandidate","applied":108,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/54]
E240502 06:25:26.648723 3299612 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r52/1:/Table/5{0-1}] 1793  [n1,s1,r52/1:/Table/5{0-1}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r52:/Table/5{0-1} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630868.464415262,0 (2024-05-02 06:21:08); raft status: {"id":"1","term":8,"vote":"1","commit":41,"lead":"0","raftState":"StatePreCandidate","applied":41,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Table/50]
E240502 06:25:26.649113 3299613 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r24/1:/Table/2{1-2}] 1794  [n1,s1,r24/1:/Table/2{1-2}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r24:/Table/2{1-2} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630875.315373110,0 (2024-05-02 06:21:15); raft status: {"id":"1","term":7,"vote":"0","commit":46,"lead":"0","raftState":"StatePreCandidate","applied":46,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/21]
E240502 06:25:26.649506 3299614 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r56/1:/Table/5{4-5}] 1795  [n1,s1,r56/1:/Table/5{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r56:/Table/5{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630857.424347936,0 (2024-05-02 06:20:57); raft status: {"id":"1","term":9,"vote":"1","commit":108,"lead":"0","raftState":"StatePreCandidate","applied":108,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/54]
E240502 06:25:26.650216 3299635 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r28/1:/Table/2{5-6}] 1796  [n1,s1,r28/1:/Table/2{5-6}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r28:/Table/2{5-6} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630869.076533290,0 (2024-05-02 06:21:09); raft status: {"id":"1","term":11,"vote":"1","commit":41,"lead":"0","raftState":"StatePreCandidate","applied":41,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/25]
E240502 06:25:26.650617 3299636 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r48/1:/Table/4{5-6}] 1797  [n1,s1,r48/1:/Table/4{5-6}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r48:/Table/4{5-6} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630855.425441632,0 (2024-05-02 06:20:55); raft status: {"id":"1","term":11,"vote":"1","commit":40,"lead":"0","raftState":"StatePreCandidate","applied":40,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/45]
E240502 06:25:26.651489 3299647 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r28/1:/Table/2{5-6}] 1798  [n1,s1,r28/1:/Table/2{5-6}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r28:/Table/2{5-6} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630869.076533290,0 (2024-05-02 06:21:09); raft status: {"id":"1","term":11,"vote":"1","commit":41,"lead":"0","raftState":"StatePreCandidate","applied":41,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/25]
E240502 06:25:26.651876 3299648 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r48/1:/Table/4{5-6}] 1799  [n1,s1,r48/1:/Table/4{5-6}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r48:/Table/4{5-6} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630855.425441632,0 (2024-05-02 06:20:55); raft status: {"id":"1","term":11,"vote":"1","commit":40,"lead":"0","raftState":"StatePreCandidate","applied":40,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/45]
E240502 06:25:26.652656 3299659 kv/kvserver/queue.go:1202  [T1,Vsystem,n3,replicate,s3,r14/2:/Table/1{1-2}] 1800  [n3,s3,r14/2:/Table/1{1-2}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r14:/Table/1{1-2} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630867.829180892,0 (2024-05-02 06:21:07); raft status: {"id":"2","term":8,"vote":"1","commit":176,"lead":"0","raftState":"StatePreCandidate","applied":176,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Table/11]
E240502 06:25:26.653230 3299660 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r47/1:/Table/4{4-5}] 1801  [n1,s1,r47/1:/Table/4{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r47:/Table/4{4-5} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630868.243609667,0 (2024-05-02 06:21:08); raft status: {"id":"1","term":7,"vote":"1","commit":39,"lead":"0","raftState":"StatePreCandidate","applied":39,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/44]
E240502 06:25:26.653885 3299681 kv/kvserver/queue.go:1202  [T1,Vsystem,n3,replicate,s3,r32/2:/{Table/29-NamespaceTab…}] 1802  [n3,s3,r32/2:/{Table/29-NamespaceTab…}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r32:/{Table/29-NamespaceTable/30} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630873.745492719,0 (2024-05-02 06:21:13); raft status: {"id":"2","term":10,"vote":"1","commit":42,"lead":"0","raftState":"StatePreCandidate","applied":42,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/29]
E240502 06:25:26.654523 3299682 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r47/1:/Table/4{4-5}] 1803  [n1,s1,r47/1:/Table/4{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r47:/Table/4{4-5} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630868.243609667,0 (2024-05-02 06:21:08); raft status: {"id":"1","term":7,"vote":"1","commit":39,"lead":"0","raftState":"StatePreCandidate","applied":39,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/44]
E240502 06:25:26.655089 3299693 kv/kvserver/queue.go:1202  [T1,Vsystem,n3,replicate,s3,r20/2:/Table/1{7-8}] 1804  [n3,s3,r20/2:/Table/1{7-8}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r20:/Table/1{7-8} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630866.804846702,0 (2024-05-02 06:21:06); raft status: {"id":"2","term":8,"vote":"1","commit":39,"lead":"0","raftState":"StatePreCandidate","applied":39,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/17]
E240502 06:25:26.655651 3299694 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r9/1:/Table/{5-6}] 1805  [n1,s1,r9/1:/Table/{5-6}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r9:/Table/{5-6} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630872.740566404,0 (2024-05-02 06:21:12); raft status: {"id":"1","term":10,"vote":"1","commit":58,"lead":"0","raftState":"StatePreCandidate","applied":58,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/5]
E240502 06:25:26.656213 3299705 kv/kvserver/queue.go:1202  [T1,Vsystem,n3,replicate,s3,r35/2:/Table/3{2-3}] 1806  [n3,s3,r35/2:/Table/3{2-3}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r35:/Table/3{2-3} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630857.424347936,0 (2024-05-02 06:20:57); raft status: {"id":"2","term":8,"vote":"2","commit":36,"lead":"0","raftState":"StatePreCandidate","applied":36,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/32]
E240502 06:25:26.658274 3299706 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r35/1:/Table/3{2-3}] 1807  [n1,s1,r35/1:/Table/3{2-3}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r35:/Table/3{2-3} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630857.424347936,0 (2024-05-02 06:20:57); raft status: {"id":"1","term":8,"vote":"2","commit":36,"lead":"0","raftState":"StatePreCandidate","applied":36,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/32]
E240502 06:25:26.658708 3299748 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r21/1:/Table/1{8-9}] 1808  [n1,s1,r21/1:/Table/1{8-9}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r21:/Table/1{8-9} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630871.304740579,0 (2024-05-02 06:21:11); raft status: {"id":"1","term":6,"vote":"1","commit":40,"lead":"0","raftState":"StatePreCandidate","applied":40,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/18]
E240502 06:25:26.659083 3299749 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r9/1:/Table/{5-6}] 1809  [n1,s1,r9/1:/Table/{5-6}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r9:/Table/{5-6} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630872.740566404,0 (2024-05-02 06:21:12); raft status: {"id":"1","term":10,"vote":"1","commit":58,"lead":"0","raftState":"StatePreCandidate","applied":58,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/5]
E240502 06:25:26.659939 3299760 kv/kvserver/queue.go:1202  [T1,Vsystem,n3,replicate,s3,r31/2:/Table/2{8-9}] 1810  [n3,s3,r31/2:/Table/2{8-9}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r31:/Table/2{8-9} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630868.464415262,0 (2024-05-02 06:21:08); raft status: {"id":"2","term":8,"vote":"2","commit":45,"lead":"0","raftState":"StatePreCandidate","applied":45,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/28]
E240502 06:25:26.660557 3299727 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r7/1:/Table/{3-4}] 1811  [n1,s1,r7/1:/Table/{3-4}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r7:/Table/{3-4} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630866.804846702,0 (2024-05-02 06:21:06); raft status: {"id":"1","term":7,"vote":"1","commit":47,"lead":"0","raftState":"StatePreCandidate","applied":47,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/3]
E240502 06:25:26.660958 3299771 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r17/1:/Table/1{4-5}] 1812  [n1,s1,r17/1:/Table/1{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r17:/Table/1{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630857.424347936,0 (2024-05-02 06:20:57); raft status: {"id":"1","term":9,"vote":"1","commit":36,"lead":"0","raftState":"StatePreCandidate","applied":36,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/14]
E240502 06:25:26.661402 3299772 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] 1813  [n1,s1,r3/1:/System/{NodeLive…-tsd}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r3:/System/{NodeLivenessMax-tsd} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630871.427326243,0 (2024-05-02 06:21:11); raft status: {"id":"1","term":7,"vote":"2","commit":114,"lead":"0","raftState":"StatePreCandidate","applied":114,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/System/NodeLivenessMax]
E240502 06:25:26.662263 3299783 kv/kvserver/queue.go:1202  [T1,Vsystem,n3,replicate,s3,r52/2:/Table/5{0-1}] 1814  [n3,s3,r52/2:/Table/5{0-1}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r52:/Table/5{0-1} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630868.464415262,0 (2024-05-02 06:21:08); raft status: {"id":"2","term":8,"vote":"1","commit":41,"lead":"0","raftState":"StatePreCandidate","applied":41,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/50]
E240502 06:25:26.662830 3299784 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r7/1:/Table/{3-4}] 1815  [n1,s1,r7/1:/Table/{3-4}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r7:/Table/{3-4} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630866.804846702,0 (2024-05-02 06:21:06); raft status: {"id":"1","term":7,"vote":"1","commit":47,"lead":"0","raftState":"StatePreCandidate","applied":47,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/3]
E240502 06:25:26.663521 3299805 kv/kvserver/queue.go:1202  [T1,Vsystem,n3,replicate,s3,r4/2:/System{/tsd-tse}] 1816  [n3,s3,r4/2:/System{/tsd-tse}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r4:/System{/tsd-tse} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630860.708105018,0 (2024-05-02 06:21:00); raft status: {"id":"2","term":9,"vote":"1","commit":99,"lead":"0","raftState":"StatePreCandidate","applied":99,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/System/tsd]
E240502 06:25:26.664195 3299806 kv/kvserver/queue.go:1202  [T1,Vsystem,n3,replicate,s3,r8/2:/Table/{4-5}] 1817  [n3,s3,r8/2:/Table/{4-5}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r8:/Table/{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630867.829180892,0 (2024-05-02 06:21:07); raft status: {"id":"2","term":8,"vote":"2","commit":41,"lead":"0","raftState":"StatePreCandidate","applied":41,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Table/4]
E240502 06:25:26.664805 3299817 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r12/1:/Table/{8-9}] 1818  [n1,s1,r12/1:/Table/{8-9}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r12:/Table/{8-9} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630867.829180892,0 (2024-05-02 06:21:07); raft status: {"id":"1","term":7,"vote":"1","commit":39,"lead":"0","raftState":"StatePreCandidate","applied":39,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/8]
E240502 06:25:26.665180 3299818 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r51/1:/Table/{48-50}] 1819  [n1,s1,r51/1:/Table/{48-50}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r51:/Table/{48-50} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630856.908159549,0 (2024-05-02 06:20:56); raft status: {"id":"1","term":11,"vote":"1","commit":37,"lead":"0","raftState":"StatePreCandidate","applied":37,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/48]
E240502 06:25:26.665573 3299819 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r61/1:/Table/{59-60}] 1820  [n1,s1,r61/1:/Table/{59-60}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r61:/Table/{59-60} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630868.243609667,0 (2024-05-02 06:21:08); raft status: {"id":"1","term":7,"vote":"2","commit":39,"lead":"0","raftState":"StatePreCandidate","applied":39,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/59]
E240502 06:25:26.666147 3299830 kv/kvserver/queue.go:1202  [T1,Vsystem,n3,replicate,s3,r17/2:/Table/1{4-5}] 1821  [n3,s3,r17/2:/Table/1{4-5}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r17:/Table/1{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630857.424347936,0 (2024-05-02 06:20:57); raft status: {"id":"2","term":9,"vote":"1","commit":36,"lead":"0","raftState":"StatePreCandidate","applied":36,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/14]
--- FAIL: TestDebugCheckStore (329.93s)
E240502 06:25:26.672598 3299831 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r25/1:/Table/2{2-3}] 1822  [n1,s1,r25/1:/Table/2{2-3}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r25:/Table/2{2-3} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n3,s3):2,(n2,s2):3); closed timestamp: 1714630871.518659348,0 (2024-05-02 06:21:11); raft status: {"id":"1","term":8,"vote":"1","commit":49,"lead":"0","raftState":"StatePreCandidate","applied":49,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Table/22]
E240502 06:25:26.672998 3299832 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r17/1:/Table/1{4-5}] 1823  [n1,s1,r17/1:/Table/1{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r17:/Table/1{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630857.424347936,0 (2024-05-02 06:20:57); raft status: {"id":"1","term":9,"vote":"1","commit":36,"lead":"0","raftState":"StatePreCandidate","applied":36,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/14]
E240502 06:25:26.673447 3299833 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r3/1:/System/{NodeLive…-tsd}] 1824  [n1,s1,r3/1:/System/{NodeLive…-tsd}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r3:/System/{NodeLivenessMax-tsd} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630871.427326243,0 (2024-05-02 06:21:11); raft status: {"id":"1","term":7,"vote":"2","commit":114,"lead":"0","raftState":"StatePreCandidate","applied":114,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/System/NodeLivenessMax]
E240502 06:25:26.673838 3299834 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r12/1:/Table/{8-9}] 1825  [n1,s1,r12/1:/Table/{8-9}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r12:/Table/{8-9} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630867.829180892,0 (2024-05-02 06:21:07); raft status: {"id":"1","term":7,"vote":"1","commit":39,"lead":"0","raftState":"StatePreCandidate","applied":39,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/8]
E240502 06:25:26.674224 3299835 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r51/1:/Table/{48-50}] 1826  [n1,s1,r51/1:/Table/{48-50}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r51:/Table/{48-50} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630856.908159549,0 (2024-05-02 06:20:56); raft status: {"id":"1","term":11,"vote":"1","commit":37,"lead":"0","raftState":"StatePreCandidate","applied":37,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/48]
E240502 06:25:26.674598 3299836 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r61/1:/Table/{59-60}] 1827  [n1,s1,r61/1:/Table/{59-60}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r61:/Table/{59-60} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630868.243609667,0 (2024-05-02 06:21:08); raft status: {"id":"1","term":7,"vote":"2","commit":39,"lead":"0","raftState":"StatePreCandidate","applied":39,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/59]
E240502 06:25:26.674980 3299837 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r32/1:/{Table/29-NamespaceTab…}] 1828  [n1,s1,r32/1:/{Table/29-NamespaceTab…}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r32:/{Table/29-NamespaceTable/30} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630873.745492719,0 (2024-05-02 06:21:13); raft status: {"id":"1","term":10,"vote":"1","commit":42,"lead":"0","raftState":"StatePreCandidate","applied":42,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/29]
E240502 06:25:26.675337 3299838 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r58/1:/Table/5{6-7}] 1829  [n1,s1,r58/1:/Table/5{6-7}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r58:/Table/5{6-7} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630867.829180892,0 (2024-05-02 06:21:07); raft status: {"id":"1","term":8,"vote":"1","commit":39,"lead":"0","raftState":"StatePreCandidate","applied":39,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Table/56]
E240502 06:25:26.675879 3299839 kv/kvserver/queue.go:1202  [T1,Vsystem,n3,replicate,s3,r68/2:/{Table/66-Max}] 1830  [n3,s3,r68/2:/{Table/66-Max}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r68:/{Table/66-Max} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630867.829180892,0 (2024-05-02 06:21:07); raft status: {"id":"2","term":8,"vote":"2","commit":39,"lead":"0","raftState":"StatePreCandidate","applied":39,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/66]
E240502 06:25:26.676678 3299840 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r41/1:/Table/3{8-9}] 1831  [n1,s1,r41/1:/Table/3{8-9}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r41:/Table/3{8-9} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n2,s2):2,(n3,s3):3); closed timestamp: 1714630871.478771401,0 (2024-05-02 06:21:11); raft status: {"id":"1","term":7,"vote":"2","commit":53,"lead":"0","raftState":"StatePreCandidate","applied":53,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Table/38]
E240502 06:25:26.677381 3299851 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r37/1:/Table/3{4-5}] 1832  [n1,s1,r37/1:/Table/3{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r37:/Table/3{4-5} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630875.315373110,0 (2024-05-02 06:21:15); raft status: {"id":"1","term":7,"vote":"2","commit":42,"lead":"0","raftState":"StatePreCandidate","applied":42,"progress":{},"leadtransferee":"0"}: have been waiting 62.00s for slow proposal RequestLease [/Table/34]
E240502 06:25:26.677815 3299852 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r32/1:/{Table/29-NamespaceTab…}] 1833  [n1,s1,r32/1:/{Table/29-NamespaceTab…}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r32:/{Table/29-NamespaceTable/30} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630873.745492719,0 (2024-05-02 06:21:13); raft status: {"id":"1","term":10,"vote":"1","commit":42,"lead":"0","raftState":"StatePreCandidate","applied":42,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/29]
E240502 06:25:26.678561 3299863 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,lease,s1,r23/1:/Table/2{0-1}] 1834  [n1,s1,r23/1:/Table/2{0-1}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r23:/Table/2{0-1} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630871.304740579,0 (2024-05-02 06:21:11); raft status: {"id":"1","term":6,"vote":"1","commit":41,"lead":"0","raftState":"StatePreCandidate","applied":41,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/20]
E240502 06:25:26.679071 3299864 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicaGC,s2,r67/2:/Table/6{5-6}] 1835  replica unavailable: (n2,s2):3 unable to serve request to r1:/{Min-System/NodeLiveness} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):2); closed timestamp: 1714630871.298175213,0 (2024-05-02 06:21:11); raft status: {"id":"3","term":7,"vote":"0","commit":336,"lead":"0","raftState":"StatePreCandidate","applied":336,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Min]

Parameters:

• attempt=1
• run=16
• shard=6

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• cli: TestDebugCheckStore failed #122328 cli: TestDebugCheckStore failed [C-bug C-test-failure O-robot P-3 T-kv branch-master]

/cc @cockroachdb/kv @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-38357

[andrewbaptist]

dupe of #112219 - assigning P3 label.