Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

github.com/cockroachdb/pebble/internal/metamorphic: TestMeta failed #3610

Closed
cockroach-teamcity opened this issue May 12, 2024 · 3 comments · Fixed by #3613
Closed

github.com/cockroachdb/pebble/internal/metamorphic: TestMeta failed #3610

cockroach-teamcity opened this issue May 12, 2024 · 3 comments · Fixed by #3613
Assignees
@jbowens
Labels
A-storage branch-refs/heads/master C-test-failure metamorphic-failure O-robot release-blocker T-storage
Projects
Storage

Comments

@cockroach-teamcity
Copy link
Member

github.com/cockroachdb/pebble/internal/metamorphic.TestMeta failed with artifacts on refs/heads/master @ 5ee10bd4b992:

=== CONT  TestMeta/execution/random-000
=== RUN   TestMeta/execution/random-013
=== PAUSE TestMeta/execution/random-013
=== CONT  TestMeta/execution/random-013
=== RUN   TestMeta/compare/standard-016
=== RUN   TestMeta/compare/standard-017
=== RUN   TestMeta/execution/standard-005
=== PAUSE TestMeta/execution/standard-005
=== CONT  TestMeta/execution/standard-005
=== RUN   TestMeta/execution/random-020
=== PAUSE TestMeta/execution/random-020
=== CONT  TestMeta/execution/random-020
=== RUN   TestMeta/execution/standard-026
=== PAUSE TestMeta/execution/standard-026
=== CONT  TestMeta/execution/standard-026
=== RUN   TestMeta/execution/random-004
=== PAUSE TestMeta/execution/random-004
=== CONT  TestMeta/execution/random-004
=== RUN   TestMeta/execution/standard-004
=== PAUSE TestMeta/execution/standard-004
=== CONT  TestMeta/execution/standard-004
=== RUN   TestMeta/execution/standard-006
=== PAUSE TestMeta/execution/standard-006
=== CONT  TestMeta/execution/standard-006
=== RUN   TestMeta/execution/standard-019
=== PAUSE TestMeta/execution/standard-019
=== CONT  TestMeta/execution/standard-019
=== RUN   TestMeta/execution/standard-021
=== PAUSE TestMeta/execution/standard-021
=== CONT  TestMeta/execution/standard-021
=== RUN   TestMeta/execution/standard-003
=== PAUSE TestMeta/execution/standard-003
=== CONT  TestMeta/execution/standard-003
=== RUN   TestMeta/execution/standard-016
=== PAUSE TestMeta/execution/standard-016
=== CONT  TestMeta/execution/standard-016
=== RUN   TestMeta/compare/random-001
=== RUN   TestMeta/compare/standard-007
=== RUN   TestMeta/compare/standard-009
=== RUN   TestMeta/compare/standard-028
=== RUN   TestMeta/execution/random-015
=== PAUSE TestMeta/execution/random-015
=== CONT  TestMeta/execution/random-015
=== RUN   TestMeta/compare/standard-014
=== RUN   TestMeta/execution/random-019
=== PAUSE TestMeta/execution/random-019
=== CONT  TestMeta/execution/random-019
=== RUN   TestMeta/execution/standard-002
=== PAUSE TestMeta/execution/standard-002
=== CONT  TestMeta/execution/standard-002
Help

To reproduce, try: 

go test -tags 'invariants' -exec 'stress -p 1' -timeout 0 -test.v -run 'TestMeta$' ./internal/metamorphic -seed 1715490363514105400 -ops "uniform:5000-10000"

This test on roachdash | Improve this report!
@blathers-crl blathers-crl bot added this to Incoming in Storage May 12, 2024
@jbowens
Copy link
Collaborator

jbowens commented May 12, 2024 •
edited 

===== SEED =====
1715490363514105400
===== DIFF =====
/artifacts/meta/240512-050603.5143181144998/{standard-000,random-013}
@@ -1197,11 +1197,11 @@
 iter10 = db1.NewIter("", "", 2 /* key types */, 0, 0, false /* use L6 filters */, "" /* masking suffix */) // <nil> #1196
 db1.Set("upqpcz@2", "rlnhf") // <nil> #1197
 iter10.SeekPrefixGE("mgheqkv@6") // [true,"mgheqkv@6",<no point>,["mgheqkv","mgheqkv\x00")=>{"@6"="chyitvknodlqcvfbgesk"}*] <nil> #1198
 db1.Delete("kucaygxoh") // <nil> #1199
 iter9.NextPrefix() // [true,"bxoeqyxe",<no point>,["bxoeqyxe","drfw")=>{"@6"="vjgtofxwbbhuyfpwy","@4"="ucvjaaiedhdmm"}*] <nil> #1200
-iter10.SeekPrefixGE("uznszonw@4") // [true,"uznszonw@4","uychbstvqto",<no range>*] <nil> #1201
+iter10.SeekPrefixGE("uznszonw@4") // [false] <nil> #1201
 iter10.Next("") // [false] <nil> #1202
 iter10.Prev("") // [false] pebble: unsupported reverse prefix iteration #1203
 db1.Delete("bxoeqyxe@1") // <nil> #1204
 db1.DeleteRange("isycphwfdbbh@5", "ohzblxra") // <nil> #1205
 iter9.Next("") // [true,"fcqygccosb",<no point>,["fcqygccosb","pzelvmwgzp")=>{"@6"="chyitvknodlqcvfbgesk"}*] <nil> #1206

Can reproduce the errant run easily with

go test -tags invariants -v -run TestMeta$ -run-dir ~/Downloads/Nightlies_Nightly_Pebble_Metamorphic_28547_artifacts/meta/240512-050603.5143181144998/random-013 -fail 'false.*#1201$' -max-threads 1 | grep -B 100 '#1201$'

@cockroach-teamcity
Copy link
Member Author

github.com/cockroachdb/pebble/internal/metamorphic.TestMeta failed with artifacts on refs/heads/master @ 5ee10bd4b992:

db1.IngestExternalFiles(external0, "cmvuwqzcyhibrt" /* start */, "cmvuwqzcyhibrtzm" /* end */, "@327" /* syntheticSuffix */, "cm" /* syntheticPrefix */)
db1.IngestExternalFiles(external0, "plrvuwqzcyhibrt" /* start */, "plrvuwqzcyhibrty" /* end */, "" /* syntheticSuffix */, "plr" /* syntheticPrefix */)
db1.IngestExternalFiles(external0, "vuwqzcyhibrt" /* start */, "vuwqzcyhibrtngs" /* end */, "" /* syntheticSuffix */, "" /* syntheticPrefix */)
db1.Get("rbbzby@15")
db1.Close()

To reduce:  go test ./internal/metamorphic -tags invariants -run 'TestMeta$' --compare "/artifacts/meta/240513-055300.7043438092545/{standard-000,random-006}" --try-to-reduce -v
=== RUN   TestMeta/execution/random-011
=== PAUSE TestMeta/execution/random-011
=== CONT  TestMeta/execution/random-011
=== RUN   TestMeta/execution/random-022
=== PAUSE TestMeta/execution/random-022
=== CONT  TestMeta/execution/random-022
=== RUN   TestMeta/execution/standard-010
=== PAUSE TestMeta/execution/standard-010
=== CONT  TestMeta/execution/standard-010
=== RUN   TestMeta/execution/standard-013
=== PAUSE TestMeta/execution/standard-013
=== CONT  TestMeta/execution/standard-013
=== RUN   TestMeta/execution/standard-018
=== PAUSE TestMeta/execution/standard-018
=== CONT  TestMeta/execution/standard-018
=== RUN   TestMeta/execution/standard-026
=== PAUSE TestMeta/execution/standard-026
=== CONT  TestMeta/execution/standard-026
=== RUN   TestMeta/compare/random-005
=== RUN   TestMeta/compare/standard-015
=== RUN   TestMeta/compare/standard-016
=== RUN   TestMeta/compare/standard-020
=== RUN   TestMeta/compare/standard-026
=== RUN   TestMeta/execution/random-021
=== PAUSE TestMeta/execution/random-021
=== CONT  TestMeta/execution/random-021
=== RUN   TestMeta/execution/standard-012
=== PAUSE TestMeta/execution/standard-012
=== CONT  TestMeta/execution/standard-012
=== RUN   TestMeta/execution/standard-014
=== PAUSE TestMeta/execution/standard-014
=== CONT  TestMeta/execution/standard-014
=== RUN   TestMeta/compare/standard-001
=== RUN   TestMeta/compare/standard-011
=== RUN   TestMeta/execution/random-007
=== PAUSE TestMeta/execution/random-007
=== CONT  TestMeta/execution/random-007
=== RUN   TestMeta/execution/random-012
=== PAUSE TestMeta/execution/random-012
=== CONT  TestMeta/execution/random-012
=== RUN   TestMeta/execution/random-017
=== PAUSE TestMeta/execution/random-017
=== CONT  TestMeta/execution/random-017
Help

To reproduce, try: 

go test -tags 'invariants' -exec 'stress -p 1' -timeout 0 -test.v -run 'TestMeta$' ./internal/metamorphic -seed 1715579580704542328 -ops "uniform:5000-10000"

This test on roachdash | Improve this report!

@jbowens jbowens self-assigned this May 13, 2024
@jbowens jbowens moved this from Incoming to In Progress (this milestone) in Storage May 13, 2024
@jbowens jbowens mentioned this issue May 13, 2024
Closed
@jbowens
Copy link
Collaborator

jbowens commented May 13, 2024

This is a side effect of #3596 with an interaction of TrySeekUsingNext:

During this operation:

iter10.SeekPrefixGE("mgheqkv@6") // [true,"mgheqkv@6",<no point>,["mgheqkv","mgheqkv\x00")=>{"@6"="chyitvknodlqcvfbgesk"}*] <nil> #1198
  1. We position a levelIter onto 000089:[crmuudvmizei@1#355,DEL-zrogino@5#361,MERGE] and return a range deletion start boundary key: mgheqkv@6#72057594037927935,RANGEDEL.
  2. The merging iterator steps the levelIter onto the range deletion end boundary key: mgheqkv@1#72057594037927935,RANGEDEL.
  3. The merging iterator steps the levelIter off the range deletion end boundary key, exhausting the levelIter's internal iterator (because bloom filters excluded the remaining point keys in the file). This moves on to the next file in the level, but there isn't one, exhausting the manifest.LevelIterator.

Then during the diverging operation iter10.SeekPrefixGE("uznszonw@4"):

  1. We perform a levelIter.SeekPrefixGE("uznszonw", "uznszonw@4", TrySeekUsingNext()=true)
  2. findFileGE finds that we've already exhausted the level iterator and returns nil, missing a relevant point key in 000089.

We need to not move the levelIter beyond the file 000089, but when performing the levelIter.Next don't today have context that we're in prefix iteration mode and that we shouldn't progress to the next file.

jbowens added a commit to jbowens/pebble that referenced this issue May 13, 2024
@jbowens
db: avoid stepping beyond iteration prefix in levelIter
e9ba520 
During prefix iteration mode, avoid stepping the levelIter beyond the current
iteration prefix. This fixes a bug whereby a levelIter.Next may position a
levelIter to a file beyond the iteration prefix and a subsequent SeekPrefixGE
to a later key with TrySeekUsingNext would fail to observe keys in the skipped
file.

Fix cockroachdb#3610.
@jbowens jbowens mentioned this issue May 13, 2024
Merged
jbowens added a commit to jbowens/pebble that referenced this issue May 13, 2024
@jbowens
db: avoid stepping beyond iteration prefix in levelIter
68a3f91 
During prefix iteration mode, avoid stepping the levelIter beyond the current
iteration prefix. This fixes a bug whereby a levelIter.Next may position a
levelIter to a file beyond the iteration prefix and a subsequent SeekPrefixGE
to a later key with TrySeekUsingNext would fail to observe keys in the skipped
file.

Fix cockroachdb#3610.
jbowens added a commit that referenced this issue May 13, 2024
@jbowens
db: avoid stepping beyond iteration prefix in levelIter
c4b8017 
During prefix iteration mode, avoid stepping the levelIter beyond the current
iteration prefix. This fixes a bug whereby a levelIter.Next may position a
levelIter to a file beyond the iteration prefix and a subsequent SeekPrefixGE
to a later key with TrySeekUsingNext would fail to observe keys in the skipped
file.

Fix #3610.
Storage automation moved this from In Progress (this milestone) to Done May 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jbowens jbowens

Labels
A-storage branch-refs/heads/master C-test-failure metamorphic-failure O-robot release-blocker T-storage
Projects
Storage
  
Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

db: avoid stepping beyond iteration prefix in levelIter jbowens/pebble
2 participants
@jbowens @cockroach-teamcity