Tracking: deterministic simulation testing

[wangrunji0408]

We are utilizing deterministic simulation testing to find bugs efficiently and reproduce them at any time.

Status and TODO

• Unit test
  • Find out why some tests failed
  • Simulate fs API for disk object store
  • deterministic compaction test #6672
• End-to-end test
  • Batch and Streaming
  • Cluster: 3 compute + 1 frontend
  • Make things deterministic
  • Compactor
  • test: build a kafka simulator #5181
  • Parallel test @KveinAxel
  • Etcd
  • test: add S3 simulator #5161
  • test: long-running nexmark on madsim #5170
  • test: simulate SQL-backend meta store #12707
• Fault injection
  • Tracking: test failover on deterministic simulator #4527
  • fuzz test deterministic simulation #5010
  • test: inject blocking, network delay and partition in simulation test #6424
  • scale: deterministic e2e tests for scaling #5655
• Other tools
  • Tracking: deterministic testing with hermit #6665

Pull Requests

• feat(test): initial support for deterministic simulation testing #2477
• feat(test): deterministic testing for more crates #2718
• fix(build): lock madsim version #2787
• ci: add deterministic simulation test #2888
• fix(test): make deterministic test compile #3951
• feat(net): add tcp simulation madsim-rs/madsim#16
• feat(test): e2e simulation test #4176
• feat(test): make simulation test deterministic! #4336
• fix(test): fix or ignore several unit tests on madsim #4460
• feat(test): make deterministic test ready #4476
• fix(test): fix deterministic test #4550
• feat(test): parallel e2e test under deterministic simulation #4675
• ci(test): add deterministic parallel e2e test #4738
• feat(test): test fault-recovery on madsim #4843
• feat(test): add compactor to deterministic test #4917
• feat(test): deterministic e2e fuzzing #4950
• feat(test): add etcd simulator #5084
• doc(test): add developer guide for deterministic test #5342
• feat(test): add kafka simulator #5536
• test(scale): introduce deterministic scaling tests #5657
• refactor(test): merge simulation and simulation_scale #6676
• feat(test): more chaos when killing nodes in recovery test #6708
• feat(test): support load/dump for etcd simulator #6778

Hunted Bugs

• type: execution order
  introduced by feat(batch-task): init task status management #4437
  fixed by fix(batch): consume dml status and ensure task id always found #4516
• bug(recovery): panicked at 'no table reader exists' #4875
• bug(recovery): panicked at 'sender not exist' in compute node #4782
• bug(sqlsmith): should allow 'Expr error: Out of range' #5143
• bug: several problems caused by "etcdserver: request timed out" #5103

Related Links

• "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson
• sled simulation guide (jepsen-proof engineering)
• Testing sync at Dropbox
• BUGGIFY
• Project Coyote from Microsoft
• Project Hermit from Facebook

[lmatz]

Is it possible to add e2e simulation tests for the connector sink or source?
e.g. #6631

[wangrunji0408]

Is it possible to add e2e simulation tests for the connector sink or source?
e.g. #6631

Theoretically it's possible. But we have to build a dedicated simulator for each kind of source and sink. Each of them will take us at least 0.5 man-months in my prediction. Currently we already have a simulator for kafka. With this, the complete data flow from source to sink has been covered in the test. So any more simulator for a connector will only result in the connector code itself being tested, which I don't think the ROI is high. 🤔

By the way, I just found a new project hermit published by Meta a week ago. It is a system-level deterministic testing tool. It seems like a good solution to this case. I'm going to do some research on it soon. (UPDATE: tracking in #6665)