test(scale): introduce deterministic scaling tests #5657

BugenZhao · 2022-09-30T04:47:10Z

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

What's changed and what's your intention?

As explained in #5655.

The nexmark_q4.rs shows how to manually write scaling cases, and tests issue #5523. I'm planning to test more queries with random plans after resolving some blockers:

scale: cache invalidation when scaling #5567
Some columns are missing in nexmark source, so not all queries can run.
The results of some queries are generated slowly or updated infrequently, which might not be suitable for tests. Need to control the timing carefully.

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
All checks passed in ./risedev check (or alias, ./risedev c)

Refer to a related PR or issue link (optional)

Close Introduce a framework with utilities to allow writing scaling test cases manually. #5656

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

codecov · 2022-09-30T08:00:28Z

Codecov Report

Merging #5657 (1870972) into main (bd3bd59) will decrease coverage by 0.01%.
The diff coverage is 54.22%.

@@            Coverage Diff             @@
##             main    #5657      +/-   ##
==========================================
- Coverage   74.30%   74.29%   -0.02%     
==========================================
  Files         924      924              
  Lines      144308   144263      -45     
==========================================
- Hits       107225   107175      -50     
- Misses      37083    37088       +5

Flag	Coverage Δ
rust	`74.29% <54.22%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/common/src/catalog/column.rs	`77.17% <0.00%> (-11.03%)`	⬇️
src/common/src/error.rs	`72.30% <ø> (ø)`
src/common/src/hash/key.rs	`84.54% <ø> (ø)`
...c/compute/src/compute_observer/observer_manager.rs	`64.70% <ø> (ø)`
src/compute/src/server.rs	`0.00% <0.00%> (ø)`
src/connector/src/lib.rs	`100.00% <ø> (ø)`
src/connector/src/source/dummy_connector.rs	`0.00% <0.00%> (ø)`
.../src/source/filesystem/s3/source/s3_file_reader.rs	`0.00% <0.00%> (ø)`
...rc/connector/src/source/kafka/enumerator/client.rs	`0.00% <0.00%> (ø)`
src/connector/src/source/kafka/source/reader.rs	`0.00% <0.00%> (ø)`
... and 132 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

BugenZhao

--

wangrunji0408

LGTM!

wangrunji0408 · 2022-09-30T06:00:15Z

Cargo.toml

@@ -30,6 +30,7 @@ members = [
  "src/test_runner",
  "src/tests/regress",
  "src/tests/simulation",
+  "src/tests/simulation_scale",


I prefer merging it into the existing simulation crate, maybe in a future PR.

BugenZhao

cc @wangrunji0408

src/meta/src/storage/mem_meta_store.rs

BugenZhao · 2022-10-08T06:03:37Z

Makefile.toml

+category = "RiseDev - Archive simulation scaling tests"
+description = "Archive integration scaling tests in deterministic simulation mode"
+dependencies = ["warn-on-missing-tools"]
+env = { RUSTFLAGS = "-Ctarget-cpu=native --cfg tokio_unstable --cfg madsim", RUSTDOCFLAGS = "--cfg madsim", CARGO_TARGET_DIR = "target/sim" }


We can't decide how to enable SIMD this way through environments. Thus, the results of JSON parsing might be different due to precision errors like #5487. 🥵

BugenZhao · 2022-10-08T06:05:03Z

src/connector/src/source/nexmark/source/generator.rs

+            if chunk.is_empty() {
+                yield pending().await;


This could be CPU intensive if there are no remaining records to generate, which is problematic with madsim. 🤣

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

yezizp2012

LSTM!!!

src/tests/simulation_scale/src/cluster.rs

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

wangrunji0408 · 2022-10-08T07:35:21Z

src/connector/src/source/nexmark/source/generator.rs

+        // The connector assumes that the stream will never end, so if the `event_num` is hit, we
+        // pend the stream forever.
+        // TODO: should we allow the stream to finish?
+        let () = pending().await;


I think the stream generator themselves should always end the stream gracefully, as long as they are not infinite streams. Even for those down-streams, they should end themselves as well when the upstream is closed. This way the errors can be propagated to the sink. The system won't be blocking implicitly.

We've discussed this issue and decided not to use the Stream terminate or TryStream error to represent the stream state in RisingWave as we don't want to propagate the error through the network. 🥵 The error yielded in actor internally should be collected by the Actor instance and find a way to report it to meta service if possible. cc @fuyufjh

For the stream reader, I find that the refactor just merged has removed the assumption (as we're all-in async stream), so maybe the workaround can be removed. Note that there's a select of source reader and barrier receiver, the source executor will work correctly after the source is gracefully terminated. On error, it will propagate to the actor of this source executor.

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

BugenZhao · 2022-10-08T11:12:10Z

After allowing the source reader part to terminate, the select_with_strategy will terminate unexpectedly (which causes CI to fail). I believe this is a bug of futures. 🤯
rust-lang/futures-rs#2635

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

BugenZhao added 8 commits September 29, 2022 16:12

initial scale sim

2fa5c9c

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

test q4

ffd39db

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

locate fragment

d3b75ff

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

fix predicate & add cascade

d30b66c

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

add docs

08e61b1

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

add workflows

53d222f

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

add license header

9183918

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

fix workflow

99cd61a

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

github-actions bot added the component/test Test related issue. label Sep 30, 2022

BugenZhao added 2 commits September 30, 2022 13:10

fix ci & add cfg madsim

6743964

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

try use ci-sim

b7861bc

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

wangrunji0408 mentioned this pull request Sep 30, 2022

Tracking: deterministic simulation testing #4180

Open

22 tasks

BugenZhao added 3 commits September 30, 2022 14:56

align cfg with sslt

8e89c8c

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

align with sslt

d887c4e

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

correct result & soft fail

62703bb

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

do not use shared mem store

821012c

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

BugenZhao commented Sep 30, 2022

View reviewed changes

BugenZhao requested review from wangrunji0408, fuyufjh, shanicky and yezizp2012 September 30, 2022 08:41

wangrunji0408 approved these changes Oct 8, 2022

View reviewed changes

BugenZhao commented Oct 8, 2022

View reviewed changes

BugenZhao added 2 commits October 8, 2022 14:15

Merge remote-tracking branch 'origin/main' into bz/scale-sim

f68b67b

avoid hard coded thoughput

9baad0e

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

yezizp2012 approved these changes Oct 8, 2022

View reviewed changes

src/tests/simulation_scale/src/cluster.rs Outdated Show resolved Hide resolved

src/tests/simulation_scale/src/cluster.rs Outdated Show resolved Hide resolved

src/tests/simulation_scale/src/cluster.rs Outdated Show resolved Hide resolved

BugenZhao added 3 commits October 8, 2022 14:42

use ci-sim profile

c1d8bba

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

remove comments

aa96954

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

fix connector

f88edfa

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

increase opt level for ci-sim

c2ab7ba

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

wangrunji0408 reviewed Oct 8, 2022

View reviewed changes

BugenZhao added 2 commits October 8, 2022 15:55

remove stream workaround

4d1ee73

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

fix clippy

21df22d

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

BugenZhao added 2 commits October 8, 2022 19:18

minor fixes

b68306b

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

bump futures to 0.3.24

e275e05

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

BugenZhao added the mergify/can-merge Indicates that the PR can be added to the merge queue label Oct 8, 2022

Merge branch 'main' into bz/scale-sim

1870972

mergify bot merged commit cb0d309 into main Oct 8, 2022

mergify bot deleted the bz/scale-sim branch October 8, 2022 11:59

This was referenced Oct 9, 2022

chore: collect static log features to a separate feature #5725

Merged

chore(test): add task for running scale tests in risedev #5803

Merged

BugenZhao mentioned this pull request Nov 11, 2022

source: source reader will stall after replacing if previous reader has finished #6300

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(scale): introduce deterministic scaling tests #5657

test(scale): introduce deterministic scaling tests #5657

BugenZhao commented Sep 30, 2022 •

edited

codecov bot commented Sep 30, 2022 •

edited

BugenZhao left a comment •

edited

wangrunji0408 left a comment

wangrunji0408 Sep 30, 2022

BugenZhao left a comment

BugenZhao Oct 8, 2022

BugenZhao Oct 8, 2022

yezizp2012 left a comment

wangrunji0408 Oct 8, 2022

BugenZhao Oct 8, 2022

BugenZhao commented Oct 8, 2022 •

edited

test(scale): introduce deterministic scaling tests #5657

test(scale): introduce deterministic scaling tests #5657

Conversation

BugenZhao commented Sep 30, 2022 • edited

What's changed and what's your intention?

Checklist

Refer to a related PR or issue link (optional)

codecov bot commented Sep 30, 2022 • edited

Codecov Report

BugenZhao left a comment • edited

Choose a reason for hiding this comment

wangrunji0408 left a comment

Choose a reason for hiding this comment

wangrunji0408 Sep 30, 2022

Choose a reason for hiding this comment

BugenZhao left a comment

Choose a reason for hiding this comment

BugenZhao Oct 8, 2022

Choose a reason for hiding this comment

BugenZhao Oct 8, 2022

Choose a reason for hiding this comment

yezizp2012 left a comment

Choose a reason for hiding this comment

wangrunji0408 Oct 8, 2022

Choose a reason for hiding this comment

BugenZhao Oct 8, 2022

Choose a reason for hiding this comment

BugenZhao commented Oct 8, 2022 • edited

BugenZhao commented Sep 30, 2022 •

edited

codecov bot commented Sep 30, 2022 •

edited

BugenZhao left a comment •

edited

BugenZhao commented Oct 8, 2022 •

edited