Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add extended_checks to aptos-transactional-test-harness #13337

Merged
merged 6 commits into from
May 21, 2024

Conversation

brmataptos
Copy link
Contributor

Description

Run extended_checks as part of the aptos-transactional-test-harness.

Fixes #13297.

To run extended_checks we need a model (GlobalEnv) to be generated
by compilation. For Compiler V2 this is pretty straightforward, as we
just have to enable the "attach-compiled-module" experiment. For
Compiler V1, the compilation entry point currently used by the test
framework doesn't generate a model as part of compilation, and reusing
the process used in BuiltPackage::build() (defined in
aptos-move/framework/src/build_package.rs) is overly complex.

So instead, we follow the process that was used in
CompiledPackage::build_all() (defined in
third_party/move/tools/move-package/src/compilation/compiled_package.rs)
to build a model under compiler V1, needed for ABIs or Docgen with
compiler V1.

The main implication is that we need input PackagePaths for
dependencies, notably the Stdlibs, so we make a copy of that in the
framework to be re-used in compile_source_unit() (defined in
third_party/move/testing-infra/transactional-test-runner/src/framework.rs)
if need_model is true.

We just learned that Rust Traits can't call a default implementation
from a specialized implementation, so we must rename the default
implementations of MoveTestAdapter trait functions compile_module
and compile_script as *_default(), leaving the original functions
as forwarders that can be overwritten by new implementations for
AptosTestAdapter (defined in
aptos-move/aptos-transactional-test-harness/src/aptos_test_harness.rs)
to call the default compile methods and then run extended checks as
appropriate.

How Has This Been Tested?

Added a new test that should fails extended checks.

Was reminded that an existing test (diamond_clicker.move) should have been failing
extended checks, as it now fails in bytcode verification.

Key Areas to Review

It's a little messy, but much of the mess relates to compiler-v1 and
the third_party/aptos-move boundary which both should be going away
soonish, so let's not get hung up on it. :-) We really need to
refactor to simplify the compiler/module-building interfaces, but skip
that for now.

Type of Change

  • New feature
  • Bug fix
  • Breaking change
  • Performance improvement
  • Refactoring
  • Dependency update
  • Documentation update
  • Tests

Which Components or Systems Does This Change Impact?

  • Validator Node
  • Full Node (API, Indexer, etc.)
  • Move/Aptos Virtual Machine
  • Aptos Framework
  • Aptos CLI/SDK
  • Developer Infrastructure
  • Other (specify)

Checklist

  • I have read and followed the CONTRIBUTING doc
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I identified and added all stakeholders and component owners affected by this change as reviewers
  • I tested both happy and unhappy path of the functionality
  • [] I have made corresponding changes to the documentation

Copy link

trunk-io bot commented May 18, 2024

⏱️ 16h 59m total CI duration on this PR
Job Cumulative Duration Recent Runs
rust-targeted-unit-tests 2h 25m 🟩🟩🟩🟩 (+2 more)
windows-build 2h 5m 🟩🟩🟩
rust-move-unit-coverage 1h 50m 🟩🟩🟩🟩🟩 (+1 more)
rust-smoke-tests 1h 49m 🟩🟩🟩
rust-move-tests 1h 24m 🟩🟩🟩🟩 (+2 more)
execution-performance / single-node-performance 57m 🟥🟩🟩
forge-framework-upgrade-test / forge 54m 🟥🟥🟥🟩
cli-e2e-tests / run-cli-tests 48m 🟥🟥🟩🟥
forge-e2e-test / forge 45m 🟩🟩🟩
rust-lints 44m 🟥🟥🟩🟩 (+2 more)
rust-images / rust-all 40m 🟩🟩🟩
forge-compat-test / forge 38m 🟩🟩🟩
run-tests-main-branch 33m 🟩🟥🟩🟩🟩 (+3 more)
rust-build-cached-packages 18m 🟩🟩🟩
test-target-determinator 13m 🟩🟩🟩
execution-performance / test-target-determinator 13m 🟩🟩🟩
general-lints 12m 🟩🟩🟩🟩🟩 (+2 more)
check 12m 🟩🟩🟩
check-dynamic-deps 9m 🟩🟩🟩🟩🟩 (+2 more)
semgrep/ci 3m 🟩🟩🟩🟩🟩 (+2 more)
node-api-compatibility-tests / node-api-compatibility-tests 3m 🟩🟩🟩
file_change_determinator 1m 🟩🟩🟩🟩🟩 (+3 more)
file_change_determinator 1m 🟩🟩🟩🟩🟩 (+2 more)
file_change_determinator 35s 🟩🟩🟩
permission-check 24s 🟩🟩🟩🟩🟩 (+3 more)
permission-check 22s 🟩🟩🟩🟩🟩 (+3 more)
permission-check 22s 🟩🟩🟩🟩🟩 (+3 more)
permission-check 20s 🟩🟩🟩🟩🟩 (+3 more)
permission-check 15s 🟩🟩🟩
determine-docker-build-metadata 9s 🟩🟩🟩

🚨 4 jobs on the last run were significantly faster/slower than expected

Job Duration vs 7d avg Delta
cli-e2e-tests / run-cli-tests 11m 6m +74%
rust-move-tests 16m 9m +71%
rust-targeted-unit-tests 24m 19m +27%
forge-framework-upgrade-test / forge 17m 56m -69%

settingsfeedbackdocs ⋅ learn more about trunk.io

Copy link

codecov bot commented May 19, 2024

Codecov Report

Attention: Patch coverage is 4.54545% with 21 lines in your changes are missing coverage. Please review.

Project coverage is 33.0%. Comparing base (ef60a52) to head (0d9e0f8).
Report is 1 commits behind head on main.

Files Patch % Lines
third_party/move/move-compiler/src/shared/mod.rs 0.0% 21 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##             main   #13337     +/-   ##
=========================================
- Coverage    33.0%    33.0%   -0.1%     
=========================================
  Files        1760     1760             
  Lines      338918   338939     +21     
=========================================
- Hits       112023   111961     -62     
- Misses     226895   226978     +83     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@@ -113,7 +113,7 @@ impl<'a> ExtendedChecker<'a> {

fn run(&mut self) {
for ref module in self.env.get_modules() {
if module.is_target() {
if module.is_primary_target() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this @brmataptos. After changing to is_primary_target, we need to be careful and make sure for the V1 path, all target modules are checked.

@@ -874,7 +951,8 @@ fn compile_ir_module<'a>(
) -> Result<CompiledModule> {
use move_ir_compiler::Compiler as IRCompiler;
let code = std::fs::read_to_string(file_name).unwrap();
IRCompiler::new(deps.collect()).into_compiled_module(&code)
let module = IRCompiler::new(deps.collect()).into_compiled_module(&code)?;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A noob question, what is the difference between before and after this code change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing. At some point I had added an Option<GlobalEnv> result to a tuple here, but then realized it wasn't useful. I guess i didn't clean up completely.

}
struct OuterStruct has key {
any_field: vector<InnerStruct>
bug: BYTECODE VERIFICATION FAILED
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This just reveals that the issue #13191 still exists?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

@@ -130,9 +135,10 @@ fn annotated_module_from_unit(unit: &AnnotatedCompiledUnit) -> Option<&Annotated
}
}

impl PreCompiledModules for FullyCompiledProgram {
impl PreCompiledModules for (FullyCompiledProgram, Vec<PackagePaths>) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not even know we can do something like this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only tried it on a whim, and was surprised to find it compiles.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Copy link
Contributor

✅ Forge suite compat success on 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> 0d9e0f8186cd4546d041e1115c874e2e401402b6

Compatibility test results for 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> 0d9e0f8186cd4546d041e1115c874e2e401402b6 (PR)
1. Check liveness of validators at old version: 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411
compatibility::simple-validator-upgrade::liveness-check : committed: 7130.058836375168 txn/s, latency: 4637.164291452442 ms, (p50: 4800 ms, p90: 6000 ms, p99: 7200 ms), latency samples: 248960
2. Upgrading first Validator to new version: 0d9e0f8186cd4546d041e1115c874e2e401402b6
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1827.8265451537045 txn/s, latency: 15620.37712916576 ms, (p50: 18300 ms, p90: 20900 ms, p99: 22500 ms), latency samples: 91820
3. Upgrading rest of first batch to new version: 0d9e0f8186cd4546d041e1115c874e2e401402b6
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1848.106502324234 txn/s, latency: 15446.292321774714 ms, (p50: 19200 ms, p90: 21700 ms, p99: 22200 ms), latency samples: 92860
4. upgrading second batch to new version: 0d9e0f8186cd4546d041e1115c874e2e401402b6
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3581.911241956251 txn/s, latency: 8762.179269655076 ms, (p50: 9700 ms, p90: 12600 ms, p99: 12900 ms), latency samples: 143220
5. check swarm health
Compatibility test for 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> 0d9e0f8186cd4546d041e1115c874e2e401402b6 passed
Test Ok

Copy link
Contributor

✅ Forge suite realistic_env_max_load success on 0d9e0f8186cd4546d041e1115c874e2e401402b6

two traffics test: inner traffic : committed: 6734.9501773879665 txn/s, latency: 5799.647361889785 ms, (p50: 5400 ms, p90: 7200 ms, p99: 13500 ms), latency samples: 2925200
two traffics test : committed: 99.92462438889223 txn/s, latency: 2020.9540229885058 ms, (p50: 2000 ms, p90: 2300 ms, p99: 4700 ms), latency samples: 1740
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.210, avg: 0.202", "QsPosToProposal: max: 0.432, avg: 0.331", "ConsensusProposalToOrdered: max: 0.497, avg: 0.467", "ConsensusOrderedToCommit: max: 0.427, avg: 0.405", "ConsensusProposalToCommit: max: 0.893, avg: 0.871"]
Max round gap was 1 [limit 4] at version 1468347. Max no progress secs was 4.275587 [limit 15] at version 1468347.
Test Ok

Copy link
Contributor

✅ Forge suite framework_upgrade success on 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> 0d9e0f8186cd4546d041e1115c874e2e401402b6

Compatibility test results for 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> 0d9e0f8186cd4546d041e1115c874e2e401402b6 (PR)
Upgrade the nodes to version: 0d9e0f8186cd4546d041e1115c874e2e401402b6
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1215.476568050846 txn/s, submitted: 1218.5117573294979 txn/s, failed submission: 3.035189278651748 txn/s, expired: 3.035189278651748 txn/s, latency: 2681.4979446792163 ms, (p50: 2100 ms, p90: 4600 ms, p99: 6700 ms), latency samples: 104120
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1125.947179507423 txn/s, submitted: 1128.0262777051182 txn/s, failed submission: 2.0790981976952825 txn/s, expired: 2.0790981976952825 txn/s, latency: 2794.981165367255 ms, (p50: 2100 ms, p90: 5100 ms, p99: 7800 ms), latency samples: 97480
5. check swarm health
Compatibility test for 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> 0d9e0f8186cd4546d041e1115c874e2e401402b6 passed
Upgrade the remaining nodes to version: 0d9e0f8186cd4546d041e1115c874e2e401402b6
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 803.8752972078032 txn/s, submitted: 805.8623550696186 txn/s, failed submission: 1.9870578618154981 txn/s, expired: 1.9870578618154981 txn/s, latency: 3924.0746498214776 ms, (p50: 3100 ms, p90: 6800 ms, p99: 8000 ms), latency samples: 72820
Test Ok

@brmataptos brmataptos merged commit 327fc17 into main May 21, 2024
53 of 54 checks passed
@brmataptos brmataptos deleted the brm-issue-13297 branch May 21, 2024 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]aptos-transactional-test-harness doesn't run extended_checks when compiling examples
3 participants