assertion failed: prev.start > max on first invocation (without fs cache)

[kdy1]

Describe the bug

swc-project/swc#7304

The Wasm plugin runner of SWC fails with an assertion failure.

Versions of wasmer & deps

wasmer v3.2.0
├── bytes v1.4.0
│   └── serde v1.0.160
│       └── serde_derive v1.0.160 (proc-macro)
│           ├── proc-macro2 v1.0.56
│           │   └── unicode-ident v1.0.8
│           ├── quote v1.0.26
│           │   └── proc-macro2 v1.0.56 (*)
│           └── syn v2.0.15
│               ├── proc-macro2 v1.0.56 (*)
│               ├── quote v1.0.26 (*)
│               └── unicode-ident v1.0.8
├── cfg-if v1.0.0
├── derivative v2.2.0 (proc-macro)
│   ├── proc-macro2 v1.0.56 (*)
│   ├── quote v1.0.26 (*)
│   └── syn v1.0.109
│       ├── proc-macro2 v1.0.56 (*)
│       ├── quote v1.0.26 (*)
│       └── unicode-ident v1.0.8
├── indexmap v1.9.3
│   ├── hashbrown v0.12.3
│   │   └── ahash v0.7.6
│   │       ├── getrandom v0.2.9
│   │       │   ├── cfg-if v1.0.0
│   │       │   └── libc v0.2.141
│   │       ├── once_cell v1.17.1
│   │       └── serde v1.0.160 (*)
│   │       [build-dependencies]
│   │       └── version_check v0.9.4
│   ├── rayon v1.7.0
│   │   ├── either v1.8.1
│   │   └── rayon-core v1.11.0
│   │       ├── crossbeam-channel v0.5.8
│   │       │   ├── cfg-if v1.0.0
│   │       │   └── crossbeam-utils v0.8.15
│   │       │       └── cfg-if v1.0.0
│   │       ├── crossbeam-deque v0.8.3
│   │       │   ├── cfg-if v1.0.0
│   │       │   ├── crossbeam-epoch v0.9.14
│   │       │   │   ├── cfg-if v1.0.0
│   │       │   │   ├── crossbeam-utils v0.8.15 (*)
│   │       │   │   ├── memoffset v0.8.0
│   │       │   │   │   [build-dependencies]
│   │       │   │   │   └── autocfg v1.1.0
│   │       │   │   └── scopeguard v1.1.0
│   │       │   │   [build-dependencies]
│   │       │   │   └── autocfg v1.1.0
│   │       │   └── crossbeam-utils v0.8.15 (*)
│   │       ├── crossbeam-utils v0.8.15 (*)
│   │       └── num_cpus v1.15.0
│   │           └── libc v0.2.141
│   └── serde v1.0.160 (*)
│   [build-dependencies]
│   └── autocfg v1.1.0
├── more-asserts v0.2.2
├── target-lexicon v0.12.6
├── thiserror v1.0.40
│   └── thiserror-impl v1.0.40 (proc-macro)
│       ├── proc-macro2 v1.0.56 (*)
│       ├── quote v1.0.26 (*)
│       └── syn v2.0.15 (*)
├── wasmer-compiler v3.2.0
│   ├── backtrace v0.3.67
│   │   ├── addr2line v0.19.0
│   │   │   └── gimli v0.27.2
│   │   ├── cfg-if v1.0.0
│   │   ├── libc v0.2.141
│   │   ├── miniz_oxide v0.6.2
│   │   │   └── adler v1.0.2
│   │   ├── object v0.30.3
│   │   │   └── memchr v2.5.0
│   │   └── rustc-demangle v0.1.22
│   │   [build-dependencies]
│   │   └── cc v1.0.79
│   ├── cfg-if v1.0.0
│   ├── enum-iterator v0.7.0
│   │   └── enum-iterator-derive v0.7.0 (proc-macro)
│   │       ├── proc-macro2 v1.0.56 (*)
│   │       ├── quote v1.0.26 (*)
│   │       └── syn v1.0.109 (*)
│   ├── enumset v1.0.12
│   │   └── enumset_derive v0.6.1 (proc-macro)
│   │       ├── darling v0.14.4
│   │       │   ├── darling_core v0.14.4
│   │       │   │   ├── fnv v1.0.7
│   │       │   │   ├── ident_case v1.0.1
│   │       │   │   ├── proc-macro2 v1.0.56 (*)
│   │       │   │   ├── quote v1.0.26 (*)
│   │       │   │   └── syn v1.0.109 (*)
│   │       │   └── darling_macro v0.14.4 (proc-macro)
│   │       │       ├── darling_core v0.14.4 (*)
│   │       │       ├── quote v1.0.26 (*)
│   │       │       └── syn v1.0.109 (*)
│   │       ├── proc-macro2 v1.0.56 (*)
│   │       ├── quote v1.0.26 (*)
│   │       └── syn v1.0.109 (*)
│   ├── lazy_static v1.4.0
│   ├── leb128 v0.2.5
│   ├── memmap2 v0.5.10
│   │   └── libc v0.2.141
│   ├── more-asserts v0.2.2
│   ├── region v3.0.0
│   │   ├── bitflags v1.3.2
│   │   ├── libc v0.2.141
│   │   └── mach v0.3.2
│   │       └── libc v0.2.141
│   ├── rustc-demangle v0.1.22
│   ├── smallvec v1.10.0
│   ├── thiserror v1.0.40 (*)
│   ├── wasmer-types v3.2.0
│   │   ├── bytecheck v0.6.10
│   │   │   ├── bytecheck_derive v0.6.10 (proc-macro)
│   │   │   │   ├── proc-macro2 v1.0.56 (*)
│   │   │   │   ├── quote v1.0.26 (*)
│   │   │   │   └── syn v1.0.109 (*)
│   │   │   ├── ptr_meta v0.1.4
│   │   │   │   └── ptr_meta_derive v0.1.4 (proc-macro)
│   │   │   │       ├── proc-macro2 v1.0.56 (*)
│   │   │   │       ├── quote v1.0.26 (*)
│   │   │   │       └── syn v1.0.109 (*)
│   │   │   └── simdutf8 v0.1.4
│   │   ├── enum-iterator v0.7.0 (*)
│   │   ├── enumset v1.0.12 (*)
│   │   ├── indexmap v1.9.3 (*)
│   │   ├── more-asserts v0.2.2
│   │   ├── rkyv v0.7.41
│   │   │   ├── bytecheck v0.6.10 (*)
│   │   │   ├── hashbrown v0.12.3 (*)
│   │   │   ├── indexmap v1.9.3 (*)
│   │   │   ├── ptr_meta v0.1.4 (*)
│   │   │   ├── rend v0.4.0
│   │   │   │   └── bytecheck v0.6.10 (*)
│   │   │   ├── rkyv_derive v0.7.41 (proc-macro)
│   │   │   │   ├── proc-macro2 v1.0.56 (*)
│   │   │   │   ├── quote v1.0.26 (*)
│   │   │   │   └── syn v1.0.109 (*)
│   │   │   └── seahash v4.1.0
│   │   ├── target-lexicon v0.12.6
│   │   └── thiserror v1.0.40 (*)
│   ├── wasmer-vm v3.2.0
│   │   ├── backtrace v0.3.67 (*)
│   │   ├── cfg-if v1.0.0
│   │   ├── corosensei v0.1.3
│   │   │   ├── cfg-if v1.0.0
│   │   │   ├── libc v0.2.141
│   │   │   └── scopeguard v1.1.0
│   │   │   [build-dependencies]
│   │   │   └── autocfg v1.1.0
│   │   ├── dashmap v5.4.0
│   │   │   ├── cfg-if v1.0.0
│   │   │   ├── hashbrown v0.12.3 (*)
│   │   │   ├── lock_api v0.4.9
│   │   │   │   └── scopeguard v1.1.0
│   │   │   │   [build-dependencies]
│   │   │   │   └── autocfg v1.1.0
│   │   │   ├── once_cell v1.17.1
│   │   │   └── parking_lot_core v0.9.7
│   │   │       ├── cfg-if v1.0.0
│   │   │       ├── libc v0.2.141
│   │   │       └── smallvec v1.10.0
│   │   ├── derivative v2.2.0 (proc-macro) (*)
│   │   ├── enum-iterator v0.7.0 (*)
│   │   ├── fnv v1.0.7
│   │   ├── indexmap v1.9.3 (*)
│   │   ├── lazy_static v1.4.0
│   │   ├── libc v0.2.141
│   │   ├── mach v0.3.2 (*)
│   │   ├── memoffset v0.8.0 (*)
│   │   ├── more-asserts v0.2.2
│   │   ├── region v3.0.0 (*)
│   │   ├── scopeguard v1.1.0
│   │   ├── thiserror v1.0.40 (*)
│   │   └── wasmer-types v3.2.0 (*)
│   │   [build-dependencies]
│   │   └── cc v1.0.79
│   └── wasmparser v0.95.0
│       ├── indexmap v1.9.3 (*)
│       └── url v2.3.1
│           ├── form_urlencoded v1.1.0
│           │   └── percent-encoding v2.2.0
│           ├── idna v0.3.0
│           │   ├── unicode-bidi v0.3.13
│           │   └── unicode-normalization v0.1.22
│           │       └── tinyvec v1.6.0
│           │           └── tinyvec_macros v0.1.1
│           ├── percent-encoding v2.2.0
│           └── serde v1.0.160 (*)
├── wasmer-compiler-cranelift v3.2.0
│   ├── cranelift-codegen v0.91.1
│   │   ├── arrayvec v0.7.2
│   │   ├── bumpalo v3.12.0
│   │   ├── cranelift-bforest v0.91.1
│   │   │   └── cranelift-entity v0.91.1
│   │   ├── cranelift-codegen-shared v0.91.1
│   │   ├── cranelift-egraph v0.91.1
│   │   │   ├── cranelift-entity v0.91.1
│   │   │   ├── fxhash v0.2.1
│   │   │   │   └── byteorder v1.4.3
│   │   │   ├── hashbrown v0.12.3 (*)
│   │   │   ├── indexmap v1.9.3 (*)
│   │   │   ├── log v0.4.17
│   │   │   │   └── cfg-if v1.0.0
│   │   │   └── smallvec v1.10.0
│   │   ├── cranelift-entity v0.91.1
│   │   ├── gimli v0.26.2
│   │   │   ├── fallible-iterator v0.2.0
│   │   │   ├── indexmap v1.9.3 (*)
│   │   │   └── stable_deref_trait v1.2.0
│   │   ├── log v0.4.17 (*)
│   │   ├── regalloc2 v0.5.1
│   │   │   ├── fxhash v0.2.1 (*)
│   │   │   ├── log v0.4.17 (*)
│   │   │   ├── slice-group-by v0.3.0
│   │   │   └── smallvec v1.10.0
│   │   ├── smallvec v1.10.0
│   │   └── target-lexicon v0.12.6
│   │   [build-dependencies]
│   │   ├── cranelift-codegen-meta v0.91.1
│   │   │   └── cranelift-codegen-shared v0.91.1
│   │   └── cranelift-isle v0.91.1
│   ├── cranelift-entity v0.91.1
│   ├── cranelift-frontend v0.91.1
│   │   ├── cranelift-codegen v0.91.1 (*)
│   │   ├── log v0.4.17 (*)
│   │   ├── smallvec v1.10.0
│   │   └── target-lexicon v0.12.6
│   ├── gimli v0.26.2 (*)
│   ├── more-asserts v0.2.2
│   ├── rayon v1.7.0 (*)
│   ├── smallvec v1.10.0
│   ├── target-lexicon v0.12.6
│   ├── tracing v0.1.37
│   │   ├── cfg-if v1.0.0
│   │   ├── log v0.4.17 (*)
│   │   ├── pin-project-lite v0.2.9
│   │   ├── tracing-attributes v0.1.23 (proc-macro)
│   │   │   ├── proc-macro2 v1.0.56 (*)
│   │   │   ├── quote v1.0.26 (*)
│   │   │   └── syn v1.0.109 (*)
│   │   └── tracing-core v0.1.30
│   │       └── once_cell v1.17.1
│   ├── wasmer-compiler v3.2.0 (*)
│   └── wasmer-types v3.2.0 (*)
├── wasmer-derive v3.2.0 (proc-macro)
│   ├── proc-macro-error v1.0.4
│   │   ├── proc-macro-error-attr v1.0.4 (proc-macro)
│   │   │   ├── proc-macro2 v1.0.56 (*)
│   │   │   └── quote v1.0.26 (*)
│   │   │   [build-dependencies]
│   │   │   └── version_check v0.9.4
│   │   ├── proc-macro2 v1.0.56 (*)
│   │   ├── quote v1.0.26 (*)
│   │   └── syn v1.0.109 (*)
│   │   [build-dependencies]
│   │   └── version_check v0.9.4
│   ├── proc-macro2 v1.0.56 (*)
│   ├── quote v1.0.26 (*)
│   └── syn v1.0.109 (*)
├── wasmer-types v3.2.0 (*)
├── wasmer-vm v3.2.0 (*)
└── wat v1.0.61
    └── wast v55.0.0
        ├── leb128 v0.2.5
        ├── memchr v2.5.0
        ├── unicode-width v0.1.10
        └── wasm-encoder v0.25.0
            └── leb128 v0.2.5

rustc -vV:

rustc 1.70.0-nightly (da7c50c08 2023-03-19)
binary: rustc
commit-hash: da7c50c089d5db2d3ebaf227fe075bb1346bfaec
commit-date: 2023-03-19
host: aarch64-apple-darwin
release: 1.70.0-nightly
LLVM version: 15.0.7

Steps to reproduce

1. git clone https://github.com/await-ovo/reproduce-swc-plugins-crash-issue
2. cd reproduce-swc-plugins-crash-issue
3. pnpm i
4. RUST_BACKTRACE=1 node ./index.mjs:

Note that if you run it once again, it works. It happens again if you delete the fs cache (.swc directory)

Expected behavior

It should not crash.

Actual behavior

It crashes on the first invocation.

Additional context

The author of the SWC issue uses

  System:
    OS: macOS 11.7.4
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
    Memory: 42.06 MB / 16.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 16.15.1 - ~/.nvm/versions/node/v16.15.1/bin/node
    npm: 8.11.0 - ~/.nvm/versions/node/v16.15.1/bin/npm
  Managers:
    Cargo: 1.68.2 - ~/.cargo/bin/cargo
  Utilities:
    Make: 3.81 - /usr/bin/make
    GCC: 4.2.1 - /usr/bin/gcc
    Clang: 13.0.0 - /usr/bin/clang
  Languages:
    Rust: 1.68.2 - /Users/await-ovo/.cargo/bin/rustc

Also, this only happens if someone specify multiple plugins.

[kdy1]

I'll try reproducing this from rust code..

[await-ovo]

Did this related to #2434 ?

[kdy1]

I don't think so, because it works after the first run (compilation)

[ohazda]

@theduke Hey, if you don't mind me asking if there is a chance that this can be fixed before 4.0? Lots of libraries are migrating to SWC and this bug might block some of the migrations. No pressure tho, totally understand that there is more important stuff to work on, cheers :)

[ptitSeb]

So, I reproduce the issue. I only reproduce it on Mac M1, not on Linux x64. I suspect this is the same issue as before. Is wasmer built with LTO there? That was the cause of the issue earlier.

[shouze]

@ptitSeb I reproduce on Linux x64 on my side with styled components, superjson & lingui plugins.

[ptitSeb]

Can you detail of I can reproduce the issue localy @shouze ?

[shouze]

@ptitSeb I'm not sure it can help unless I provide a minimal/simplified reproduction nextjs repo based on our nextjs app. I can just say that I cannot upgrade nextjs at the moment because of that on our app and that I have exactly the same wasmer bug.

[ptitSeb]

How do you build wasmer on you side? Can you remove some build optimisation to see if it "fixes" the issue?

[shouze]

Unfortunately I don't build wasmer by myself, looks like it is a dependency at some point of swc so @kdy1 can probably say more about that as the main maintainer of the project.

[webmaster128]

This is how Wasmer is integrated in the project: https://github.com/swc-project/swc/tree/v1.3.52/crates/swc_plugin_runner.

---

At CosmWasm we are suffering the same panic after we upgraded from Wasmer 2 to Wasmer 3. We can reproduce it as follows:

• Checkout https://github.com/CosmWasm/cosmwasm at 6a5c3bcf77ea4 (current main)
• Run while (cd packages/vm && cargo test) ; do :; done ; echo "Result from $(date)" and wait a bit
• One test thread shows "assertion failed: prev.start > max" and the others will be poisoned

We observe the error on M1 Macs as well as Windows. We'll test Linux more now but it seems less or not affected.

Full Rust stack trace: CosmWasm/cosmwasm#1700

[webmaster128]

I am running the same tests with RUST_TEST_THREADS=1 for hours now and can't reproduce the issue anymore. So it seems like multithreading is needed to trigger it. That might also explain why some 10 core MacBook has higher chances of hitting it than a system with less cores.

[chipshort]

I have an almost minimal reproduction for this. This should make it much easier to look into, as it triggers it very quickly and reliably (at least on my M1 Macbook).
It seems to me like it has to do with cloning the engine, but I might be wrong.
Would be great if someone from the wasmer team could look into this.

[ptitSeb]

@kdy1 @chipshort I have analyzed this and finaly understood the root cause of the issue.

First, there is a PR not yet merge that improved EngineInner multi-thread robustness here: #4011

But trying this on your sample you'll see it doesn't fix the issue.

That is because the sample create a new Engine for every thread.
If you create only 1 engine and use .clone() on each thread, the sample will be stable.

The issue is that, because each thread use a different engine instance, there is some inconsticencies beetwen code memory (some mmap memory) and the GlobalFrameInfo (a global structure registering all code memory used). When the Engine is cloned, all the thread will create it's own Module with the same EngineInner, allocating memory and registering it in a sequence garantied by mutex.
If Engines are different, creating the Module and registering it will also be done in a safe way. But the issue is if Engine get dropped at the same time (which is happening here): then some mmap memory will get freed while the GlobalFrameInfo has not yet unregistered it, and a new Module might get created on that just freed mmap address, and registered while the other is still registered, and you get the assertion in that case.

Getting a fix for this doesn't seems trivial, and might have some performances issues.

The workaround is to use a single Engine and clone it instead of creating a now one. Do you need 1 engine per thread on your usecase?

[kdy1]

cc @kwonoj

[ptitSeb]

Ok, I found a fix for the issue, you can now use and dispose multiple engine at once with PR #4011 . I still think you should reuse the same engine, it seems more ressouce efficient (unless each one has really different settings).

[webmaster128]

I still think you should reuse the same engine, it seems more ressouce efficient (unless each one has really different settings).

Do you suggest one engine per Wasm module or just one engine globally for everything in the process? Since Engine owns the tunables this would limit users to one set of tunables.

[ptitSeb]

I still think you should reuse the same engine, it seems more ressouce efficient (unless each one has really different settings).

Do you suggest one engine per Wasm module or just one engine globally for everything in the process? Since Engine owns the tunables this would limit users to one set of tunables.

True. I would say recycle Engine as much as possible.

[webmaster128]

We discovered another reason why we can't reuse the Engine easily. The metering middleware says it cannot be used from multiple modules

Metering::transform_module_info: Attempting to use a Metering middleware from multiple modules.

I.e. we need to use multiple engines with one compiler and one metering middleware each. Are we doing something wrong here?

[ptitSeb]

Ah, yes, metering. It makes sense to have different metering if you want to mesure different things.