Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test Suggestion: Add Fuzzing Tests #780

Open
Mossaka opened this issue Dec 1, 2023 · 1 comment
Open

Test Suggestion: Add Fuzzing Tests #780

Mossaka opened this issue Dec 1, 2023 · 1 comment
Labels
help wanted Extra attention is needed

Comments

@Mossaka
Copy link
Member

Mossaka commented Dec 1, 2023

Fuzz testing is testing mechanism that involves providing pseudo-random data as input into the generators to find correctness issues. There are multiple Bytecode Alliance projects that heavily leverage fuzz testing to improve code quality and find security and correctness issues (e.g. wasmtime and wasm-tools). So far, there is no fuzz testing existing in this repo. Part of the reasons applying fuzz testing against wit-bindgen repo is to raise the bar for generator code quality across multiple languages.

The goal of this issue is to suggest a fuzz framework adding to this repo. It could be as simply as feeding generated valid WIT packages to each wit-bindgen generator and check if the generated code are buildable. As the wit-bindgen-go maintainer, I hope there is a continuous fuzzing running in the background and invokes TinyGo compiler to compile generated WIT bindings. It could go as difficult as making sure the generated code is "correct" - the compiled Wasm modules / components are correct to our expectations.

To get started, I propose to use wit-smith and libfuzzer-sys crate and cargo-fuzz tool for fuzz testing.

@Mossaka Mossaka added the help wanted Extra attention is needed label Dec 1, 2023
@alexcrichton
Copy link
Member

I think this is a great idea myself. There's two main caveats I'd caution for anyone interested in picking up this work:

  • Fuzzing throughput is likely to be relatively low since all validation of generated code currently goes through spawned processes (e.g. execute clang for C or rustc for Rust code)
  • Fuzzing is likely to turn up a lot of little edge cases the current generators don't handle, so it'd be good to be prepared for some work on the generators themselves too

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants