runtime: wasi preview2

[zshipko]

Now that wasi preview2 is stable, we should start looking into using wasmtime_wasi::preview2::WasiCtx - from what I can tell there will be some minor breaking changes on our end, particularly around how Extism config values were getting re-exported as environment variables. We would also be able to add support for wasi sockets without additional configuration by reusing the allowed hosts list.

preview2 also enables more advanced permissions for files/directories, we will need to think about how to integrate that information into the manifest.

[nilslice]

Curious about support in our other host runtimes too...

[YourTechBud]

Hi. Absolutely love this project.

This would be a great feature to have. Especially the WebAssembly component model if possible. Wasmtime already has it: https://component-model.bytecodealliance.org/introduction.html

[nilslice]

Thanks @YourTechBud! We would like to see more maturity and runtime support from the component model before we bring it into Extism.

unfortunately, it's far too limited to seriously consider given the much greater language support Extism has today.

I'm curious how you would like to use components in Extism though... care to share some ideas?

[YourTechBud]

Thanks for the quick reply. Let me preface the reply by saying that I'm fairly new to this ecosystem and have limited experience here.

I am actually drawing inspiration from Wasmcloud and Fermyon Spin. The idea which really stuck with me was the notion of contract driven plugins by means of the WIT file.

The fact that i can generate boilerplate trait definitions from the WIT file eliminates the need for me to make my own wrapper to implement things like that. Also, the host sdk seems a lot cleaner as well since the host would have to implement a similar trait when passing host functions.

Lastly, the nature of Wasmtime allows all plugins calls to be async. Also if a plugin calls a host function, that just makes wasmtime switch to the host's stack and continue from there. Not sure if I'm able to explain this right. Here's my ref: bytecodealliance/wasmtime#3638
This last one might have nothing to do with Wasm components... Probably.

[nilslice]

Thanks! I'd still like to know a bit more of actual use cases for bringing component support to Extism. To me it's a solution looking for a problem.

The complication with all of this that you stated, is that it's precisely limited to wasmtime.

We only use wasmtime for some of the Host SDKs that Extism supports. We use Wazero for Go, and the browser runtimes for web JS, V8 in Node/Deno, and JSC for Bun. Using wasmtime is not a guarantee we make either - we may decide to pull it out of Extism to use another wasm engine altogether.

If the component model matures to the point where its implemented natively in browsers, and actually becomes a real standard (not one that is desperately trying to force its way into standardization) it will be much easier for us to consider supporting it.

I don't take much inspiration from the serverless/cloud + wasm products - there isn't enough overlap with what we are trying to accomplish. It would be nice in some cases to have async support across the plugin call boundary, but until we can bring this kind of experience across all the runtimes and languages we support, we won't do it.

Extism's goal is to make all software programmable/extensible - which also means that plugins should be portable & run in every extism host (assuming host functions exist where needed), and that doesn't seem like it's the goal of the component model - at least not now.

[nilslice]

I'd do some more research about the background of the component model and its own goals, referencing some of the difficulties with it in the ecosystem:

https://www.assemblyscript.org/standards-objections.html#component-model-2022-09

[YourTechBud]

Ah. That does make sense.

My usecase is primarily allowing my users to run certain guest functions (which essentially add or modify metadata fields) basis certain events in my system. Each function may require access to some external systems (example a db). I do not want to allow my plugins to access my systems directly but instead through a bunch of hosts functions.

The major requirement is making sure all plugins closely follow the correct signatures.

Having understood that one of extism's goals is being runtime agnostic, i guess it wouldn't make sense doing such a overhaul for a seemingly tiny feature which I can technically implement myself in a wrapper library.

I primarily work with rust btw.

[nilslice]

We're tracking things closely and will always do what's best for users -- so it's possible that components come to Extism in the future. But it's definitely not happening now or soon.

One of the benefits of Extism for your use case is that the functions all have the same signature. Instead of taking arguments, plugins load input and set output.

Host Functions can have more complex signatures though, and Extism's Rust SDK and PDK have really good support for generating nice types around lower level wasm values - so Rust is a great language for Extism use. But it goes much further beyond Rust too!

We have unit testing support for your or user plugins (to assert behavior & performance):

https://dylibso.com/blog/testing-extism-plugins/ (these can be written in Rust, Go or TypeScript - and test plugins written in any PDK language).

We also build tools to do further validation and introspection of wasm modules, such as Modsurfer:

• https://modsurfer.dylibso.com
• CLI: https://github.com/dylibso/modsurfer

You'll find there is a lot of support for core wasm in our ecosystem - please also join us on Discord where we regularly chat with our community and collaborate on projects: https://extism.org/discord

[YourTechBud]

This sounds really good. Thanks. I totally understand and even agree with where you are coming from.

[nilslice]

Looking forward to seeing what you build (using Extism or not!)

Please keep in touch

[aschrijver]

https://www.assemblyscript.org/standards-objections.html#component-model-2022-09

Given how fast the Wasm ecosystem has moved, I'd consider that resource to possibly be well outdated. What my consideration for choosing Extism would be, is the extent to which I will get locked-in by custom API's and other mechanisms unique to Extism.

Above @YourTechBud mentioned Wasmcloud who recently got their v1.0 release which is completely WASI Preview 2 compliant. One of the selling points is the creation of "vendor-neutral components". In the recent Cloud Native WASM Day conference Taylor Thomas of Cosmonic outlines how this works. The vid is half an hour and truly a recommended watch:

Choosing Customizability and Distributing Dependencies with Wasm Components - Taylor Thomas

In the demo Taylor live-codes a component that runs unmodified against Wasmcloud as well as Wasmtime (or any other compliant runtime):

[nilslice]

Given how fast the Wasm ecosystem has moved, I'd consider that resource to possibly be well outdated.

I disagree, but appreciate your interest here -- we occasionally consider component model support, and while we'd like to make it happen when it's ready, there is no indication that this is the case in any near-term future. We want to make Wasm as portable as possible, and components are still very far from that. Ideologically, all of the things listed in that resource are still very much in place.

What my consideration for choosing Extism would be, is the extent to which I will get locked-in by custom API's and other mechanisms unique to Extism.

You're going to be locked in to a degree no matter what you choose, unless you pick core modules and handle everything on your own (which is totally possible!). There is a non-trivial amount of work though to handle module linking, guest/host interop, allocation/memory management, etc. So at some level, there are choices you must make to let some other abstraction handle some of those things for you.

In the most basic case, consider having to convert your internal data structures to WIT. You used to only need JSON or Protobuf, and now you've locked yourself in to a new format. Supporters will claim that WIT is a standard, but standardized by whom?

With the component model, you're locked in to a very narrow set of runtimes that support components. I'm not holding my breath any longer -- after waiting for years for things to materialize.

For cloud applications, it might have some benefits, but we're not specifying or focusing on making Extism a serverless / FaaS optimized library. Extism doesn't care where you run it, what kind of data you bring, which language you're embedded in, etc.