Feat: add Docker+wasm examples (docker#309)

* Add a Docker+wasm sample application featuring a WasmEdge-based microservice, a MySQL database and an Nginx web server for frontend UI files. Signed-off-by: Michael Yuan <michael@secondstate.io> * Add a logo to indicate Docker+wasm compatibility. Add project descriptions to README. Signed-off-by: Michael Yuan <michael@secondstate.io> * Add the example for WasmEdge + Kafka / Redpanda + MySQL application to take messages from a queue and save into a database table. Signed-off-by: Michael Yuan <michael@secondstate.io> * Add a SVG icon to indicate Docker + Wasm req Signed-off-by: Michael Yuan <michael@michaelyuan.com> Signed-off-by: Michael Yuan <michael@michaelyuan.com> * Update the docker compose files for the new Docker Desktop release Signed-off-by: Michael Yuan <michael@secondstate.io> * Use the correct platform to be compatible with Docker Desktop 4.15 Signed-off-by: Michael Yuan <michael@secondstate.io> * Update README.md Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> Signed-off-by: Michael Yuan <michael@michaelyuan.com> * Update wasmedge-kafka-mysql/README.md Signed-off-by: Michael Yuan <michael@michaelyuan.com> Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> Signed-off-by: Michael Yuan <michael@michaelyuan.com> * Update wasmedge-kafka-mysql/README.md Signed-off-by: Michael Yuan <michael@michaelyuan.com> Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> Signed-off-by: Michael Yuan <michael@michaelyuan.com> * Update wasmedge-kafka-mysql/etl/Dockerfile Signed-off-by: Michael Yuan <michael@michaelyuan.com> Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> Signed-off-by: Michael Yuan <michael@michaelyuan.com> * Update wasmedge-mysql-nginx/README.md Signed-off-by: Michael Yuan <michael@michaelyuan.com> Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> Signed-off-by: Michael Yuan <michael@michaelyuan.com> * Update wasmedge-mysql-nginx/README.md Signed-off-by: Michael Yuan <michael@michaelyuan.com> Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> Signed-off-by: Michael Yuan <michael@michaelyuan.com> * Update wasmedge-mysql-nginx/README.md Signed-off-by: Michael Yuan <michael@michaelyuan.com> Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> Signed-off-by: Michael Yuan <michael@michaelyuan.com> * Update wasmedge-mysql-nginx/README.md Signed-off-by: Michael Yuan <michael@michaelyuan.com> Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> Signed-off-by: Michael Yuan <michael@michaelyuan.com> * Update wasmedge-mysql-nginx/README.md Signed-off-by: Michael Yuan <michael@michaelyuan.com> Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> Signed-off-by: Michael Yuan <michael@michaelyuan.com> * Update wasmedge-mysql-nginx/README.md Signed-off-by: Michael Yuan <michael@michaelyuan.com> Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> Signed-off-by: Michael Yuan <michael@michaelyuan.com> * Update wasmedge-mysql-nginx/README.md Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> Signed-off-by: Michael Yuan <michael@michaelyuan.com> * Update wasmedge-mysql-nginx/README.md Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> Signed-off-by: Michael Yuan <michael@michaelyuan.com> * Update wasmedge-mysql-nginx/README.md Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> Signed-off-by: Michael Yuan <michael@michaelyuan.com> * Update wasmedge-mysql-nginx/README.md Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> Signed-off-by: Michael Yuan <michael@michaelyuan.com> * Change the Nginx port to the default non-privileged 8090 Signed-off-by: Michael Yuan <michael@secondstate.io> * My apologies. Need to correct the syntax for the Nginx port 8090. Signed-off-by: Michael Yuan <michael@secondstate.io> * Remove commented lines Signed-off-by: Michael Yuan <michael@secondstate.io> * Change wasi/wasm32 to wasi/wasm to conform with the latest spec Signed-off-by: Michael Yuan <michael@secondstate.io> * Update README.md Co-authored-by: Michael Irwin <mikesir87@gmail.com> Signed-off-by: Michael Yuan <michael@michaelyuan.com> Signed-off-by: Michael Yuan <michael@secondstate.io> Signed-off-by: Michael Yuan <michael@michaelyuan.com> Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> Co-authored-by: Michael Irwin <mikesir87@gmail.com>
ghouldaemon · Jan 9, 2023 · e6b1d27 · e6b1d27
1 parent 6f15838
commit e6b1d27
Show file tree

Hide file tree

Showing 20 changed files with 1,057 additions and 0 deletions.
diff --git a/README.md b/README.md
@@ -20,6 +20,8 @@ These samples provide a starting point for how to integrate different services u
 
 <a href="https://docs.docker.com/desktop/dev-environments/"><img src="icon_devenvs.svg" alt="Use with Docker Dev Environments" height="30" align="top"/></a> Icon indicates Sample is compatible with [Docker Dev Environments](https://docs.docker.com/desktop/dev-environments/) in Docker Desktop version 4.10 or later.
 
+<a href="https://docs.docker.com/desktop/wasm/"><img src="icon_wasm.svg" alt="Docker + wasm" height="30" align="top"/></a> Icon indicates Sample is compatible with [Docker+Wasm](https://docs.docker.com/desktop/wasm/).
+
 - [`ASP.NET / MS-SQL`](aspnet-mssql) - Sample ASP.NET core application
 with MS SQL server database.
 - [`Elasticsearch / Logstash / Kibana`](elasticsearch-logstash-kibana) - Sample Elasticsearch, Logstash, and Kibana stack.
@@ -49,6 +51,8 @@ application with a Rust backend and a Postgres database.&nbsp;<a href="react-rus
 - [`React / Nginx`](react-nginx) - Sample React application with Nginx.&nbsp;<a href="react-nginx"><img src="icon_devenvs.svg" alt="Use with Docker Dev Environments" height="30" align="top"/></a>
 - [`Spring / PostgreSQL`](spring-postgres) - Sample Java application
 with Spring framework and a Postgres database.&nbsp;<a href="spring-postgres"><img src="icon_devenvs.svg" alt="Use with Docker Dev Environments" height="30" align="top"/></a>
+- [`WasmEdge / MySQL / Nginx`](wasmedge-mysql-nginx) - Sample Wasm-based web application with a static HTML frontend, using a MySQL (MariaDB) database. The frontend connects to a Wasm microservice written in Rust, that runs using the WasmEdge runtime.&nbsp;<a href="wasmedge-mysql-nginx"><img src="icon_wasm.svg" alt="Compatible with Docker+wasm" height="30" align="top"/></a>
+- [`WasmEdge / Kafka / MySQL`](wasmedge-kafka-mysql) - Sample Wasm-based microservice that subscribes to a Kafka (Redpanda) queue topic, and transforms and saves any incoming message into a MySQL (MariaDB) database.&nbsp;<a href="wasmedge-kafka-mysql"><img src="icon_wasm.svg" alt="Compatible with Docker+wasm" height="30" align="top"/></a>
 
 ## Single service samples
 

diff --git a/icon_wasm.svg b/icon_wasm.svg
diff --git a/wasmedge-kafka-mysql/.docker/docker-compose.yml b/wasmedge-kafka-mysql/.docker/docker-compose.yml
@@ -0,0 +1,36 @@
+services:
+  redpanda:
+    image: docker.redpanda.com/vectorized/redpanda:v22.2.2
+    command:
+      - redpanda start
+      - --smp 1
+      - --overprovisioned
+      - --node-id 0
+      - --kafka-addr PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
+      - --advertise-kafka-addr PLAINTEXT://redpanda:29092,OUTSIDE://redpanda:9092
+      - --pandaproxy-addr 0.0.0.0:8082
+      - --advertise-pandaproxy-addr localhost:8082
+    ports:
+      - 8081:8081
+      - 8082:8082
+      - 9092:9092
+      - 9644:9644
+      - 29092:29092
+    volumes:
+      - ./kafka:/app
+  etl:
+    image: etl-kafka
+    platform: wasi/wasm
+    build:
+      context: etl
+    environment:
+      DATABASE_URL: mysql://root:whalehello@db:3306/mysql
+      KAFKA_URL: kafka://redpanda:9092/order
+      RUST_BACKTRACE: full
+      RUST_LOG: info
+    restart: unless-stopped
+    runtime: io.containerd.wasmedge.v1
+  db:
+    image: mariadb:10.9
+    environment:
+      MYSQL_ROOT_PASSWORD: whalehello
diff --git a/wasmedge-kafka-mysql/README.md b/wasmedge-kafka-mysql/README.md
@@ -0,0 +1,117 @@
+# Compose sample application
+
+![Compatible with Docker+Wasm](../icon_wasm.svg)
+
+This sample demonstrates a WebAssembly (Wasm) microservice written in Rust. It subscribes to a Kafka queue topic on a Redpanda server, and then transforms and saves each message into a MySQL (MariaDB) database table. The microservice is compiled into Wasm and runs in the WasmEdge runtime, which is a secure and lightweight alternative to natively compiled Rust apps in Linux containers.
+
+## Use with Docker Development Environments
+
+You will need a version of Docker Desktop or Docker CLI with Wasm support.
+
+* [Install Docker Desktop + Wasm (Beta)](https://docs.docker.com/desktop/wasm/)
+* [Install Docker CLI + Wasm](https://github.com/chris-crone/wasm-day-na-22/tree/main/server)
+
+## WasmEdge server with Redpanda and MySQL database
+
+Project structure:
+
+```
+.
++-- compose.yml
+|-- etl
+    |-- Dockerfile
+    |-- Cargo.toml
+    +-- src
+        |-- main.rs
+|-- kafka
+    |-- order.json
+|-- db
+    |-- db-password.txt
+```
+
+The [compose.yml](compose.yml) is as follows.
+
+```yaml
+services:
+  redpanda:
+    image: docker.redpanda.com/vectorized/redpanda:v22.2.2
+    command:
+      - redpanda start
+      - --smp 1
+      - --overprovisioned
+      - --node-id 0
+      - --kafka-addr PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
+      - --advertise-kafka-addr PLAINTEXT://redpanda:29092,OUTSIDE://redpanda:9092
+      - --pandaproxy-addr 0.0.0.0:8082
+      - --advertise-pandaproxy-addr localhost:8082
+    ports:
+      - 8081:8081
+      - 8082:8082
+      - 9092:9092
+      - 9644:9644
+      - 29092:29092
+    volumes:
+      - ./kafka:/app
+
+  etl:
+    image: etl-kafka
+    build:
+      context: etl
+      platforms:
+        - wasi/wasm32
+    environment:
+      DATABASE_URL: mysql://root:whalehello@db:3306/mysql
+      KAFKA_URL: kafka://redpanda:9092/order
+      RUST_BACKTRACE: full
+      RUST_LOG: info
+    restart: unless-stopped
+    runtime: io.containerd.wasmedge.v1
+
+  db:
+    image: mariadb:10.9
+    environment:
+      MYSQL_ROOT_PASSWORD: whalehello
+```
+
+The compose file defines an application with three services `redpanda`, `etl` and `db`. The `redpanda` service is a Kafka-compatible messaging server that produces messages in a queue topic. The `etl` service, in the WasmEdge container that subscribes to the queue topic and receives incoming messages. Each incoming message is parsed and stored in the `db` MySQL (MariaDB) database server.
+
+## Deploy with docker compose
+
+```bash
+$ docker compose up -d
+...
+ ⠿ Network wasmedge-kafka-mysql_default       Created                      0.1s
+ ⠿ Container wasmedge-kafka-mysql-redpanda-1  Created                      0.3s
+ ⠿ Container wasmedge-kafka-mysql-etl-1       Created                      0.3s
+ ⠿ Container wasmedge-kafka-mysql-db-1        Created                      0.3s
+```
+
+## Expected result
+
+```bash
+$ docker compose ps
+NAME                              COMMAND                  SERVICE             STATUS              PORTS
+wasmedge-kafka-mysql-db-1         "docker-entrypoint.s…"   db                  running             3306/tcp
+wasmedge-kafka-mysql-etl-1        "kafka.wasm"             etl                 running
+wasmedge-kafka-mysql-redpanda-1   "/entrypoint.sh 'red…"   redpanda            running             0.0.0.0:8081-8082->8081-8082/tcp, :::8081-8082->8081-8082/tcp, 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp, 0.0.0.0:9644->9644/tcp, :::9644->9644/tcp, 0.0.0.0:29092->29092/tcp, :::29092->29092/tcp
+```
+
+After the application starts, 
+log into the Redpanda container and send a message to the queue topic `order` as follows.
+
+```bash
+$ docker compose exec redpanda /bin/bash
+redpanda@1add2615774b:/$ cd /app
+redpanda@1add2615774b:/app$ cat order.json | rpk topic produce order
+Produced to partition 0 at offset 0 with timestamp 1667922788523.
+```
+
+To see the data in the database container, you can use the following commands.
+
+```bash
+$ docker compose exec db /bin/bash
+root@c97c472db02e:/# mysql -u root -pwhalehello mysql
+mysql> select * from orders;
+... ...
+```
+
diff --git a/wasmedge-kafka-mysql/compose.yml b/wasmedge-kafka-mysql/compose.yml
@@ -0,0 +1,36 @@
+services:
+  redpanda:
+    image: docker.redpanda.com/vectorized/redpanda:v22.2.2
+    command:
+      - redpanda start
+      - --smp 1
+      - --overprovisioned
+      - --node-id 0
+      - --kafka-addr PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
+      - --advertise-kafka-addr PLAINTEXT://redpanda:29092,OUTSIDE://redpanda:9092
+      - --pandaproxy-addr 0.0.0.0:8082
+      - --advertise-pandaproxy-addr localhost:8082
+    ports:
+      - 8081:8081
+      - 8082:8082
+      - 9092:9092
+      - 9644:9644
+      - 29092:29092
+    volumes:
+      - ./kafka:/app
+  etl:
+    image: etl-kafka
+    platform: wasi/wasm
+    build:
+      context: etl
+    environment:
+      DATABASE_URL: mysql://root:whalehello@db:3306/mysql
+      KAFKA_URL: kafka://redpanda:9092/order
+      RUST_BACKTRACE: full
+      RUST_LOG: info
+    restart: unless-stopped
+    runtime: io.containerd.wasmedge.v1
+  db:
+    image: mariadb:10.9
+    environment:
+      MYSQL_ROOT_PASSWORD: whalehello
diff --git a/wasmedge-kafka-mysql/db/db-password.txt b/wasmedge-kafka-mysql/db/db-password.txt
@@ -0,0 +1 @@
+whalehello
diff --git a/wasmedge-kafka-mysql/etl/Cargo.toml b/wasmedge-kafka-mysql/etl/Cargo.toml
@@ -0,0 +1,17 @@
+[package]
+name = "kafka"
+version = "0.1.0"
+edition = "2021"
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]
+anyhow = "1.0.65"
+mega_etl = {git = "https://github.com/second-state/MEGA.git"}
+tokio_wasi = {version = '1.21', features = ["rt", "macros"]}
+env_logger = "0.9"
+log  = "0.4"
+serde = { version = "1.0", features = ["derive"] }
+serde_json = "1.0"
+http_req_wasi  = "0.10"
+lazy_static = "1.4.0"
diff --git a/wasmedge-kafka-mysql/etl/Dockerfile b/wasmedge-kafka-mysql/etl/Dockerfile
@@ -0,0 +1,27 @@
+# syntax=docker/dockerfile:1
+FROM --platform=$BUILDPLATFORM rust:1.64 AS buildbase
+RUN <<EOT bash
+    set -ex
+    apt-get update
+    apt-get install -y \
+        git \
+        clang
+    rustup target add wasm32-wasi
+EOT
+# This line installs WasmEdge including the AOT compiler
+RUN curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash
+
+FROM buildbase AS build
+COPY Cargo.toml .
+COPY src ./src 
+# Build the Wasm binary
+RUN --mount=type=cache,target=/usr/local/cargo/git/db \
+    --mount=type=cache,target=/usr/local/cargo/registry/cache \
+    --mount=type=cache,target=/usr/local/cargo/registry/index \
+    cargo build --target wasm32-wasi --release
+# This line builds the AOT Wasm binary
+RUN /root/.wasmedge/bin/wasmedgec target/wasm32-wasi/release/kafka.wasm kafka.wasm
+
+FROM scratch
+ENTRYPOINT [ "kafka.wasm" ]
+COPY --link --from=build /kafka.wasm /kafka.wasm
diff --git a/wasmedge-kafka-mysql/etl/src/main.rs b/wasmedge-kafka-mysql/etl/src/main.rs
@@ -0,0 +1,58 @@
+use mega_etl::{async_trait, Pipe, Transformer, TransformerError, TransformerResult};
+
+use serde::{Deserialize, Serialize};
+#[derive(Serialize, Deserialize, Debug)]
+struct Order {
+    order_id: i32,
+    product_id: i32,
+    quantity: i32,
+    amount: f32,
+    shipping: f32,
+    tax: f32,
+    shipping_address: String,
+}
+
+#[async_trait]
+impl Transformer for Order {
+    async fn transform(inbound_data: &Vec<u8>) -> TransformerResult<Vec<String>> {
+        let s = std::str::from_utf8(&inbound_data)
+            .map_err(|e| TransformerError::Custom(e.to_string()))?;
+        let order: Order = serde_json::from_str(String::from(s).as_str())
+            .map_err(|e| TransformerError::Custom(e.to_string()))?;
+        log::info!("{:?}", &order);
+        let mut ret = vec![];
+        let sql_string = format!(
+            r"INSERT INTO orders VALUES ({:?}, {:?}, {:?}, {:?}, {:?}, {:?}, {:?}, CURRENT_TIMESTAMP);",
+            order.order_id,
+            order.product_id,
+            order.quantity,
+            order.amount,
+            order.shipping,
+            order.tax,
+            order.shipping_address,
+        );
+        dbg!(sql_string.clone());
+        ret.push(sql_string);
+        Ok(ret)
+    }
+
+    async fn init() -> TransformerResult<String> {
+        Ok(String::from(
+            r"CREATE TABLE IF NOT EXISTS orders (order_id INT, product_id INT, quantity INT, amount FLOAT, shipping FLOAT, tax FLOAT, shipping_address VARCHAR(50), date_registered TIMESTAMP DEFAULT CURRENT_TIMESTAMP);",
+        ))
+    }
+}
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> anyhow::Result<()> {
+    env_logger::init();
+
+    // can use builder later
+    let database_uri = std::env::var("DATABASE_URL")?;
+    let kafka_uri = std::env::var("KAFKA_URL")?;
+    let mut pipe = Pipe::new(database_uri, kafka_uri).await;
+
+    // This is async because this calls the async transform() function in Order
+    pipe.start::<Order>().await?;
+    Ok(())
+}
diff --git a/wasmedge-kafka-mysql/kafka/order.json b/wasmedge-kafka-mysql/kafka/order.json
@@ -0,0 +1 @@
+{"order_id": 1,"product_id": 12,"quantity": 2,"amount": 56.0,"shipping": 15.0,"tax": 2.0,"shipping_address": "Mataderos 2312"}
diff --git a/wasmedge-mysql-nginx/.docker/docker-compose.yml b/wasmedge-mysql-nginx/.docker/docker-compose.yml
@@ -0,0 +1,25 @@
+services:
+  frontend:
+    image: nginx:alpine
+    ports:
+      - 8090:80
+    volumes:
+      - ./frontend:/usr/share/nginx/html
+
+  backend:
+    image: demo-microservice
+    platform: wasi/wasm
+    build:
+      context: backend/
+    ports:
+      - 8080:8080
+    environment:
+      DATABASE_URL: mysql://root:whalehello@db:3306/mysql
+      RUST_BACKTRACE: full
+    restart: unless-stopped
+    runtime: io.containerd.wasmedge.v1
+
+  db:
+    image: mariadb:10.9
+    environment:
+      MYSQL_ROOT_PASSWORD: whalehello