Skip to main content

Wasm functions

Beta

The following feature is not yet stable.

The WasmRuntime enables one to use any langauge/ecosystem with a toolchain capable of producing wasm artifacts to author custom functions. Rust is one such a language and has shaped up to be the hotspot of development in the wasm ecosystem (The Metatype itself platform has many rusty parts). In this guide, we'll see how to set up a workflow for using Rust for our custom functions.

Tooling​

We need to install several programs to be able to produce the components. The following checklist contains links to get you started:

  1. Rust compiler toolchain: this guide assumes moderate familiartiy of development with rust and won't spend many words on how to get it functional. In any case, you can get started with rust here.
  2. wasm32-unknown-unknown target for rustc: This is the backend that rustc uses to produce wasi compatible wasm components. If you're using rustup to manage your toolchain, Cargo will automatically install the target when you're building.
  3. wasm-tools: this is the swiss army knife for working with wasm artifacts, courtesy of the Bytecode Alliance. Installation instructions can be found here.

Typegraph​

The WasmRuntime currently comes in two flavours that are both based on the wasm component spec. This guide focues on the wire flavour, where your component is expected to implement a standard WIT interface that's designed around a simple Json based RPC wire format. Thankfully, all of that boilerplate is easy to automate away and that's exactly what we'll do using metagen to generate the binding code.

Before anything though, we need to author the typegraph:

Loading...

Note that the WasmRuntime constructor mentions a non-existent wasm file on disk. This won't be a problem for the metagen generators but we'll need to produce the artifact before we deploy the typegraph. We'll see what buliding the artifact entails in just a minute.

Metagen​

We can now tackle the boilerplate. Metagen bundles the mdk_rust generator which can generate all the glue code along with Rust types that correspond to our typegraph types. Let's configure a metagen target in our configuration file to get just that done.

metagen:
  targets:
    metagen_rs:
      # this is the generator we're interested in
      - generator: mdk_rust
        # the location where to put the generated files
        path: ./metagen/rs/
        # the path to our typegraph
        typegraph_path: ./metagen-rs.ts

The configuration file is read by the meta CLI which also bundles the metagen suite. This means we can invoke the target from the command line like so:

meta gen metagen_rs

This should give us the following files:

❯ lsd --tree metagen/rs/
 rs
├──  Cargo.toml
├──  lib.rs
└──  mdk.rs

By default, the mdk_rust generator outputs all the necessary files required to build our wasm file. This includes the Cargo.toml manifest for our Rust crate.

package.name = "metagen_rs_mdk"
package.edition = "2021"
package.version = "0.0.1"

# we need to use a specific library crate type to build
# wasm components in rust
[lib]
path = "lib.rs"
crate-type = ["cdylib", "rlib"]

# the following dependencies are used by the generated code
[dependencies]
anyhow = "1" # error handling
serde = { version = "1", features = ["derive"] } # serialization
serde_json = "1" #json serialization
wit-bindgen = "0.22.0" # wasm component biding

# we set the following flags to minimize code size 
# when buliding in the release mode
# this keeps our wasm files small
[profile.release]
strip = "symbols"
opt-level = "z"

mdk_rust will not overwrite a Cargo.toml file discovered at generation path so you can add other dependencies if need be.

The mdk.rs file contains all the glue code including the typegraph types.

Code generation sample. It's collapsed here as it's for the most part an uninteresting implementation detail.

// This file was @generated by metagen and is intended
// to be generated again on subsequent metagen runs.
#![cfg_attr(rustfmt, rustfmt_skip)]

// gen-static-start
#![allow(dead_code)]

pub mod wit {
    wit_bindgen::generate!({
        pub_export_macro: true,
        
        inline: "package metatype:wit-wire;

interface typegate-wire {
  hostcall: func(op-name: string, json: string) -> result<string, string>;
}

interface mat-wire {
  type json-str = string;

  record mat-info {
    op-name: string,
    mat-title: string,
    mat-hash: string,
    mat-data-json: string,
  }

  record init-args {
    metatype-version: string,
    expected-ops: list<mat-info>
  }

  record init-response {
    ok: bool
  }

  variant init-error {
    version-mismatch(string),
    unexpected-mat(mat-info),
    other(string)
  }

  init: func(args: init-args) -> result<init-response, init-error>;

  record handle-req {
    op-name: string,
    in-json: json-str,
  }

  variant handle-err {
    no-handler,
    in-json-err(string),
    handler-err(string),
  }

  handle: func(req: handle-req) -> result<json-str, handle-err>;
}

world wit-wire {
  import typegate-wire;

  export mat-wire;
}
"
    });
}

use std::cell::RefCell;
use std::collections::HashMap;

use wit::exports::metatype::wit_wire::mat_wire::*;
use wit::metatype::wit_wire::typegate_wire::hostcall;

pub type HandlerFn = Box<dyn Fn(&str, Ctx) -> Result<String, HandleErr>>;

pub struct ErasedHandler {
    mat_id: String,
    mat_trait: String,
    mat_title: String,
    handler_fn: HandlerFn,
}

pub struct MatBuilder {
    handlers: HashMap<String, ErasedHandler>,
}

impl MatBuilder {
    pub fn new() -> Self {
        Self {
            handlers: Default::default(),
        }
    }

    pub fn register_handler(mut self, handler: ErasedHandler) -> Self {
        self.handlers.insert(handler.mat_trait.clone(), handler);
        self
    }
}

pub struct Router {
    handlers: HashMap<String, ErasedHandler>,
}

impl Router {
    pub fn from_builder(builder: MatBuilder) -> Self {
        Self {
            handlers: builder.handlers,
        }
    }

    pub fn init(&self, args: InitArgs) -> Result<InitResponse, InitError> {
        static MT_VERSION: &str = "0.4.10-rc1";
        if args.metatype_version != MT_VERSION {
            return Err(InitError::VersionMismatch(MT_VERSION.into()));
        }
        for info in args.expected_ops {
            let mat_trait = stubs::op_to_trait_name(&info.op_name);
            if !self.handlers.contains_key(mat_trait) {
                return Err(InitError::UnexpectedMat(info));
            }
        }
        Ok(InitResponse { ok: true })
    }

    pub fn handle(&self, req: HandleReq) -> Result<String, HandleErr> {
        let mat_trait = stubs::op_to_trait_name(&req.op_name);
        let Some(handler) = self.handlers.get(mat_trait) else {
            return Err(HandleErr::NoHandler);
        };
        let cx = Ctx {};
        (handler.handler_fn)(&req.in_json, cx)
    }
}

pub type InitCallback = fn() -> anyhow::Result<MatBuilder>;

thread_local! {
    pub static MAT_STATE: RefCell<Router> = panic!("MAT_STATE has not been initialized");
}

pub struct Ctx {}

impl Ctx {
    pub fn gql<O>(
        &self,
        query: &str,
        variables: impl Into<serde_json::Value>,
    ) -> Result<O, GraphqlRunError>
    where
        O: serde::de::DeserializeOwned,
    {
        match hostcall(
            "gql",
            &serde_json::to_string(&serde_json::json!({
                "query": query,
                "variables": variables.into(),
            }))?,
        ) {
            Ok(json) => Ok(serde_json::from_str(&json[..])?),
            Err(json) => Err(GraphqlRunError::HostError(serde_json::from_str(&json)?)),
        }
    }
}

#[derive(Debug)]
pub enum GraphqlRunError {
    JsonError(serde_json::Error),
    HostError(serde_json::Value),
}

impl std::error::Error for GraphqlRunError {}

impl From<serde_json::Error> for GraphqlRunError {
    fn from(value: serde_json::Error) -> Self {
        Self::JsonError(value)
    }
}

impl std::fmt::Display for GraphqlRunError {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        match self {
            GraphqlRunError::JsonError(msg) => write!(f, "json error: {msg}"),
            GraphqlRunError::HostError(serde_json::Value::Object(map))
                if map.contains_key("message") =>
            {
                write!(f, "host error: {}", map["message"])
            }
            GraphqlRunError::HostError(val) => write!(f, "host error: {val:?}"),
        }
    }
}

#[macro_export]
macro_rules! init_mat {
    (hook: $init_hook:expr) => {
        struct MatWireGuest;
        use wit::exports::metatype::wit_wire::mat_wire::*;
        wit::export!(MatWireGuest with_types_in wit);

        #[allow(unused)]
        impl Guest for MatWireGuest {
            fn handle(req: HandleReq) -> Result<String, HandleErr> {
                MAT_STATE.with(|router| {
                    let router = router.borrow();
                    router.handle(req)
                })
            }

            fn init(args: InitArgs) -> Result<InitResponse, InitError> {
                let hook = $init_hook;
                let router = Router::from_builder(hook());
                let resp = router.init(args)?;
                MAT_STATE.set(router);
                Ok(resp)
            }
        }
    };
}
// gen-static-end
use types::*;
pub mod types {
    pub type StringDateTime4 = String;
    pub type StringUri5 = String;
    #[derive(Debug, serde::Serialize, serde::Deserialize)]
    pub struct Idv3 {
        pub title: String,
        pub artist: String,
        #[serde(rename = "releaseTime")]
        pub release_time: StringDateTime4,
        #[serde(rename = "mp3Url")]
        pub mp3_url: StringUri5,
    }
}
pub mod stubs {
    use super::*;
    pub trait RemixTrack: Sized + 'static {
        fn erased(self) -> ErasedHandler {
            ErasedHandler {
                mat_id: "remix_track".into(),
                mat_title: "remix_track".into(),
                mat_trait: "RemixTrack".into(),
                handler_fn: Box::new(move |req, cx| {
                    let req = serde_json::from_str(req)
                        .map_err(|err| HandleErr::InJsonErr(format!("{err}")))?;
                    let res = self
                        .handle(req, cx)
                        .map_err(|err| HandleErr::HandlerErr(format!("{err}")))?;
                    serde_json::to_string(&res)
                        .map_err(|err| HandleErr::HandlerErr(format!("{err}")))
                }),
            }
        }

        fn handle(&self, input: Idv3, cx: Ctx) -> anyhow::Result<Idv3>;
    }
    pub fn op_to_trait_name(op_name: &str) -> &'static str {
        match op_name {
            "remix_track" => "RemixTrack",
            _ => panic!("unrecognized op_name: {op_name}"),
        }
    }
}

When working on the typegraph, we can run metagen again to regenerate this file and get the latest types.

The generator also includes a sample lib.rs entrypoint file for our crate. We'll modify it now to implement our custom function.

mod mdk;
pub use mdk::*;

// the macro sets up all the glue
init_mat! {
    // the hook is expected to return a MatBuilder instance
    hook: || {
        // initialize global stuff here if you need it
        MatBuilder::new()
            // register function handlers here
            // each trait will map to the name of the
            // handler found in the typegraph
            .register_handler(stubs::RemixTrack::erased(MyMat))
    }
}

struct MyMat;

impl stubs::RemixTrack for MyMat {
    fn handle(&self, input: types::Idv3, _cx: Ctx) -> anyhow::Result<types::Idv3> {
        Ok(types::Idv3 {
            title: format!("{} (Remix)", input.title),
            artist: format!("{} + DJ Cloud", input.artist),
            release_time: input.release_time,
            mp3_url: "https://mp3.url/shumba2".to_string(),
        })
    }
}

Building​

We'll now use the rust toolchain and wasm-tools to build the wasm component. This requires multiple commands. It's presented below as a shell script that you can modify from.

# flags to make script execution visible
set -eux

# regenerate code before building
meta gen metagen_rs

# variablize common names
TARGET=wasm32-wasi
CRATE_NAME=metagen_rs_mdk

# build in release mode for smallest sizes
cargo build -p $CRATE_NAME --target $TARGET --release
# use wasm-tools to change wasm file into wasm component
wasm-tools component new \
  # rust builds the wasm file under the name of the crate
  ./target/$TARGET/debug/$CRATE_NAME.wasm \
  -o ./target/rust-component.wasm \

# copy the component to a location that we specified
# in our typegraph
cp ./target/rust-component.wasm ./rust.wasm

Put the shell script into a file like build.sh and execute it with a posix compatible shell like bash. You should now have all the files to deploy your typegraph.

Loading...