Wasm functions
The following feature is not yet stable.
The WasmRuntime
enables one to use any langauge/ecosystem with a toolchain capable of producing wasm artifacts to author custom functions. Rust is one such a language and has shaped up to be the hotspot of development in the wasm ecosystem (The Metatype itself platform has many rusty parts). In this guide, we'll see how to set up a workflow for using Rust for our custom functions.
Tooling
We need to install several programs to be able to produce the components. The following checklist contains links to get you started:
- Rust compiler toolchain: this guide assumes moderate familiartiy of development with rust and won't spend many words on how to get it functional. In any case, you can get started with rust here.
wasm32-unknown-unknown
target for rustc: This is the backend that rustc uses to produce wasi compatible wasm components. If you're usingrustup
to manage your toolchain, Cargo will automatically install the target when you're building.wasm-tools
: this is the swiss army knife for working with wasm artifacts, courtesy of the Bytecode Alliance. Installation instructions can be found here.
Typegraph
The WasmRuntime
currently comes in two flavours that are both based on the wasm component spec. This guide focues on the wire
flavour, where your component is expected to implement a standard WIT interface that's designed around a simple Json based RPC wire format. Thankfully, all of that boilerplate is easy to automate away and that's exactly what we'll do using metagen to generate the binding code.
Before anything though, we need to author the typegraph:
Note that the WasmRuntime
constructor mentions a non-existent wasm file on disk. This won't be a problem for the metagen generators but we'll need to produce the artifact before we deploy the typegraph. We'll see what buliding the artifact entails in just a minute.
Metagen
We can now tackle the boilerplate. Metagen bundles the mdk_rust
generator which can generate all the glue code along with Rust types that correspond to our typegraph types. Let's configure a metagen target in our configuration file to get just that done.
metagen:
targets:
metagen_rs:
# this is the generator we're interested in
- generator: mdk_rust
# the location where to put the generated files
path: ./metagen/rs/
# the path to our typegraph
typegraph_path: ./metagen-rs.ts
The configuration file is read by the meta CLI which also bundles the metagen suite. This means we can invoke the target from the command line like so:
meta gen metagen_rs
This should give us the following files:
❯ lsd --tree metagen/rs/
rs
├── Cargo.toml
├── lib.rs
└── mdk.rs
By default, the mdk_rust
generator outputs all the necessary files required to build our wasm file. This includes the Cargo.toml
manifest for our Rust crate.
package.name = "metagen_rs_mdk"
package.edition = "2021"
package.version = "0.0.1"
# we need to use a specific library crate type to build
# wasm components in rust
[lib]
path = "lib.rs"
crate-type = ["cdylib", "rlib"]
# the following dependencies are used by the generated code
[dependencies]
anyhow = "1" # error handling
serde = { version = "1", features = ["derive"] } # serialization
serde_json = "1" #json serialization
wit-bindgen = "0.22.0" # wasm component biding
# we set the following flags to minimize code size
# when buliding in the release mode
# this keeps our wasm files small
[profile.release]
strip = "symbols"
opt-level = "z"
mdk_rust
will not overwrite a Cargo.toml
file discovered at generation path so you can add other dependencies if need be.
The mdk.rs
file contains all the glue code including the typegraph types.
Code generation sample. It's collapsed here as it's for the most part an
uninteresting implementation detail.
// This file was @generated by metagen and is intended
// to be generated again on subsequent metagen runs.
#![cfg_attr(rustfmt, rustfmt_skip)]
// gen-static-start
#![allow(dead_code)]
pub mod wit {
wit_bindgen::generate!({
pub_export_macro: true,
inline: "package metatype:wit-wire;
interface typegate-wire {
hostcall: func(op-name: string, json: string) -> result<string, string>;
}
interface mat-wire {
type json-str = string;
record mat-info {
op-name: string,
mat-title: string,
mat-hash: string,
mat-data-json: string,
}
record init-args {
metatype-version: string,
expected-ops: list<mat-info>
}
record init-response {
ok: bool
}
variant init-error {
version-mismatch(string),
unexpected-mat(mat-info),
other(string)
}
init: func(args: init-args) -> result<init-response, init-error>;
record handle-req {
op-name: string,
in-json: json-str,
}
variant handle-err {
no-handler,
in-json-err(string),
handler-err(string),
}
handle: func(req: handle-req) -> result<json-str, handle-err>;
}
world wit-wire {
import typegate-wire;
export mat-wire;
}
"
});
}
use std::cell::RefCell;
use std::collections::HashMap;
use wit::exports::metatype::wit_wire::mat_wire::*;
use wit::metatype::wit_wire::typegate_wire::hostcall;
pub type HandlerFn = Box<dyn Fn(&str, Ctx) -> Result<String, HandleErr>>;
pub struct ErasedHandler {
mat_id: String,
mat_trait: String,
mat_title: String,
handler_fn: HandlerFn,
}
pub struct MatBuilder {
handlers: HashMap<String, ErasedHandler>,
}
impl MatBuilder {
pub fn new() -> Self {
Self {
handlers: Default::default(),
}
}
pub fn register_handler(mut self, handler: ErasedHandler) -> Self {
self.handlers.insert(handler.mat_trait.clone(), handler);
self
}
}
pub struct Router {
handlers: HashMap<String, ErasedHandler>,
}
impl Router {
pub fn from_builder(builder: MatBuilder) -> Self {
Self {
handlers: builder.handlers,
}
}
pub fn init(&self, args: InitArgs) -> Result<InitResponse, InitError> {
static MT_VERSION: &str = "0.4.10-rc1";
if args.metatype_version != MT_VERSION {
return Err(InitError::VersionMismatch(MT_VERSION.into()));
}
for info in args.expected_ops {
let mat_trait = stubs::op_to_trait_name(&info.op_name);
if !self.handlers.contains_key(mat_trait) {
return Err(InitError::UnexpectedMat(info));
}
}
Ok(InitResponse { ok: true })
}
pub fn handle(&self, req: HandleReq) -> Result<String, HandleErr> {
let mat_trait = stubs::op_to_trait_name(&req.op_name);
let Some(handler) = self.handlers.get(mat_trait) else {
return Err(HandleErr::NoHandler);
};
let cx = Ctx {};
(handler.handler_fn)(&req.in_json, cx)
}
}
pub type InitCallback = fn() -> anyhow::Result<MatBuilder>;
thread_local! {
pub static MAT_STATE: RefCell<Router> = panic!("MAT_STATE has not been initialized");
}
pub struct Ctx {}
impl Ctx {
pub fn gql<O>(
&self,
query: &str,
variables: impl Into<serde_json::Value>,
) -> Result<O, GraphqlRunError>
where
O: serde::de::DeserializeOwned,
{
match hostcall(
"gql",
&serde_json::to_string(&serde_json::json!({
"query": query,
"variables": variables.into(),
}))?,
) {
Ok(json) => Ok(serde_json::from_str(&json[..])?),
Err(json) => Err(GraphqlRunError::HostError(serde_json::from_str(&json)?)),
}
}
}
#[derive(Debug)]
pub enum GraphqlRunError {
JsonError(serde_json::Error),
HostError(serde_json::Value),
}
impl std::error::Error for GraphqlRunError {}
impl From<serde_json::Error> for GraphqlRunError {
fn from(value: serde_json::Error) -> Self {
Self::JsonError(value)
}
}
impl std::fmt::Display for GraphqlRunError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
GraphqlRunError::JsonError(msg) => write!(f, "json error: {msg}"),
GraphqlRunError::HostError(serde_json::Value::Object(map))
if map.contains_key("message") =>
{
write!(f, "host error: {}", map["message"])
}
GraphqlRunError::HostError(val) => write!(f, "host error: {val:?}"),
}
}
}
#[macro_export]
macro_rules! init_mat {
(hook: $init_hook:expr) => {
struct MatWireGuest;
use wit::exports::metatype::wit_wire::mat_wire::*;
wit::export!(MatWireGuest with_types_in wit);
#[allow(unused)]
impl Guest for MatWireGuest {
fn handle(req: HandleReq) -> Result<String, HandleErr> {
MAT_STATE.with(|router| {
let router = router.borrow();
router.handle(req)
})
}
fn init(args: InitArgs) -> Result<InitResponse, InitError> {
let hook = $init_hook;
let router = Router::from_builder(hook());
let resp = router.init(args)?;
MAT_STATE.set(router);
Ok(resp)
}
}
};
}
// gen-static-end
use types::*;
pub mod types {
pub type StringDateTime4 = String;
pub type StringUri5 = String;
#[derive(Debug, serde::Serialize, serde::Deserialize)]
pub struct Idv3 {
pub title: String,
pub artist: String,
#[serde(rename = "releaseTime")]
pub release_time: StringDateTime4,
#[serde(rename = "mp3Url")]
pub mp3_url: StringUri5,
}
}
pub mod stubs {
use super::*;
pub trait RemixTrack: Sized + 'static {
fn erased(self) -> ErasedHandler {
ErasedHandler {
mat_id: "remix_track".into(),
mat_title: "remix_track".into(),
mat_trait: "RemixTrack".into(),
handler_fn: Box::new(move |req, cx| {
let req = serde_json::from_str(req)
.map_err(|err| HandleErr::InJsonErr(format!("{err}")))?;
let res = self
.handle(req, cx)
.map_err(|err| HandleErr::HandlerErr(format!("{err}")))?;
serde_json::to_string(&res)
.map_err(|err| HandleErr::HandlerErr(format!("{err}")))
}),
}
}
fn handle(&self, input: Idv3, cx: Ctx) -> anyhow::Result<Idv3>;
}
pub fn op_to_trait_name(op_name: &str) -> &'static str {
match op_name {
"remix_track" => "RemixTrack",
_ => panic!("unrecognized op_name: {op_name}"),
}
}
}
When working on the typegraph, we can run metagen again to regenerate this file and get the latest types.
The generator also includes a sample lib.rs
entrypoint file for our crate. We'll modify it now to implement our custom function.
mod mdk;
pub use mdk::*;
// the macro sets up all the glue
init_mat! {
// the hook is expected to return a MatBuilder instance
hook: || {
// initialize global stuff here if you need it
MatBuilder::new()
// register function handlers here
// each trait will map to the name of the
// handler found in the typegraph
.register_handler(stubs::RemixTrack::erased(MyMat))
}
}
struct MyMat;
impl stubs::RemixTrack for MyMat {
fn handle(&self, input: types::Idv3, _cx: Ctx) -> anyhow::Result<types::Idv3> {
Ok(types::Idv3 {
title: format!("{} (Remix)", input.title),
artist: format!("{} + DJ Cloud", input.artist),
release_time: input.release_time,
mp3_url: "https://mp3.url/shumba2".to_string(),
})
}
}
Building
We'll now use the rust toolchain and wasm-tools to build the wasm component. This requires multiple commands. It's presented below as a shell script that you can modify from.
# flags to make script execution visible
set -eux
# regenerate code before building
meta gen metagen_rs
# variablize common names
TARGET=wasm32-wasi
CRATE_NAME=metagen_rs_mdk
# build in release mode for smallest sizes
cargo build -p $CRATE_NAME --target $TARGET --release
# use wasm-tools to change wasm file into wasm component
wasm-tools component new \
# rust builds the wasm file under the name of the crate
./target/$TARGET/debug/$CRATE_NAME.wasm \
-o ./target/rust-component.wasm \
# copy the component to a location that we specified
# in our typegraph
cp ./target/rust-component.wasm ./rust.wasm
Put the shell script into a file like build.sh
and execute it with a posix compatible shell like bash
. You should now have all the files to deploy your typegraph.