Metagen

Beta

The following feature is not yet stable.

Metagen is a code-generator suite that contains implementations that help with development on the Metatype platform. Today, this means a set of generators to help with custom functions by generating types, serializers and bindings. It's availaible bundled within the meta CLI and the typegraph SDKs.

Access through CLI

The meta-cli has a dedicated gen command for interacting with metagen. We configure the generators through the standard configuration file under the metagen key.

typegates:
  # bla bla

typegraphs:
  # bla bla

metagen:
  targets:
    main:
      # generator to use
      - generator: fdk_rust
        # path to generate to
        path: ./bff/
        # typegraph path to use
        typegraph_path: ./typegraphs/svc-bff.ts
        # we can have multiple generators per target
      - generator: fdk_rust
        path: ./telemetry/
        typegraph_path: ./typegraphs/svc-telemetry.ts
        # generators might have custom keys
        stubbed_runtimes: ["wasm_wire", "deno"]
    # more than one targets avail if you need them
    iter:
      - generator: fdk_typescript
        path: ./ts/
        # name of typegraph to read from typegate
        typegraph: svc_products

This allows us to invoke the targets from the CLI.

meta cli gen main

This will resolve the requisite typegraphs, serialize as needed and put the resulting files at the appropriate locations. If no target name is provied, the CLI will look for a target under the key main and invoke it instead.

Access through SDK

Metagen is availaible through the SDK for programmatic access needs and can be helpful when writing tests or when relying on the CLI is not an option.

Loading...

Generators

Chicken or the egg?

As most of the generators are intended for types to be used by custom functions, they'll require that you declare the custom functions in your typegraph first. This begs the question, how does one declare custom functions that depend on artifacts that are yet to be generated? Typegraphs error out when referenced artifacts aren't found, how does it work in this scenario?

To resolve this concern, the SDKs support a serialization mode that skips resolution of artifacts. This mode is activated when serialization is done for codegen purposes. What this means is that, you can declare non-existent files in your typegraph and codegen should work. Some generators are even smart enough to work around your expected files. Of course, if the files aren't present when you're trying to deply to the typegate, it'll raise an error.

fdk_typescript

This generator supports:

• Typescript types that map to typegraph types
• Stub function types for custom functions implementors that adhere to typegraph functions.
  • By default, all function types from the DenoRuntime get stub types.
  • Use stubbed_runtimes to select which runtimes get stubs.
• Types for interacting with the typegate from within custom functions.

The following example showcases the generator.

Typegraph:

Loading...

Custom function:

Loading...

Code generation sample.

Loading...

It supports the following extra configuration keys.

Key	Type	Default	Description
stubbed_runtimes	string[]	["deno"]	Runtimes for which to generate stub types.

fdk_python

This generator supports:

• Python classes that map to typegraph types
• Decorators for custom functions implementors that require adherance to typegraph function types.
  • By default, all functions from the PythonRuntime get stub types.
  • TODO: stubbed_runtimes for fdk_python
• TODO: types for interacting with the typegate from within custom functions.

If the referenced module for the custom function is not found, the generator will also output stub implementation (in addition to the types) at the given type. It will not replace our code on a second run.

The following example showcases the generator.

Typegraph:

Loading...

Custom function:

Loading...

Code generation sample.

Loading...

fdk_rust

This generator generates types, serializers and bindings needed to implement custom functions in Rust. Rust implementations will need to be compiled to wasm components to be executed on the metatype platform and the generator assumes such usage.

To be more specific, it supports:

• Rust types that map to typegraph defined types
  • Serialization is handled out of sight through serde_json
• Stub traits for custom functions implementors that adhere to typegraph functions.
  • By default, all functions from the WasmRuntime get stub types.
  • The generator assumes the wire based wasm interface is being targetted.
  • stubbed_runtimes key can be used to configure stub generation from additional runtimes.
• Types for interacting with the typegate from within custom functions.
• Glue code for setting up the wasm component to be run within the WasmRuntime.

By default the generator will also output a library crate entrypoint and a functional Cargo.toml with all the required dependencies. These additional files wlil not be overwritten on a second run. The generator can also be configured to avoid generating them even if not present.

The following example showcases the generator.

Typegraph:

Loading...

Custom function:

mod fdk;
pub use fdk::*;

// the macro sets up all the glue
init_mat! {
    // the hook is expected to return a MatBuilder instance
    hook: || {
        // initialize global stuff here if you need it
        MatBuilder::new()
            // register function handlers here
            // each trait will map to the name of the
            // handler found in the typegraph
            .register_handler(stubs::RemixTrack::erased(MyMat))
    }
}

struct MyMat;

impl stubs::RemixTrack for MyMat {
    fn handle(&self, input: types::Idv3, _cx: Ctx) -> anyhow::Result<types::Idv3> {
        Ok(types::Idv3 {
            title: format!("{} (Remix)", input.title),
            artist: format!("{} + DJ Cloud", input.artist),
            release_time: input.release_time,
            mp3_url: "https://mp3.url/shumba2".to_string(),
        })
    }
}

Code generation sample.

// This file was @generated by metagen and is intended
// to be generated again on subsequent metagen runs.
#![cfg_attr(rustfmt, rustfmt_skip)]

// gen-static-start
#![allow(dead_code)]

pub mod wit {
    wit_bindgen::generate!({
        pub_export_macro: true,
        
        inline: "package metatype:wit-wire;

interface typegate-wire {
  hostcall: func(op-name: string, json: string) -> result<string, string>;
}

interface mat-wire {
  type json-str = string;

  record mat-info {
    op-name: string,
    mat-title: string,
    mat-hash: string,
    mat-data-json: string,
  }

  record init-args {
    metatype-version: string,
    expected-ops: list<mat-info>
  }

  record init-response {
    ok: bool
  }

  variant init-error {
    version-mismatch(string),
    unexpected-mat(mat-info),
    other(string)
  }

  init: func(args: init-args) -> result<init-response, init-error>;

  record handle-req {
    op-name: string,
    in-json: json-str,
  }

  variant handle-err {
    no-handler,
    in-json-err(string),
    handler-err(string),
  }

  handle: func(req: handle-req) -> result<json-str, handle-err>;
}

world wit-wire {
  import typegate-wire;

  export mat-wire;
}
"
    });
}

use std::cell::RefCell;
use std::collections::HashMap;

use wit::exports::metatype::wit_wire::mat_wire::*;
use wit::metatype::wit_wire::typegate_wire::hostcall;

pub type HandlerFn = Box<dyn Fn(&str, Ctx) -> Result<String, HandleErr>>;

pub struct ErasedHandler {
    mat_id: String,
    mat_trait: String,
    mat_title: String,
    handler_fn: HandlerFn,
}

pub struct MatBuilder {
    handlers: HashMap<String, ErasedHandler>,
}

impl MatBuilder {
    pub fn new() -> Self {
        Self {
            handlers: Default::default(),
        }
    }

    pub fn register_handler(mut self, handler: ErasedHandler) -> Self {
        self.handlers.insert(handler.mat_trait.clone(), handler);
        self
    }
}

pub struct Router {
    handlers: HashMap<String, ErasedHandler>,
}

impl Router {
    pub fn from_builder(builder: MatBuilder) -> Self {
        Self {
            handlers: builder.handlers,
        }
    }

    pub fn init(&self, args: InitArgs) -> Result<InitResponse, InitError> {
        static MT_VERSION: &str = "0.4.11-rc.0";
        if args.metatype_version != MT_VERSION {
            return Err(InitError::VersionMismatch(MT_VERSION.into()));
        }
        for info in args.expected_ops {
            let mat_trait = stubs::op_to_trait_name(&info.op_name);
            if !self.handlers.contains_key(mat_trait) {
                return Err(InitError::UnexpectedMat(info));
            }
        }
        Ok(InitResponse { ok: true })
    }

    pub fn handle(&self, req: HandleReq) -> Result<String, HandleErr> {
        let mat_trait = stubs::op_to_trait_name(&req.op_name);
        let Some(handler) = self.handlers.get(mat_trait) else {
            return Err(HandleErr::NoHandler);
        };
        let cx = Ctx {};
        (handler.handler_fn)(&req.in_json, cx)
    }
}

pub type InitCallback = fn() -> anyhow::Result<MatBuilder>;

thread_local! {
    pub static MAT_STATE: RefCell<Router> = panic!("MAT_STATE has not been initialized");
}

pub struct Ctx {}

impl Ctx {
    pub fn gql<O>(
        &self,
        query: &str,
        variables: impl Into<serde_json::Value>,
    ) -> Result<O, GraphqlRunError>
    where
        O: serde::de::DeserializeOwned,
    {
        match hostcall(
            "gql",
            &serde_json::to_string(&serde_json::json!({
                "query": query,
                "variables": variables.into(),
            }))?,
        ) {
            Ok(json) => Ok(serde_json::from_str(&json[..])?),
            Err(json) => Err(GraphqlRunError::HostError(serde_json::from_str(&json)?)),
        }
    }
}

#[derive(Debug)]
pub enum GraphqlRunError {
    JsonError(serde_json::Error),
    HostError(serde_json::Value),
}

impl std::error::Error for GraphqlRunError {}

impl From<serde_json::Error> for GraphqlRunError {
    fn from(value: serde_json::Error) -> Self {
        Self::JsonError(value)
    }
}

impl std::fmt::Display for GraphqlRunError {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        match self {
            GraphqlRunError::JsonError(msg) => write!(f, "json error: {msg}"),
            GraphqlRunError::HostError(serde_json::Value::Object(map))
                if map.contains_key("message") =>
            {
                write!(f, "host error: {}", map["message"])
            }
            GraphqlRunError::HostError(val) => write!(f, "host error: {val:?}"),
        }
    }
}

#[macro_export]
macro_rules! init_mat {
    (hook: $init_hook:expr) => {
        struct MatWireGuest;
        use wit::exports::metatype::wit_wire::mat_wire::*;
        wit::export!(MatWireGuest with_types_in wit);

        #[allow(unused)]
        impl Guest for MatWireGuest {
            fn handle(req: HandleReq) -> Result<String, HandleErr> {
                MAT_STATE.with(|router| {
                    let router = router.borrow();
                    router.handle(req)
                })
            }

            fn init(args: InitArgs) -> Result<InitResponse, InitError> {
                let hook = $init_hook;
                let router = Router::from_builder(hook());
                let resp = router.init(args)?;
                MAT_STATE.set(router);
                Ok(resp)
            }
        }
    };
}
// gen-static-end
use types::*;
pub mod types {
    pub type Idv3TitleString = String;
    pub type Idv3ReleaseTimeStringDatetime = String;
    pub type Idv3Mp3UrlStringUri = String;
    #[derive(Debug, serde::Serialize, serde::Deserialize)]
    pub struct Idv3 {
        pub title: Idv3TitleString,
        pub artist: Idv3TitleString,
        #[serde(rename = "releaseTime")]
        pub release_time: Idv3ReleaseTimeStringDatetime,
        #[serde(rename = "mp3Url")]
        pub mp3_url: Idv3Mp3UrlStringUri,
    }
}
pub mod stubs {
    use super::*;
    pub trait RemixTrack: Sized + 'static {
        fn erased(self) -> ErasedHandler {
            ErasedHandler {
                mat_id: "remix_track".into(),
                mat_title: "remix_track".into(),
                mat_trait: "RemixTrack".into(),
                handler_fn: Box::new(move |req, cx| {
                    let req = serde_json::from_str(req)
                        .map_err(|err| HandleErr::InJsonErr(format!("{err}")))?;
                    let res = self
                        .handle(req, cx)
                        .map_err(|err| HandleErr::HandlerErr(format!("{err}")))?;
                    serde_json::to_string(&res)
                        .map_err(|err| HandleErr::HandlerErr(format!("{err}")))
                }),
            }
        }

        fn handle(&self, input: Idv3, cx: Ctx) -> anyhow::Result<Idv3>;
    }
    pub fn op_to_trait_name(op_name: &str) -> &'static str {
        match op_name {
            "remix_track" => "RemixTrack",
            _ => panic!("unrecognized op_name: {op_name}"),
        }
    }
}

It supports the following extra configuration keys.

Key	Type	Default	Description
stubbed_runtimes	string[]	["wasm_wire"]	Runtimes for which to generate stub types.
crate_name	string	${typegraphName}_fdk	Name to assign to crate when generating Cargo.toml.
skip_cargo_toml	boolean	false	Do not generate Cargo.toml.
skip_lib_rs	boolean	false	Do not generate lib.rs, the sample entrypoint.