Metatype Basics
This page will walk you through a real world API with data storage and authorization.
- How to setup your development for metatype projects.
- How to run the typegate on the docker runtime.
- How to create/read/update/delete data.
- How to write custom business logic.
- How to authenticate requests.
- How to protect data with policies.
What are you building?
For this tutorial, we'll be implementing an API to power a simple feature roadmap/request hybrid as can be seen on Productlane.
Looking through the app we can see that the api should allow:
-
Unauthenticated users to submit new "ideas" or vote on any of those already listed.
-
Specify or vote on the importance of an "idea" from "medium" to "critical" or even submit text with more description.
-
Admins will be able to move ideas across buckets like "Backlog", "Planned", "In Progress".
Setup
To setup your Metatype development environment, please follow the installation guide here
Create a new project
Metatype projects are composed of modular bundles of types, logic and policies called typegraphs. We author typegraphs using modern programming languages & environments. Python and Typescript are currently available for use. The meta-cli allows us to create a new project based on pre-existing templates.
Run one the following commands to create a new project under a new directory titled tg_roadmap.
# using Node/Bun runtimes
meta new --template node tg_roadmap
# ^ project name
# ^ Use `meta new --help` find out more available templates.
# using Deno
meta new --template deno tg_roadmap
When using Typescript, the @typegraph/sdk package exposes all the necessary functions and types we'll need to describe our typegraph.
The templates already specify it as as a dependency so all we need to do now is run the following command to download it:
# using Deno
deno cache api/example.ts # cache dependencies
# using Bun
bun install
# using pnpm
pnpm install
# using npm
npm install
# using yarn
yarn install
Launch typegate
The typegate is a program that runs and orchestrates our typegraphs. We can run it locally for development purposes. Typegate currently requires the Redis database to function and to make it easy to run both, we'll make use of a linux container runtime for this. The Docker runtime to be specific which has installation guides located here.
We'll also need the Docker Compose orchestrator which usually comes by default with the docker command. Use the following command to check if it is available:
docker compose version
# Docker Compose version 2.23.0
...and if not, the official installation guide can be found here.
If you have your docker runtime installed and running correctly, you will be able to launch the compose.yml file that's bundled in every template.
The compose file by default includes the postgres and mongo databases.
You can disable the latter by commenting it out or removing it as we'll not be needing it for this tutorial.
To launch the services, navigate your shell to the project directory and run the following command:
docker compose up --detach
# ^ detach means it'll run in the background.
# Omit to get the all logs in the current terminal
This should download and start typegate and its dependent services.
We can observe their log of typegate or any of the other services with the following command. It has to be run from the same project directory.
docker compose logs typegate --follow
# ^ Omit service name to look at the combined logs of all services
Make sure it's all working
Run the following to make sure everything's up and running.
meta doctor
After running the command, you should get a result similar to then one here.
Building our Models
We will be using the type system from the typegraph SDK to describe the shape of the data that flows through our application. In this case, we'll build our typegraph around types that represent "ideas", "votes" and "buckets".
Modify the file at api/example.ts to look something like the following.
// we'll need the following imports
import { t, typegraph } from "@typegraph/sdk.js";
typegraph("roadmap", (g) => {
// ^ each typegraph has a name
const bucket = t.struct({
// asId and other config items describe the logical properties
// of our types beyond just the shape
"id": t.integer({}, { asId: true }),
"name": t.string(),
});
const idea = t.struct({
// uuid is just a shorthand alias for `t.string({format: "uuid"})`
"id": t.uuid({ asId: true }),
"name": t.string(),
// another string shorthand
"authorEmail": t.email(),
});
const vote = t.struct({
"id": t.uuid(),
"authorEmail": t.email(),
// `enum_` is also a shorthand over `t.string`
"importance": t.enum_(["medium", "important", "critical"]).optional(),
// makes it optional
"desc": t.string().optional(),
});
});
The types here are very simple and we haven't yet added any thing that models their relationships but they should do for our purposes.
Exposing our application
Typegraphs expose an API to the external world using Materializer objects. Materializers describe functions that transform some input type into an output type and we define them in scope of different Runtimes, where the actual logic runs. At this early stage, we can make use of the Random runtime which allows us to generate random test data for our types to get a feel of our API.
// add need the following imports
import { Policy } from "@typegraph/sdk/index.js";
import { RandomRuntime } from "@typegraph/sdk/runtimes/random.js";
typegraph("roadmap", (g) => {
// ...
// every exposed materializer requires access control policies
// for now, just use the public policy, anyone can access it
const pub = Policy.public();
const random = new RandomRuntime({});
g.expose(
{
// generates a random object in the shape of idea
"get_idea": random.gen(idea).withPolicy(pub),
},
);
});
At this point, we can push our typegraph to the locally running typegate node and access it. Run the following command in your project root:
# features auto-reload on any changes to your source files
meta dev
Typegate has first-class support for consuming the API through a GraphQl interface and it is enabled by default. It also bundles the GrahpiQl API explorer and you should be able to access it at http://localhost:7890/roadmap once meta-cli has successfully pushed your typegraph.
You can go ahead and try out the following graphql on the interface and get a feel for it.
query {
get_idea {
id,
name,
authorEmail
}
}
Or, you can mess around on the playground below.
The Prisma Runtime
Now that we have created a simple endpoint that generates random values for our idea model/type, let's add a CRUD support to our app.
A runtime most apps will be depend on is the Prisma Runtime which allows us to connect to different databases and peform database operations.
For this tutorial, we'll be making use of the PostgreSQL database.
If you made use of the compose.yml to run typegate as outlined in this tutorial, there should be an instance of Postgres already up.
You can check if postgres container is currently running by using the meta doctor command.
If a typegraph needs to access a database, it first needs to be made aware of its address.
This is done through secrets.
In the root of your project directory, you'll find a file titled metatype.yaml.
It contains metatype specific configuration for our project such as the top level typegates object which we use to specify the location and credentials of the different typegate nodes we'll be using.
Each typegate entry also takes an secrets object where we can specify secret to be passed to our typegraphs.
The keys in the secrets object are the names of the typegraphs and the values are objects mapping secret names to their values/sources.
The metatype.yaml should already have a few sample environment variables.
Add an entry like the following to give our typegraph access to the database's address:
typegates:
dev:
# ..
secrets:
TG_ROADMAP_POSTGRES: "postgresql://postgres:password@postgres:5432/db"
Meta-cli will auto-reload when it detects changes to metatype.yaml. This is because Meta-cli was run in dev mode(through the meta dev command).
We can add the Prisma runtime to our typegraph now.
// new imports
import { PrismaRuntime } from "@typegraph/sdk/providers/prisma.js";
typegraph("roadmap", (g) => {
// ...
// the constructor takes the name of the env var directly
const db = new PrismaRuntime("db", "POSTGRES");
// ...
});
One of the features that the Prisma runtime allows us to implement is relationships. Here, we are creating a one to many relationship between bucket and ideas, also another one to many between ideas and vote. We will be specifiying relationships by using the t.list List type and g.ref(method which accepts the name of the model/entity as a parameter) for creating the link. Check the example below for better understanding.
import { PrismaRuntime } from "@typegraph/sdk/providers/prisma.js";
typegraph("roadmap", (g) => {
// ...
const db = new PrismaRuntime("db", "POSTGRES");
const bucket = t.struct({
"id": t.integer({}, {
asId: true,
// auto generate ids during creation
config: { auto: true }
}),
"name": t.string(),
// one-to many relationship
"ideas": t.list(g.ref("idea")),
})
// explicitly naming our types makes reference later easier
.rename("bucket");
const idea = t.struct({
"id": t.uuid({ asId: true, config: { auto: true } }),
"name": t.string(),
"authorEmail": t.email(),
// we need to specify the relationships on both types
"bucket": g.ref("bucket"),
"votes": t.list(g.ref("vote")),
})
.rename("idea");
const vote = t.struct({
"id": t.uuid({ asId: true, config: { auto: true } }),
"authorEmail": t.email(),
"importance": t.enum_(["medium", "important", "critical"]).optional(),
"desc": t.string().optional(),
"idea": g.ref("idea")
})
.rename("vote");
// ...
});
g.ref declares logical relationships between our types which the Prisma runtime will be able to pick up.
If you need more control on what the relationships will look like on the database, you can use the db.link function.
More information can be found on the Prisma runtime reference.
When we save our file at this point, the meta dev watcher should automatically create and push the necessary migrations to our database to get it in its intended shape.
You should see a new subdirectory in your project called prisma.
It's where the generated migrations are contained.
If you mess something up in the migrations and want a clean slate, you can reset everything by recreating the containers like so:
# remove all containers and their volumes
docker compose down -v
# launch
docker compose up --detach
# meta dev will auto apply any pending changes to databases
meta dev
At this point, we're ready to add materializers to expose database queries to create or read data. The Prisma runtime allows us to run raw queries directly on the database but it also provides handy functions we can use for basic CRUD operations. We'll make use of those.
import { PrismaRuntime } from "@typegraph/sdk/providers/prisma.js";
typegraph("roadmap", (g) => {
// ...
const pub = Policy.public();
const db = new PrismaRuntime("db", "POSTGRES");
// ...
g.expose(
{
"get_buckets": db.findMany(bucket),
"create_bucket": db.create(bucket),
"get_idea": db.findFirst(idea),
"create_ideas": db.create(idea),
}, pub // make all materializers public by default
);
});
We should be able to add a few buckets and ideas now.
Policies
We now have the tools enough to allow coarse CRUD of our data. The next thing we usually add at this point is authorization. A way to control who can read or write what. The primary mechanism typegraphs use for this purpose are policies.
Policies are small functions that get the context of a request as input and return a boolean signaling weather access should be granted.
Metatype currently supports policies based on javascript functions that are run on the Deno runtime.
For this tutorial, we'll be making use of the basic auth extractor.
It expects a string in the format "Basic token" to be set in the Authorization http header.
The token is expected to be a base64 encoded string in the format username:secret.
import { DenoRuntime } from "@typegraph/sdk/runtimes/deno.js";
import { Auth } from "@typegraph/sdk/params.js";
typegraph("roadmap", (g) => {
// ...
const deno = new DenoRuntime();
// The basic extractor only populates the context when
// it recognizes the username and the secret matches
g.auth(Auth.basic(["andim", /*more users*/]))
// the `username` value is only availaible if the basic
// extractor was successful
const admins = deno.policy("admins", `
(_args, { context }) => !!context.username
`);
g.expose(
{
// ..
// only admins are allowed to create new buckets
"create_bucket": db.create(bucket).withPolicy(admins),
// ..
}, pub
);
// ...
});
The basic extractors expects the secrets in environment variables named in a specific format.
Add the following entries to the metatype.yaml file:
typegates:
dev:
# ..
secrets:
roadmap: # your typegraph name
# ..
# the basic extractor secret format
# BASIC_[username]
BASIC_ADMIN: hunter2
When you save the files, meta-cli will reload the new additions to your typegraph.
create_bucket is now only accessible to requests bearing the right tokens
(For the provided example, Basic YW5kaW06aHVudGVyMg== should work).
If you are using the GraphiQl interface from earlier, there should be a panel in the bottom left called "Headers" for setting http headers
More Customization for our app
Preventing Unauthorized Creation of Related Objects
Reference: Parameter transformations
By default, Prisma generates types that supports the whole suite of usecases one might have on a CRUD operation such as allowing creation of objects of related types in a single operation.
We don't always want this and in our case, we want to prevent users from being able to create buckets, which are protected, through the create_idea materializer which's public.
We can use the reduce method to modify the input types of functions.
mutation CIdea {
create_idea(
data: {
# we want to prevent bucket creation through `create_idea`
bucket: {
create: {name: "Backlog"}
},
authorEmail: "[email protected]",
name: "Add support for WASM GC"
}
) {
id
name
}
}
Even though the reduce method doesn't allow us to change the shape of the type, we can change the types of members and importantly here, hide the ones we don't need.
typegraph("roadmap", (g) => {
// ...
g.expose(
{
// ..
"create_idea": db.create(idea).reduce({
"data": {
// `g.inherit` specifies that we keep the member
// type of the original
"name": g.inherit(),
"authorEmail": g.inherit(),
"votes": g.inherit(),
"bucket": {
"connect": g.inherit(),
// by omitting the `create` member, we hide it
}
}
}),
// ..
}, pub
);
});
Requests are now only able to connect new ideas with pre-existing buckets and won't be able to create them.
If you try to create new bucket through create_idea, the typgate will return this response.
{
"errors": [
{
"message": "Unexpected property 'create' for argument 'data.bucket' of type 'object' ('object_288') at create_idea; valid properties are: connect",
"locations": [],
"path": [],
"extensions": {
"timestamp": "2024-04-21T09:46:33.177Z"
}
}
]
}
Instead, If you try using this mutation, it will work as expected. You can only specify buckets that are already created.
mutation {
create_idea(
data: {
# we want to prevent bucket creation through `create_idea`
bucket: {
connect: {id: 1}
},
authorEmail: "[email protected]",
name: "Add support for WASM GC"
}
) {
id
name,
bucket {
id,
name
}
}
}
Restrict Update Operation on Selected Fields
You'll notice that we had set the importance field on votes as optional.
This is to allow users to just up-vote an idea from the main list without opening a form.
If they want to add importance or a description to their vote at a later point, we want to update their already existing vote.
It should be easy to expose a materializer for this using Prisma's db.update helper and reduce to restrict changes to only those field.
But we'll take this opportunity to explore the feature of the Prisma runtime to execute raw queries.
import * as effects from "@typegraph/sdk/effects.js";
typegraph("roadmap", (g) => {
// ...
g.expose(
{
// ..
"set_vote_importance": db.execute(
// query parameters are matched by name from the input type
'UPDATE "vote" SET importance = ${importance} WHERE id = ${vote_id}::uuid',
// our input type
t.struct({
"vote_id": t.uuid(),
"importance": t.enum_(["medium", "important", "critical"]),
}),
// we use effects to signal what kind of operation we're doing
// updates and creates will be exposed as mutations in GraphQl
// the boolean signals that the query is idempotent
effects.update(true),
)
// ..
}, pub
);
});
Our query is exposed like any other materializer in the GraphQl api.
Creating REST endpoints
We can easily expose an HTTP API for our typegraph using the g.rest method.
It takes a string describe a graphql query to be executed when the http path is requested.
typegraph("roadmap", (g) => {
// ...
g.rest(
`
query get_buckets {
get_buckets {
id
name
ideas {
id
name
authorEmail
}
}
}
`
)
g.rest(
// query parameters present
// expects a request of the type `roadmap/rest/get_bucket?id=uuidstr`
`
query get_bucket($id: Integer) {
get_bucket(where:{
id: $id
}) {
id
name
ideas {
id
name
authorEmail
}
}
}
`
)
});
The exposed query is served at the path {typegate_url}/{typegraph_name}/rest/{query_name}.
Any parameters that the query takes are processed from the search params of the request. You can check this link on your local machine and check the results.
This is it for this tutorial and thanks for following till the end! This was a long one but we hope it gave you an overview to the vast capabilties of Metatype. We ecourage you to keep exploring!