Exporting & Importing Package APIs

Kinode packages can export APIs, as discussed here. Processes can also import APIs. These APIs can consist of types as well as functions. This recipe focuses on:

  1. Simple examples of exporting and importing APIs (find the full code here).
  2. Demonstrations of kit tooling to help build and export or import APIs.

Exporting an API

APIs are defined in a WIT file. A brief summary of more thorough discussion is provided here:

  1. WIT (Wasm Interface Type) is a language to define APIs. Kinode packages may define a WIT API by placing a WIT file in the top-level api/ directory.
  2. Processes define a WIT interface.
  3. Packages define a WIT world.
  4. APIs define their own WIT world that exports at least one processes WIT interface.

Example: Remote File Storage Server

WIT API

#![allow(unused)]
fn main() {
interface server {
    variant client-request {
        put-file(string),
        get-file(string),
        list-files,
    }

    variant client-response {
        put-file(result<_, string>),
        get-file(result<_, string>),
        list-files(result<list<string>, string>),
    }

    /// `put-file()`: take a file from local VFS and store on remote `host`.
    put-file: func(host: string, path: string, name: string) -> result<_, string>;

    /// `get-file()`: retrieve a file from remote `host`.
    /// The file populates the lazy load blob and can be retrieved
    /// by a call of `get-blob()` after calling `get-file()`.
    get-file: func(host: string, name: string) -> result<_, string>;

    /// `list-files()`: list all files we have stored on remote `host`.
    list-files: func(host: string) -> result<list<string>, string>;
}

world server-template-dot-os-api-v0 {
    export server;
}

world server-template-dot-os-v0 {
    import server;
    include process-v0;
}
}

As summarized above, the server process defines an interface of the same name, and the package defines the world server-template-dot-os-v0. The API is defined by server-template-dot-os-api-v0: the functions in the server interface are defined below by wit_bindgen::generate!()ing that world.

The example covered in this document shows an interface that has functions exported. However, for interfaces that export only types, no -api- world (like server-template-dot-os-api-v0 here) is required. Instead, the WIT API alone suffices to export the types, and the importer writes a world that looks like this, below. For example, consider the chat template's api/ and its usage in the test/ package:

kit n my_chat
cat my_chat/api/my_chat\:template.os-v0.wit
cat my_chat/test/my_chat_test/api/my_chat_test\:template.os-v0.wit

API Function Definitions

#![allow(unused)]
fn main() {
use crate::exports::kinode::process::server::{ClientRequest, ClientResponse, Guest};
use kinode_process_lib::{vfs, Request, Response};

wit_bindgen::generate!({
    path: "target/wit",
    world: "server-template-dot-os-api-v0",
    generate_unused_types: true,
    additional_derives: [serde::Deserialize, serde::Serialize, process_macros::SerdeJsonInto],
});

const READ_TIMEOUT_SECS: u64 = 5;
const PUT_TIMEOUT_SECS: u64 = 5;

fn make_put_file_error(message: &str) -> anyhow::Result<Result<(), String>> {
    Response::new()
        .body(ClientResponse::PutFile(Err(message.to_string())))
        .send()?;
    return Err(anyhow::anyhow!(message.to_string()));
}

fn make_get_file_error(message: &str) -> anyhow::Result<Result<(), String>> {
    Response::new()
        .body(ClientResponse::GetFile(Err(message.to_string())))
        .send()?;
    return Err(anyhow::anyhow!(message.to_string()));
}

fn make_list_files_error(message: &str) -> anyhow::Result<Result<Vec<String>, String>> {
    Response::new()
        .body(ClientResponse::GetFile(Err(message.to_string())))
        .send()?;
    return Err(anyhow::anyhow!(message.to_string()));
}

fn put_file(host: String, path: String, name: String) -> anyhow::Result<Result<(), String>> {
    // rather than using `vfs::open_file()?.read()?`, which reads
    // the file into process memory, send the Request to VFS ourselves,
    // `inherit`ing the file contents into the ClientRequest
    //
    // let contents = vfs::open_file(path, false, None)?.read()?;
    //
    let response = Request::new()
        .target(("our", "vfs", "distro", "sys"))
        .body(serde_json::to_vec(&vfs::VfsRequest {
            path: path.to_string(),
            action: vfs::VfsAction::Read,
        })?)
        .send_and_await_response(READ_TIMEOUT_SECS)??;
    let response = response.body();
    let Ok(vfs::VfsResponse::Read) = serde_json::from_slice(&response) else {
        return make_put_file_error(&format!("Could not find file at {path}."));
    };
    let ClientResponse::PutFile(result) = Request::new()
        .target((&host, "server", "server", "template.os"))
        .inherit(true)
        .body(ClientRequest::PutFile(name))
        .send_and_await_response(PUT_TIMEOUT_SECS)??
        .body()
        .try_into()?
    else {
        return make_put_file_error(&format!("Got unexpected Response from server."));
    };
    Ok(result)
}

fn get_file(host: String, name: String) -> anyhow::Result<Result<(), String>> {
    let ClientResponse::GetFile(result) = Request::new()
        .target((&host, "server", "server", "template.os"))
        .body(ClientRequest::GetFile(name))
        .send_and_await_response(PUT_TIMEOUT_SECS)??
        .body()
        .try_into()?
    else {
        return make_get_file_error(&format!("Got unexpected Response from server."));
    };
    Ok(result)
}

fn list_files(host: String) -> anyhow::Result<Result<Vec<String>, String>> {
    let ClientResponse::ListFiles(result) = Request::new()
        .target((&host, "server", "server", "template.os"))
        .inherit(true)
        .body(ClientRequest::ListFiles)
        .send_and_await_response(PUT_TIMEOUT_SECS)??
        .body()
        .try_into()?
    else {
        return make_list_files_error(&format!("Got unexpected Response from server."));
    };
    Ok(result)
}

struct Api;
impl Guest for Api {
    fn put_file(host: String, path: String, name: String) -> Result<(), String> {
        match put_file(host, path, name) {
            Ok(result) => result,
            Err(e) => Err(format!("{e:?}")),
        }
    }

    fn get_file(host: String, name: String) -> Result<(), String> {
        match get_file(host, name) {
            Ok(result) => result,
            Err(e) => Err(format!("{e:?}")),
        }
    }

    fn list_files(host: String) -> Result<Vec<String>, String> {
        match list_files(host) {
            Ok(result) => result,
            Err(ref e) => Err(format!("{e:?}")),
        }
    }
}
export!(Api);
}

Functions must be defined if exported in an interface, as they are here. Functions are defined by creating a directory just like a process directory, but with a slightly different lib.rs (see directory structure). Note the definition of struct Api, the impl Guest for Api, and the export!(Api):

#![allow(unused)]
fn main() {
struct Api;
impl Guest for Api {

...

}
export!(Api);
}

The exported functions are defined here. Note the function signatures match those defined in the WIT API.

Process

A normal process: the server handles Requests from consumers of the API.

#![allow(unused)]
fn main() {
use std::collections::{HashMap, HashSet};

use crate::kinode::process::server::{ClientRequest, ClientResponse};
use kinode_process_lib::{
    await_message, call_init, get_blob, println, vfs, Address, Message, PackageId, Request,
    Response,
};

wit_bindgen::generate!({
    path: "target/wit",
    world: "server-template-dot-os-v0",
    generate_unused_types: true,
    additional_derives: [serde::Deserialize, serde::Serialize, process_macros::SerdeJsonInto],
});

type State = HashMap<String, HashSet<String>>;
const READ_TIMEOUT_SECS: u64 = 5;

fn make_drive_name(our: &PackageId, source: &str) -> String {
    format!("/{our}/{source}")
}

fn make_put_file_error(message: &str) -> anyhow::Result<()> {
    Response::new()
        .body(ClientResponse::PutFile(Err(message.to_string())))
        .send()?;
    return Err(anyhow::anyhow!(message.to_string()));
}

fn make_get_file_error(message: &str) -> anyhow::Result<()> {
    Response::new()
        .body(ClientResponse::GetFile(Err(message.to_string())))
        .send()?;
    return Err(anyhow::anyhow!(message.to_string()));
}

fn make_list_files_error(message: &str) -> anyhow::Result<()> {
    Response::new()
        .body(ClientResponse::ListFiles(Err(message.to_string())))
        .send()?;
    return Err(anyhow::anyhow!(message.to_string()));
}

fn handle_put_file(
    name: &str,
    our: &PackageId,
    source: &str,
    state: &mut State,
) -> anyhow::Result<()> {
    let Some(ref blob) = get_blob() else {
        return make_put_file_error("Must give a file in the blob.");
    };

    let drive = vfs::create_drive(our.clone(), source, None)?;
    vfs::create_file(&format!("{drive}/{name}"), None)?.write(blob.bytes())?;
    state
        .entry(source.to_string())
        .or_insert_with(HashSet::new)
        .insert(name.to_string());
    Response::new()
        .body(ClientResponse::PutFile(Ok(())))
        .send()?;
    Ok(())
}

fn handle_get_file(name: &str, our: &PackageId, source: &str, state: &State) -> anyhow::Result<()> {
    let Some(ref names) = state.get(source) else {
        return make_get_file_error(&format!("{source} has no files to Get."));
    };
    if !names.contains(name) {
        return make_get_file_error(&format!("{source} has no such file {name}."));
    }

    // rather than using `vfs::open_file()?.read()?`, which reads
    // the file into process memory, send the Request to VFS ourselves,
    // `inherit`ing the file contents into the ClientResponse
    //
    // let contents = vfs::open_file(path, false, None)?.read()?;
    //
    let path = format!("{}/{name}", make_drive_name(our, source));
    let response = Request::new()
        .target(("our", "vfs", "distro", "sys"))
        .body(serde_json::to_vec(&vfs::VfsRequest {
            path,
            action: vfs::VfsAction::Read,
        })?)
        .send_and_await_response(READ_TIMEOUT_SECS)??;
    let response = response.body();
    let Ok(vfs::VfsResponse::Read) = serde_json::from_slice(&response) else {
        return make_get_file_error(&format!("Could not find file at {name}."));
    };
    Response::new()
        .inherit(true)
        .body(ClientResponse::GetFile(Ok(())))
        .send()?;
    Ok(())
}

fn handle_list_files(source: &str, state: &State) -> anyhow::Result<()> {
    let Some(ref names) = state.get(source) else {
        return make_list_files_error(&format!("{source} has no files to List."));
    };
    let mut names: Vec<String> = names.iter().cloned().collect();
    names.sort();
    Response::new()
        .body(ClientResponse::ListFiles(Ok(names)))
        .send()?;
    Ok(())
}

fn handle_message(our: &Address, message: &Message, state: &mut State) -> anyhow::Result<()> {
    let source = message.source();
    if !message.is_request() {
        return Err(anyhow::anyhow!("unexpected Response from {source}"));
    }
    match message.body().try_into()? {
        ClientRequest::PutFile(ref name) => {
            handle_put_file(name, &our.package_id(), source.node(), state)?
        }
        ClientRequest::GetFile(ref name) => {
            handle_get_file(name, &our.package_id(), source.node(), state)?
        }
        ClientRequest::ListFiles => handle_list_files(source.node(), state)?,
    }
    Ok(())
}

call_init!(init);
fn init(our: Address) {
    println!("begin");

    let mut state: State = HashMap::new();

    loop {
        match await_message() {
            Err(send_error) => println!("got SendError: {send_error}"),
            Ok(ref message) => match handle_message(&our, message, &mut state) {
                Err(e) => println!("got error while handling message: {e:?}"),
                Ok(_) => {}
            },
        }
    }
}
}

Importing an API

Dependencies

metadata.json

The metadata.json file has a properties.dependencies field. When the dependencies field is populated, kit build will fetch that dependency from either:

  1. A livenet Kinode hosting it.
  2. A local path.
  3. An HTTP endpoint (coming soon).

Fetching Dependencies

kit build resolves dependencies in a few ways.

The first is from a livenet Kinode hosting the depenency. This method requires a --port (or -p for short) flag when building a package that has a non-empty dependencies field. That --port corresponds to the Kinode hosting the API dependency.

To host an API, your Kinode must either:

  1. Have that package downloaded by the app_store.
  2. Be a live node, in which case it will attempt to contact the publisher of the package, and download the package. Thus, when developing on a fake node, you must first build and start any dependencies on your fake node before building packages that depend upon them: see usage example below.

The second way kit build resolves dependencies is with a local path.

Example: Remote File Storage Client Script

WIT API

#![allow(unused)]
fn main() {
world client-template-dot-os-v0 {
    import server;
    include process-v0;
}
}

Process

The client process here is a script. In general, importers of APIs are just processes, but in this case, it made more sense for this specific functionality to write it as a script. The Args and Command structs set up command-line parsing and are unrelated to the WIT API.

#![allow(unused)]
fn main() {
use clap::{Parser, Subcommand};

use crate::kinode::process::server::{get_file, list_files, put_file};
use kinode_process_lib::{await_next_message_body, call_init, get_blob, println, Address};

wit_bindgen::generate!({
    path: "target/wit",
    world: "client-template-dot-os-v0",
    generate_unused_types: true,
    additional_derives: [serde::Deserialize, serde::Serialize, process_macros::SerdeJsonInto],
});

#[derive(Parser)]
#[command(version, about)]
struct Args {
    #[command(subcommand)]
    command: Option<Command>,
}

#[derive(Subcommand)]
enum Command {
    /// Take a file from local VFS and store on remote `host`.
    PutFile {
        host: String,
        #[arg(short, long)]
        path: String,
        #[arg(short, long)]
        name: Option<String>,
    },
    /// Retrieve a file from remove `host`.
    GetFile {
        host: String,
        #[arg(short, long)]
        name: String,
    },
    /// List all files we have stored on remote `host`.
    ListFiles { host: String },
}

fn handle_put_file(host: &str, path: &str, name: &str) -> anyhow::Result<()> {
    match put_file(host, path, name) {
        Err(e) => Err(anyhow::anyhow!("{e}")),
        Ok(_) => {
            println!("Successfully PutFile {path} to host {host}.");
            Ok(())
        }
    }
}

fn handle_get_file(host: &str, name: &str) -> anyhow::Result<()> {
    match get_file(host, name) {
        Err(e) => Err(anyhow::anyhow!("{e}")),
        Ok(_) => {
            if let Some(blob) = get_blob() {
                if let Ok(contents) = String::from_utf8(blob.bytes().to_vec()) {
                    println!("Successfully GetFile {name} from host {host}:\n\n{contents}");
                    return Ok(());
                }
            }
            println!("Successfully GetFile {name} from host {host}.");
            Ok(())
        }
    }
}

fn handle_list_files(host: &str) -> anyhow::Result<()> {
    match list_files(host) {
        Err(e) => Err(anyhow::anyhow!("{e}")),
        Ok(paths) => {
            println!("{paths:#?}");
            Ok(())
        }
    }
}

fn execute() -> anyhow::Result<()> {
    let body = await_next_message_body()?;
    let body_string = format!("client {}", String::from_utf8(body)?);
    let args = body_string.split(' ');
    match Args::try_parse_from(args)?.command {
        Some(Command::PutFile {
            ref host,
            ref path,
            name,
        }) => handle_put_file(
            host,
            path,
            &name.unwrap_or_else(|| path.split('/').last().unwrap().to_string()),
        )?,
        Some(Command::GetFile { ref host, ref name }) => handle_get_file(host, name)?,
        Some(Command::ListFiles { ref host }) => handle_list_files(host)?,
        None => {}
    }
    Ok(())
}

call_init!(init);
fn init(_our: Address) {
    match execute() {
        Ok(_) => {}
        Err(e) => println!("error: {e:?}"),
    }
}
}

Remote File Storage Usage Example

Build

# Start fake node to host server.
kit f

# Start fake node to host client.
kit f -o /tmp/kinode-fake-node-2 -p 8081 -f fake2.dev

# Build & start server.
## Note starting is required because we need a deployed copy of server's API in order to build client.
## Below is it assumed that `kinode-book` is the CWD.
kit bs src/code/remote_file_storage/server

# Build & start client.
## Here the `-p 8080` is to fetch deps for building client (see the metadata.json dependencies field).
kit b src/code/remote_file_storage/client -p 8080 && kit s src/code/remote_file_storage/client -p 8081

An alternative way to satisfy the server dependency of client:

## The `-l` satisfies the dependency using a local path.
kit b src/code/remote_file_storage/client -l src/code/remote_file_storage/server

Usage

# In fake2.dev terminal:
## Put a file onto fake.dev.
client:client:template.os put-file fake.dev -p client:template.os/pkg/manifest.json -n manifest.json

## Check the file was Put properly.
client:client:template.os list-files fake.dev

## Put a different file.
client:client:template.os put-file fake.dev -p client:template.os/pkg/scripts.json -n scripts.json

## Check the file was Put properly.
client:client:template.os list-files fake.dev

## Read out a file.
client:client:template.os get-file fake.dev -n scripts.json
Get Help: