kit run-tests

short: kit t

kit run-tests runs the tests specified by the given .toml file, or tests.toml, e.g.,

kit run-tests my_tests.toml

or

kit run-tests

to run the current working directory's tests.toml or the current package's test/.

Discussion

kit run-tests runs a series of tests specified by a .toml file. Each test is run in a fresh environment of one or more fake nodes. A test can setup one or more packages before running a series of test packages. Each test package is a single-process package that accepts and responds with certain messages.

Tests are orchestrated from the outside of the node by kit run-tests and run on the inside of the node by the tester core package. For a given test, the tester package runs the specified test packages in order. Each test package must respond to the tester package with a Pass or Fail. The tester package stops on the first Fail, or responds with a Pass if all tests Pass. If a given test Passes, the next test in the series is run.

Arguments

$ kit run-tests --help
Run Kinode tests

Usage: kit run-tests [PATH]

Arguments:
  [PATH]  Path to tests configuration file [default: tests.toml]

Options:
  -h, --help  Print help

Optional positional arg: PATH

Path to .toml file specifying tests to run; defaults to tests.toml in current working directory.

tests.toml

The testing protocol is specified by a .toml file. tests.toml, from core tests, will be used as an example:

runtime = { FetchVersion = "latest" }
# runtime = { RepoPath = "~/git/kinode" }
persist_home = false
runtime_build_release = false


[[tests]]
dependency_package_paths = []
setup_packages = []
setup_scripts = []
test_package_paths = ["key_value_test",  "sqlite_test"]
test_scripts = []
timeout_secs = 5
fakechain_router = 8545

[[tests.nodes]]
port = 8080
home = "home/first"
fake_node_name = "first.dev"
runtime_verbosity = 2

[[tests.nodes]]
port = 8081
home = "home/second"
fake_node_name = "second.dev"
runtime_verbosity = 2


[[tests]]
dependency_package_paths = []
setup_packages = []
test_package_paths = ["key_value_test"]
setup_scripts = []
test_scripts = []
timeout_secs = 5
fakechain_router = 8545

[[tests.nodes]]
port = 8080
home = "home/first"
fake_node_name = "first.dev"
runtime_verbosity = 2

The top-level of tests.toml consists of four fields:

runtime

Specify the runtime to use for the tests. Two option variants are supported. An option variant is specified with the key (e.g. FetchVersion) of a toml Table (e.g. {FetchVersion = "0.7.2"}).

The first, and recommended is FetchVersion. The value of the FetchVersion Table is the version number to fetch and use (or "latest"). That version of the runtime binary will be fetched from remote if not found locally.

The second is RepoPath. The value of the RepoPath Table is the path to a local copy of the runtime repo. Given a valid path, that repo will be compiled and used.

For example:

runtime = { FetchVersion = "latest" }

runtime_build_release

If given runtime = RepoPath, runtime_build_release decides whether to build the runtime as --release or not.

For example:

persist_home = false

persist_home

Whether or not to persist the node home directories after tests have been run. It is recommended to have this set to false except when debugging a test.

tests

An Array of Tables. Each Table specifies one test to run. That test consists of:

Each test package is a single-process package that accepts and responds with certain messages.

For example:

...
[[tests]]
dependency_package_paths = []
setup_packages = []
setup_scripts = []
test_package_paths = ["key_value_test",  "sqlite_test"]
test_scripts = []
timeout_secs = 5
fakechain_router = 8545

[[tests.nodes]]
...

nodes

Each test specifies one or more nodes: fake nodes that the tests will be run on. The first node is the "master" node that will orchestrate the test. Each node is specified by a Table. That Table consists of:

For example:

[[tests.nodes]]
port = 8080
home = "home/first"
fake_node_name = "first.dev"
runtime_verbosity = 2

[[tests.nodes]]
port = 8081
home = "home/second"
fake_node_name = "second.dev"

Test Package Interface

A test package is a single-process package that accepts and responds with certain messages. The interface is defined as:

interface tester {
    variant request {
        run(run-request),
    }

    variant response {
        run(result<_, fail-response>)
    }

    record run-request {
        input-node-names: list<string>,
        test-names: list<string>,
        test-timeout: u64,
    }

    record fail-response {
        test: string,
        file: string,
        line: u32,
        column: u32,
    }
}

world tester-sys-v0 {
    import tester;
    include process-v0;
}

A run request starts the test. A run response marks the end of a test, and is either an Ok Result, indicating success, or a Err Result with information as to where the error occurred.

In the Rust language, a helper macro for failures can be found in tester_lib.rs. The macro is fail!(): it automatically sends the Response as specified above, filing out the fields, and exits.