Development and testing

Copy Markdown View Source

These commands require a repository checkout. Follow the repository's development setup, then run commands from slap/ unless a different directory is specified.

Development server

Use the slap.server Mix task to start a server for this checkout's conformance suite, benchmarks, and crash and cluster tests. It starts the selected services and an HTTP listener, with server options passed as flags. The task is available only in the checkout's :dev and :test environments. To try the published HTTP APIs, use the standalone server script.

mix slap.server --streams --kv --store memory

Run mix slap.server --help for the full list of options, or iex -S mix slap.server --streams --kv --store memory to keep an IEx prompt while the server runs. Run one server per Erlang VM; a cluster can span several Erlang VMs on one or several hosts.

Examples

To try the getting started notebook against this checkout, start iex -S mix and paste its Elixir cells, skipping the Mix.install setup cell. To check both notebooks without Livebook, as CI does, run:

mix run scripts/livebook_test.exs notebooks/getting-started.livemd
mix run scripts/livebook_test.exs notebooks/internals.livemd

The script skips the Mix.install setup cell and runs the remaining cells in order. Cells match their expected results, so an unexpected result fails the check. The script also checks sequence diagrams for unlabeled messages.

Durable Streams conformance and benchmarks

Conformance

From the repository root, run:

just streams-conformance

The recipe starts a memory server, waits for its HTTP listener, runs the official conformance suite, and stops the server afterward. CI runs the suite on memory and RustFS stores. It covers stream operations, including forks, and skips subscriptions, which this server does not provide.

HTTP benchmarks

The official HTTP benchmarks measure append-to-long-poll round-trip latency and append throughput for 100-byte and 1 MB messages. Run them against Slap and the official Caddy server from the repository root:

just streams-bench

This compares slap-local and caddy-file. To include memory stores:

just streams-bench slap-memory caddy-memory slap-local caddy-file

scripts/streams_bench.sh runs each server in turn and durable_streams/summary.js compares the results. The Caddy binary is pinned and checksum-verified. Both the conformance and benchmark runners use DURABLE_STREAMS_URL (default http://127.0.0.1:4437) when run directly from durable_streams/ with pnpm test or pnpm bench.

slap-local and caddy-file acknowledge appends after disk persistence. slap-memory waits for a SlateDB flush; caddy-memory does not.

slap-memory retains SlateDB objects in RAM, and deleting streams does not immediately release that memory. Allow for all data appended during the run plus SlateDB overhead. The throughput benchmark runs for a fixed duration, so faster machines need more memory.

The slap_streams_bench.yml workflow compares slap-local, slap-s3 on RustFS, and caddy-file, and charts the results of runs on main.

Mix.install startup

Check that the standalone server script starts outside a Mix project using this checkout's dependencies:

scripts/mix_install_test.sh

Crash test

scripts/crash_test.exs runs the slap.server Mix task as its own OS process. It:

  1. has 16 writers append to their own JSON streams (half with idempotent producers)
  2. kills the server with kill -9 at a random moment between 0.3 and 3 s
  3. restarts the server on the same store
  4. lets the producers retry the appends that were in flight, and all writers append more
  5. reads every stream back

Every acknowledged append must be present exactly once, at the offset its acknowledgement gave, so no offset is reused after the crash. Each writer's messages must be in order, with no message stored twice.

Run on local storage or an S3 prefix:

mix run scripts/crash_test.exs --runs 20
mix run scripts/crash_test.exs --runs 20 --store s3:s3://bucket/crash

CI runs the crash test on local storage and RustFS. An append in flight when the process dies may or may not be stored.

Streams load

bench/streams_http_load.exs measures append throughput and latency. Start a Streams server in one terminal:

mix slap.server --streams --store memory

Run the load generator in another terminal:

mix run bench/streams_http_load.exs --connections 128 --seconds 20

The values shown are the defaults. The script also takes --port (default 4437), --streams (the number of streams to append to, default 1000), and --bytes (the body size in bytes, default 256).

Use --rate to set the total appends per second across all connections. The default, --rate 0, sends the next append on each connection as soon as the previous append is acknowledged.

To measure an S3-compatible store, start the server with --store s3:s3://bucket/load and set its AWS_* variables.

Choose shard count and flush_interval from the store's sustained PUT rate: each active shard can write one WAL object per flush interval. Run the benchmark on the intended store before sizing a deployment.

Cluster tests

Conformance through a cluster

scripts/cluster.sh starts three server processes sharing a local store, using distributed placement. Run the conformance suite through node 2 on port 4442, then stop the nodes:

(
  set -e
  cluster_dir=$(mktemp -d)
  trap 'scripts/cluster.sh stop "$cluster_dir"' EXIT
  scripts/cluster.sh start "$cluster_dir"
  (
    cd durable_streams
    pnpm install --frozen-lockfile
    DURABLE_STREAMS_URL=http://127.0.0.1:4442 pnpm test
  )
)

Requests, long-polls and SSE streams for shards owned by another node are routed to that node.

Faults under load

scripts/cluster_crash_test.exs starts its own three-node cluster, runs 16 writers (half with idempotent producers), and checks the same data guarantees as the single-node crash test:

mix run scripts/cluster_crash_test.exs --runs 6 --placement distributed

The test alternates between two faults:

  • kill: SIGKILL a node, wait for failover, then restart it.
  • pause: SIGSTOP a node until the others take its shards, then resume it. SlateDB fences the old writer.

The test measures how long each shard is unavailable after a kill. With distributed placement, failover waits for node failure detection and the settle interval. With object-lease, it waits for the old lease to expire and for the shard to reopen. CI runs cluster conformance with distributed placement and tests both faults with distributed and object-lease.