Mix.install([{:slap, "~> 0.7.1"}, {:kino, "~> 0.19.1"}])About this notebook
Follow a KV write through the cluster's processes, messages and stored
data. Then inspect a stream append and crash a shard to check recovery.
Kino.Process draws the supervision trees and message traces.
Requires Elixir 1.18 or later and Erlang/OTP 26.2 or later. Run the cells in order.
The storage layouts shown here are internal and can change between versions. Use the packages' APIs to read and write application data; the getting started notebook introduces them.
Start the clusters
Use a local store so you can inspect its files. Two shards per cluster and two partition writers per KV shard keep the diagrams small. The first two lines limit SlateDB's and Elixir's logs to warnings and errors:
Slap.SlateDB.set_log_level(:warning)
Logger.configure(level: :warning)
dir = Path.join(System.tmp_dir!(), "slap-#{System.unique_integer([:positive])}")
{:ok, slap} =
Slap.start_link(
store: {:local, dir},
streams: [shards: 2],
kv: [shards: 2, child_options: [partition_writers: 2]]
)Run this cell only once: running it again fails with
{:error, {:already_started, pid}}. To start over, restart the runtime
from Livebook's Runtime panel and run the cells in order.
KV processes
Draw the KV cluster's supervision tree. Solid lines show supervision;
dotted lines show process links. Here, the Registry links to the
processes registered in it.
Kino.Process.sup_tree(Slap.KV.Cluster.Supervisor, direction: :left_right)The tree contains:
Slap.Cluster.Probechecks the store's conditional writes, which support writer fencing. It starts the placement strategy and exits, so the diagram shows it as not started.StrategySupervisorruns the placement strategy,Slap.Cluster.Strategy.Local, which assigns shards to this node.Slap.Cluster.Hostopens and closes the assigned shards.Registrymaps each shard's process names to pids.ShardSupervisorsruns a supervisor for each shard. Each shard has aSlap.Cluster.ShardDbprocess that owns its SlateDB database,Slap.KV.Admissionto limit work awaiting durability, and partition writers to handle requests.
List the registry's names and pids. Use this table to identify processes in the message traces below:
registered = fn registry ->
Registry.select(registry, [{{:"$1", :"$2", :_}, [], [{{:"$1", :"$2"}}]}])
end
registry_table = fn entries, name ->
rows = for {key, pid} <- Enum.sort(entries), do: %{key: inspect(key), pid: inspect(pid)}
Kino.DataTable.new(rows, name: name, keys: [:pid, :key])
end
Slap.KV.Cluster.Registry
|> registered.()
|> registry_table.("KV registry")Row placement
A partition selects a shard. All rows in that partition share the shard:
partitions = ~w(users orders carts sessions)
Map.new(partitions, &{&1, Slap.KV.Cluster.shard_for(&1)})Within a shard, one partition writer handles all writes for a given
partition. assignments/0 shows each shard's owner. The shards here are
local, so the result includes their ShardDb pids:
Slap.KV.Cluster.assignments()Trace a KV write
Kino.Process.seq_trace/3 captures messages sent during a function call.
Write a warm-up row first so module loading does not add code-server
traffic to the trace:
{:ok, _} = Slap.KV.put("users", "warm-up", "x")
label = fn
{:"$gen_call", _from, {_owner, request}} when is_tuple(request) ->
{:ok, "CALL: #{elem(request, 0)}"}
{tag, _reply} when is_reference(tag) or is_list(tag) ->
{:ok, "REPLY"}
_message ->
:continue
end
{{:ok, version}, diagram} =
Kino.Process.seq_trace(:all, fn -> Slap.KV.put("users", "42", "Ada") end,
message_label: label
)
diagramUse the registry table to identify the pids, then follow the write:
- The caller sends
writeto the partition writer forusers. - The writer reserves capacity with the shard's admission process
(
reserve). - It marks the reservation
submitting, writes to SlateDB, and reports the resulting sequence number (submitted). It then asksShardDbfor a durability notification (notify). - Once the write is durable, the writer releases its reservation
(
release) and replies to the caller.
The SlateDB write itself is a native call, so it does not appear as a message in the trace.
How rows are stored
Each shard is a SlateDB database. A KV key encodes the partition's byte length as a 16-bit integer, followed by the partition and row key. This keeps each partition's rows together, ordered by row key. Values are stored unchanged; a row's version is SlateDB's sequence number for its write.
Read the users shard directly and decode its keys:
kv_rows = fn shard ->
{:ok, {:local, shard_ctx}} = Slap.KV.Cluster.lookup(shard)
for {<<length::16, partition::binary-size(length), key::binary>>, value} <-
Slap.SlateDB.scan(shard_ctx.db) do
%{partition: partition, key: key, value: value}
end
end
{:ok, _} = Slap.KV.put("orders", "1001", "a book")
users_shard = Slap.KV.Cluster.shard_for("users")
{:ok, %{value: "Ada", version: ^version}} = Slap.KV.get("users", "42")
Kino.DataTable.new(kv_rows.(users_shard), name: "KV shard #{users_shard}")The scan covers the whole shard, so the table can include other partitions that map to it.
Each cluster has its own directory in the store. Within each shard's directory, SlateDB uses these paths:
| Path | Contents |
|---|---|
wal/ | The write-ahead log, encoded as SST files. |
manifest/ | The database's state. |
compactions/ | The compactor's state. |
compacted/ | Data stored as SST files. |
Each cluster also records its shard count in _cluster/layout. List the
files created so far:
dir
|> Path.join("**")
|> Path.wildcard()
|> Enum.reject(&File.dir?/1)
|> Enum.map(&Path.relative_to(&1, dir))
|> Enum.join("\n")
|> IO.puts()Streams
An active stream has one stream server on its shard. Create a stream and append a message, then list the registry entries for the stream's shard:
{:ok, :created, _} = Slap.Streams.create("/events", content_type: "text/plain")
{:ok, _} = Slap.Streams.append("/events", "created")
stream_shard = Slap.Streams.Cluster.shard_for("/events")
Slap.Streams.Cluster.Registry
|> registered.()
|> Enum.filter(&match?({{^stream_shard, _name}, _pid}, &1))
|> registry_table.("Streams shard #{stream_shard} registry")The {:stream, "/events"} entry names the stream server. The registry
also lists the stream supervisor and background jobs, including expiry
and deletion.
Trace another append: the stream server writes the data and replies after
it becomes durable.
{{:ok, %{result: :appended}}, diagram} =
Kino.Process.seq_trace(:all, fn -> Slap.Streams.append("/events", "updated") end)
diagramInspect the stream's metadata, tail and message rows. Metadata is keyed by the stream's path; tail and message keys use its stream id. The tail is the offset after the last message. Large messages span numbered parts, each stored in a separate row.
Each key starts with a byte identifying its kind. Decode that byte and the remaining fields to label the rows:
{:ok, {:local, stream_ctx}} = Slap.Streams.Cluster.lookup(stream_shard)
describe = fn
<<0, "next_sid">> -> "the next stream id"
<<1, path::binary>> -> "the metadata of #{path}"
<<2, sid::64>> -> "the tail of stream #{sid}"
<<4, sid::64, offset::64, part::16>> -> "stream #{sid}'s message at #{offset}, part #{part}"
<<kind, _rest::binary>> -> "a row of kind #{kind}"
end
Kino.DataTable.new(
for({key, value} <- Slap.SlateDB.scan(stream_ctx.db),
do: %{row: describe.(key), bytes: byte_size(value)}
),
name: "Streams shard #{stream_shard}"
)Shard recovery
Kill the ShardDb process for the users shard. The host reports the
crash to the local placement strategy, which reopens the shard. The cell
output includes the error and warning that Slap.KV.Cluster logs for the
crash.
The cell waits for the cluster's [:slap, :cluster, :shard, :start]
telemetry event, then checks that the reopened shard has a new pid:
notebook = self()
:ok =
:telemetry.attach(
"internals",
[:slap, :cluster, :shard, :start],
fn _event, _measurements, metadata, _config ->
send(notebook, {:shard_started, metadata.cluster, metadata.shard})
end,
nil
)
%{^users_shard => {:local, old_shard_db}} = Slap.KV.Cluster.assignments()
Process.exit(old_shard_db, :kill)
started =
receive do
{:shard_started, Slap.KV.Cluster, ^users_shard} -> :ok
after
30_000 -> :timeout
end
:ok = :telemetry.detach("internals")
:ok = started
:ok = Slap.KV.Cluster.await_ready(30_000)
%{^users_shard => {:local, new_shard_db}} = Slap.KV.Cluster.assignments()
true = new_shard_db != old_shard_dbThe traced put returned only after its write was durable. Reopening
the database must preserve both the value and its version:
{:ok, %{value: "Ada", version: ^version}} = Slap.KV.get("users", "42")Draw the tree again and compare it with the first one to identify the shard's replacement processes:
Kino.Process.sup_tree(Slap.KV.Cluster.Supervisor, direction: :left_right)Clean up
Stop the Slap supervisor to close both clusters' databases, then remove the local store:
:ok = Supervisor.stop(slap, :normal, 60_000)
{:ok, _} = File.rm_rf(dir)Next steps
- The architecture guide
shows what
Slapstarts and how the store is laid out. - Slap Inspect browses a running node's streams, rows and files from a terminal.