Documentation

Record format, version 1

Summary. Record format version 1 is the contract of every sealed hour. It states where the files of a sealed hour sit in storage you own, and how time, JSON and hashes are encoded. It defines the raw file, the normalized tables, and the hourly and daily manifests that form the hash chain. It lists the ten checks a verifier makes. Within version 1, only optional manifest fields and new columns may be added.

View as Markdown

What follows is the format document itself, unedited apart from its title.

Status: draft. This document states what a segment contains and what a verifier checks. It is the public contract of format major 1.

Layout

<prefix>/recorder=<recorder>/venue=<venue>/channel=<channel>/date=YYYY-MM-DD/
  manifest.json          daily manifest
  manifest.json.sig      detached signature over the daily manifest
  hour=HH/
    raw.jsonl.zst        every frame exactly as received
    <table>.parquet      normalized tables of the channel
    manifest.json        hourly manifest
    manifest.json.sig    detached signature over the hourly manifest

Channels: trades, book, funding, liquidations. A frame belongs to the UTC hour of its frame time ("Raw file", below).

Encodings

• Time. int64 nanoseconds since the Unix epoch. In JSON, an RFC 3339 UTC string with the Z designator and exactly nine fractional digits (2026-10-03T14:00:00.000000000Z); one instant has one encoding.
• JSON. Manifests and raw-file lines are RFC 8785 canonical JSON. Numbers are integers within ±(2^53 − 1); every other value is a string.
• Hashes. BLAKE3, 32 bytes, lowercase hexadecimal.

Raw file

A frame's frame time is the earliest receive_time among its receipts that carry clock_error_bound, or among all its receipts when none does. Frames are ordered and assigned to hours by frame time, and normalized rows carry it in receive_time.

raw.jsonl.zst is zstd-compressed; each line is one envelope, in order of frame time:

Each receipt: replica (a or b), connection (opaque), receive_time, clock_offset (the receiving host clock's estimated offset from UTC, in signed integer nanoseconds, positive when the host clock is fast), and clock_error_bound (a non-negative integer of nanoseconds bounding the absolute error of receive_time against UTC, below). Both are absent when no estimate existed or the replica withheld it (time daemon unsynchronized, sample older than 16 s, clock stepped, or source not NTS-authenticated), which the manifest states.

Clock calibration

• Point of timestamping. receive_time is read when the replica holds the complete frame from its connection (after transport decryption and frame reassembly), for every venue.
• Bound. clock_error_bound = abs(clock_offset) + root dispersion + root delay / 2 + 15e-6 × the sample's age + 1.25 ns, rounded up to the next nanosecond, from the time daemon's latest sample: the absolute offset, plus RFC 5905's root distance (root dispersion plus half the root delay), plus dispersion growth at RFC 5905's frequency tolerance (PHI, 15 ppm), plus 1.25 ns for the nanosecond resolution at which the sample states its offset, root delay and root dispersion (0.5 ns each, halved for the root delay). The age is measured on a monotonic clock from the moment the replica starts reading the sample.
• Conditions. The bound holds when the upstream reference is correct, the host clock's frequency error stays within 15 ppm, the clock is not stepped between the sample and the receipt, and the time daemon reports itself synchronized. A replica withholds clock_offset and clock_error_bound while the time daemon reports itself unsynchronized, when the latest sample is older than 16 s, for a receipt whose sample did not come from a source authenticated with Network Time Security (NTS, RFC 8915), and after a step until the time daemon has updated the clock from a measurement taken after the step.
• Kind of bound. The bound is a maximum-error bound under these conditions, not a probabilistic interval, and no coverage factor is claimed.
• **receive_time and exchange_time.** exchange_time is the venue's stated time for the frame; the venue's clock carries no bound in this record, so the difference between the two times is not a one-way network delay and can be negative.

Normalized tables

Parquet, zstd-compressed, one per table even when it has no rows. Every row references its frame by raw_index, the zero-based line index in the hour's raw.jsonl.zst, and carries that frame's frame time in receive_time. Values are exactly as the venue sent them: no rounding, no unit or currency conversion, no inference. Venue fields the tables do not define stay in the raw file.

Tables: trades.parquet (channel trades); book_snapshots.parquet (the snapshots the venue sent, one row per level) and book_deltas.parquet (one row per changed level; a size of zero removes the level) (channel book). A frame with a value that DECIMAL(38, 18) cannot hold exactly, or a frame of a row-carrying type that does not parse as the venue's documented structure, yields no rows; the hourly manifest lists its line index in unrepresentable_frames, and the hour is partial with reason unrepresentable_value.

Hourly manifest

Each entry of reconnect_windows carries start_time, end_time (exclusive) and, when the venue closed the connection that the window follows with a close frame carrying a status code, close_code: that frame's status code (RFC 6455, section 7.4). A window that continues from the previous hour keeps its close_code. The verifier does not check close_code.

Each entry of gap_free_runs (channel trades) states an interval during which one connection of the replica was subscribed and the replica detected no gap in the venue's trade sequence for one instrument: instrument, connection, start_time, and end_time (exclusive). A run's end_time is 1 ns after the connection's last frame before the first event that breaks the run (a disconnection, a gap detected in the venue's trade sequence for that instrument, a checksum failure, or a resubscription), or the end of the hour; the next run starts where the break was detected, so each break lies between two stated times. A run states that the replica detected no gap, not that none occurred. gap_free_runs is absent when the replica states no run. The verifier checks their references (check 9), not the venue sequences they rest on, and checks conflicts by reference (check 10).

Each replica with receipts in the raw file carries clock: source (nts; present exactly when error_bound_max is), error_bound_max (the largest clock_error_bound of the replica's receipts), and unbounded (the count of its receipts without one); source and error_bound_max are absent when every receipt of the replica is unbounded. A replica with no receipts in the hour has no clock. The reason clock_offset_unknown means that at least one frame has no receipt with clock_offset; a replica's unbounded can be nonzero without it.

Every hour has a manifest, including an hour with no data, so the chain never skips an hour. Completeness reasons: reconnect, sequence_gap, checksum_failure, clock_offset_unknown, journal_torn, journal_corrupt, recorder_down, unrepresentable_value, destination_unavailable.

Daily manifest

format_version, recorder, venue, channel, date, hours (each hour, blake3, status, in order) and previous_manifest_blake3 (the previous day's daily manifest, null at the start).

Signatures

manifest.json.sig is a detached JWS (RFC 7515) with an unencoded payload (RFC 7797) over the exact bytes of manifest.json: protected header alg Ed25519 (RFC 9864), kid, b64 false, crit ["b64"]. Keys are published as a JWK Set (RFC 7517) of OKP keys (RFC 8037) with RFC 7638 thumbprints as kid.

What a verifier checks

1. Every manifest is canonical JSON with format_version 1; an unknown major is rejected.
2. Every listed file exists with the listed size and BLAKE3 hash.
3. The raw file decompresses; every line is a canonical envelope; frame times lie in the hour and never decrease; the line count equals the listed rows.
4. Every table of the channel is listed, has the v1 schema and the listed row count, and each row's raw_index exists and never decreases, and its receive_time equals the frame time of its frame.
5. Hours are consecutive and each manifest carries the previous manifest's hash; the same holds for days, and each daily manifest lists exactly its day's hourly manifests. Copies of one channel in several directories are one chain; a manifest held twice with different bytes is a fork and fails.
6. Every signature verifies against a key valid at signing time.
7. Each receipt carries clock_error_bound exactly when it carries clock_offset, and the bound is at least the absolute value of the offset.
8. Each replica with receipts carries clock, and no other replica does; its values equal those recomputed from the raw file; the reasons include clock_offset_unknown exactly when some raw-file line has no receipt with clock_error_bound.
9. Each entry of gap_free_runs lies within the hour and ends after it starts, and its connection appears in a receipt of its replica.
10. Each conflicts line's raw_index exists, its blake3 is the hash of that raw-file line's bytes without its terminating newline, U+000A, and every line of one conflict carries its trade_id.

Within major 1, only optional manifest fields and new columns may be added; readers ignore fields they do not know.