# Record format, version 1 Summary: Record format version 1 is the contract of every sealed hour. It states where the files of a sealed hour sit in storage you own, and how time, JSON and hashes are encoded. It defines the raw file, the normalized tables, and the hourly and daily manifests that form the hash chain. It lists the ten checks a verifier makes. Within version 1, only optional manifest fields and new columns may be added. What follows is the format document itself, unedited apart from its title. Status: draft. This document states what a segment contains and what a verifier checks. It is the public contract of format major 1. ## Layout ``` /recorder=/venue=/channel=/date=YYYY-MM-DD/ manifest.json daily manifest manifest.json.sig detached signature over the daily manifest hour=HH/ raw.jsonl.zst every frame exactly as received .parquet normalized tables of the channel manifest.json hourly manifest manifest.json.sig detached signature over the hourly manifest ``` Channels: `trades`, `book`, `funding`, `liquidations`. A frame belongs to the UTC hour of its frame time ("Raw file", below). ## Encodings - **Time.** int64 nanoseconds since the Unix epoch. In JSON, an RFC 3339 UTC string with the `Z` designator and exactly nine fractional digits (`2026-10-03T14:00:00.000000000Z`); one instant has one encoding. - **JSON.** Manifests and raw-file lines are RFC 8785 canonical JSON. Numbers are integers within ±(2^53 − 1); every other value is a string. - **Hashes.** BLAKE3, 32 bytes, lowercase hexadecimal. ## Raw file A frame's **frame time** is the earliest `receive_time` among its receipts that carry `clock_error_bound`, or among all its receipts when none does. Frames are ordered and assigned to hours by frame time, and normalized rows carry it in `receive_time`. `raw.jsonl.zst` is zstd-compressed; each line is one envelope, in order of frame time: | Field | Type | Meaning | |---|---|---| | `payload` | string | The frame: verbatim text, or base64 of a binary frame | | `payload_encoding` | `text` or `base64` | How `payload` encodes the frame | | `exchange_time` | time, optional | Time the venue stated for the frame; the earliest when it states several | | `receipts` | array | One per replica that received the frame | Each receipt: `replica` (`a` or `b`), `connection` (opaque), `receive_time`, `clock_offset` (the receiving host clock's estimated offset from UTC, in signed integer nanoseconds, positive when the host clock is fast), and `clock_error_bound` (a non-negative integer of nanoseconds bounding the absolute error of `receive_time` against UTC, below). Both are absent when no estimate existed or the replica withheld it (time daemon unsynchronized, sample older than 16 s, clock stepped, or source not NTS-authenticated), which the manifest states. ## Clock calibration - **Point of timestamping.** `receive_time` is read when the replica holds the complete frame from its connection (after transport decryption and frame reassembly), for every venue. - **Bound.** `clock_error_bound` = abs(`clock_offset`) + root dispersion + root delay / 2 + 15e-6 × the sample's age + 1.25 ns, rounded up to the next nanosecond, from the time daemon's latest sample: the absolute offset, plus RFC 5905's root distance (root dispersion plus half the root delay), plus dispersion growth at RFC 5905's frequency tolerance (PHI, 15 ppm), plus 1.25 ns for the nanosecond resolution at which the sample states its offset, root delay and root dispersion (0.5 ns each, halved for the root delay). The age is measured on a monotonic clock from the moment the replica starts reading the sample. - **Conditions.** The bound holds when the upstream reference is correct, the host clock's frequency error stays within 15 ppm, the clock is not stepped between the sample and the receipt, and the time daemon reports itself synchronized. A replica withholds `clock_offset` and `clock_error_bound` while the time daemon reports itself unsynchronized, when the latest sample is older than 16 s, for a receipt whose sample did not come from a source authenticated with Network Time Security (NTS, RFC 8915), and after a step until the time daemon has updated the clock from a measurement taken after the step. - **Kind of bound.** The bound is a maximum-error bound under these conditions, not a probabilistic interval, and no coverage factor is claimed. - **`receive_time` and `exchange_time`.** `exchange_time` is the venue's stated time for the frame; the venue's clock carries no bound in this record, so the difference between the two times is not a one-way network delay and can be negative. ## Normalized tables Parquet, zstd-compressed, one per table even when it has no rows. Every row references its frame by `raw_index`, the zero-based line index in the hour's `raw.jsonl.zst`, and carries that frame's frame time in `receive_time`. Values are exactly as the venue sent them: no rounding, no unit or currency conversion, no inference. Venue fields the tables do not define stay in the raw file. | Column | Parquet type | Tables | |---|---|---| | `raw_index` | int64, required | all | | `instrument` | string, required | all | | `trade_id` | string, optional | `trades` | | `side` | string; optional in `trades` (the venue's word), required in the book tables (`bid` or `ask`) | all | | `price`, `size` | DECIMAL(38, 18) on FIXED_LEN_BYTE_ARRAY(16), required | all | | `exchange_time` | TIMESTAMP(NANOS, UTC) on int64, optional | all | | `receive_time` | TIMESTAMP(NANOS, UTC) on int64, required | all | Tables: `trades.parquet` (channel `trades`); `book_snapshots.parquet` (the snapshots the venue sent, one row per level) and `book_deltas.parquet` (one row per changed level; a size of zero removes the level) (channel `book`). A frame with a value that DECIMAL(38, 18) cannot hold exactly, or a frame of a row-carrying type that does not parse as the venue's documented structure, yields no rows; the hourly manifest lists its line index in `unrepresentable_frames`, and the hour is `partial` with reason `unrepresentable_value`. ## Hourly manifest | Field | Meaning | |---|---| | `format_version` | `1` | | `recorder`, `venue`, `channel` | Identity of the chain | | `interval` | The hour: `start_time`, `end_time` (exclusive) | | `files` | Each file of the hour directory: `name`, `size`, `blake3`, `rows` | | `frame_time_bounds` | Earliest and latest frame time, absent when there are no frames | | `sequence_gaps` | `instrument`, `after_id`, `before_id` for each break in a venue sequence | | `checksum_failures` | `instrument`, `receive_time` (the `receive_time` of the receipt whose replica found the mismatch), `expected`, `computed` for each venue checksum mismatch | | `unrepresentable_frames` | Raw-file line indices kept in the raw file only; absent when empty | | `replicas` | Per replica: `replica`, `frames`, `covered_ns`, `reconnect_windows`, `gap_free_runs`, and `clock` (all below) | | `replica_agreement` | Frames seen by `both` replicas and by `one_only`, and trade IDs in `conflict` | | `conflicts` | Each `trade_id` of an `instrument` that raw-file lines carry with different `side`, `price`, `size` or `exchange_time` (channel `trades`): `instrument`, `trade_id`, and `lines` (each `raw_index` and the `blake3` of the line's bytes without its terminating newline, U+000A); absent when there are none | | `completeness` | `status` (`complete`, `partial`, `missing`) and `reasons` | | `previous_manifest_blake3` | Hash of the previous hour's manifest bytes; null at the chain's start | Each entry of `reconnect_windows` carries `start_time`, `end_time` (exclusive) and, when the venue closed the connection that the window follows with a close frame carrying a status code, `close_code`: that frame's status code (RFC 6455, section 7.4). A window that continues from the previous hour keeps its `close_code`. The verifier does not check `close_code`. Each entry of `gap_free_runs` (channel `trades`) states an interval during which one connection of the replica was subscribed and the replica detected no gap in the venue's trade sequence for one instrument: `instrument`, `connection`, `start_time`, and `end_time` (exclusive). A run's `end_time` is 1 ns after the connection's last frame before the first event that breaks the run (a disconnection, a gap detected in the venue's trade sequence for that instrument, a checksum failure, or a resubscription), or the end of the hour; the next run starts where the break was detected, so each break lies between two stated times. A run states that the replica detected no gap, not that none occurred. `gap_free_runs` is absent when the replica states no run. The verifier checks their references (check 9), not the venue sequences they rest on, and checks `conflicts` by reference (check 10). Each replica with receipts in the raw file carries `clock`: `source` (`nts`; present exactly when `error_bound_max` is), `error_bound_max` (the largest `clock_error_bound` of the replica's receipts), and `unbounded` (the count of its receipts without one); `source` and `error_bound_max` are absent when every receipt of the replica is unbounded. A replica with no receipts in the hour has no `clock`. The reason `clock_offset_unknown` means that at least one frame has no receipt with `clock_offset`; a replica's `unbounded` can be nonzero without it. Every hour has a manifest, including an hour with no data, so the chain never skips an hour. Completeness reasons: `reconnect`, `sequence_gap`, `checksum_failure`, `clock_offset_unknown`, `journal_torn`, `journal_corrupt`, `recorder_down`, `unrepresentable_value`, `destination_unavailable`. ## Daily manifest `format_version`, `recorder`, `venue`, `channel`, `date`, `hours` (each `hour`, `blake3`, `status`, in order) and `previous_manifest_blake3` (the previous day's daily manifest, null at the start). ## Signatures `manifest.json.sig` is a detached JWS (RFC 7515) with an unencoded payload (RFC 7797) over the exact bytes of `manifest.json`: protected header `alg` `Ed25519` (RFC 9864), `kid`, `b64` false, `crit` `["b64"]`. Keys are published as a JWK Set (RFC 7517) of OKP keys (RFC 8037) with RFC 7638 thumbprints as `kid`. ## What a verifier checks 1. Every manifest is canonical JSON with `format_version` 1; an unknown major is rejected. 2. Every listed file exists with the listed size and BLAKE3 hash. 3. The raw file decompresses; every line is a canonical envelope; frame times lie in the hour and never decrease; the line count equals the listed rows. 4. Every table of the channel is listed, has the v1 schema and the listed row count, and each row's `raw_index` exists and never decreases, and its `receive_time` equals the frame time of its frame. 5. Hours are consecutive and each manifest carries the previous manifest's hash; the same holds for days, and each daily manifest lists exactly its day's hourly manifests. Copies of one channel in several directories are one chain; a manifest held twice with different bytes is a fork and fails. 6. Every signature verifies against a key valid at signing time. 7. Each receipt carries `clock_error_bound` exactly when it carries `clock_offset`, and the bound is at least the absolute value of the offset. 8. Each replica with receipts carries `clock`, and no other replica does; its values equal those recomputed from the raw file; the reasons include `clock_offset_unknown` exactly when some raw-file line has no receipt with `clock_error_bound`. 9. Each entry of `gap_free_runs` lies within the hour and ends after it starts, and its `connection` appears in a receipt of its replica. 10. Each `conflicts` line's `raw_index` exists, its `blake3` is the hash of that raw-file line's bytes without its terminating newline, U+000A, and every line of one conflict carries its `trade_id`. Within major 1, only optional manifest fields and new columns may be added; readers ignore fields they do not know.