Serialization considered boring
Open a 34 MB value in a konserve store and increment one counter inside it:
require('[konserve.mmap :as kmm])
kmm/update-in!(store,
[:report :region "us-west" :views], inc,
{:durability :checked})
(require '[konserve.mmap :as kmm])
(kmm/update-in! store [:report :region "us-west" :views] inc
{:durability :checked})
The call writes one page and returns. The rest of the value is never decoded, never rebuilt as objects, never re-encoded and never rewritten. The ordinary update-in reads the whole value, reconstructs it, applies the function, encodes everything and writes it all back, and its cost grows with the value even when the change is one number.
The difference comes from the serialization format. This post is about that format, boring, and about a choice we made on purpose: the dullest standard available, plain CBOR, with no dialect on top. The usual price of that choice is speed, and the work in boring went into not paying it. What the choice buys is readers in other languages, stored values that outlive the code that wrote them, and the property the example above depends on, a value you can read into without decoding it.
A format other programs can read
CBOR, Concise Binary Object Representation, is RFC 8949, IETF STD 94. Like JSON it represents maps, arrays, strings and numbers; unlike JSON it carries binary data and tags that name additional types. Libraries exist in most languages.
boring maps Clojure values onto it. The basic API is an encoder and a decoder:
require('[boring.core :as boring])
def encoded: boring/encode({:user/name "Ada", :scores [99 100], :tags #{:x :y}})
boring/decode(encoded)
;; => {:user/name "Ada", :scores [99 100], :tags #{:x :y}}
(require '[boring.core :as boring])
(def encoded
(boring/encode
{:user/name "Ada" :scores [99 100] :tags #{:x :y}}))
(boring/decode encoded)
;; => {:user/name "Ada", :scores [99 100], :tags #{:x :y}}
Keywords, symbols, sets, ratios, records and metadata round-trip on the JVM and in ClojureScript. Where a runtime has no matching type, the value travels as a tagged representation that keeps its meaning for another reader. The compatibility guide lists the mappings and their limits.
A Python program parses the bytes with cbor2. It still has to interpret the tags for Clojure-specific values: seeing a map does not give Python a keyword or a record type. The format and the mappings are documented, so that interpretation can be written without running the Clojure application that produced the data.
That is the case for reach, and it is stronger for data than it ever was for code. Code runs in a world you control. Bytes written to storage go into an open one: the reader may be another team, another language, a cache a later service inherits, or nobody for five years. Data outlives the application, usually the platform, and often the ability to run the program that wrote it. A format only its own writer can read has encoded that constraint into every byte.
Clojure’s existing serializers made a different trade, mostly without saying so. Nippy and hako are fast and JVM-only. Fressian is portable across Clojure and speaks to nothing else. Transit was designed for reach, and its README states the bet plainly:
Transit is intended primarily as a wire protocol for transferring data between applications. If storing Transit data durably, readers and writers are expected to use the same version of Transit and you are responsible for migrating/transforming/re-storing that data when and if the transit format changes.
The distinction that matters is control, not transport versus storage, and its boundary sits at I/O: once bytes leave the process, the writer no longer decides whether they are read now or in five years. A client caches a response, a queue retains a message, a log is replayed, a browser keeps what it fetched. Datahike is the sharpest case we know: the same index block travels over kabel to a browser, which stores it in IndexedDB as a replica. The bytes are wire and archive at once.
Transit’s caveat is therefore a statement about coordinated deployment. It holds exactly as long as one team updates every reader, which is true inside a single deployment and false at the first boundary beyond it, and no format can enforce it. boring’s compatibility policy takes the other side: it fixes the meaning of every released tag, requires newer readers to decode older output, and checks both against a corpus of frozen encodings in the test suite. Whether a consumer stores the bytes is not the writer’s decision, so the format assumes they will.
Reach did not cost speed
The usual trade is that a portable format is a slow one. boring’s performance notes publish every table with the payloads and commands that reproduce it, taken on a quiet machine with criterium, a warm process rather than a warm cell, and the same object-reuse tier for every library. They show the trade not being paid.
On nippy’s own benchmark, its stress data, its filter and its timing loop, boring round-trips in 1,662 µs against nippy/fast at 1,659: decoding 13% faster, encoding 1.25× behind, at parity overall with the fastest JVM Clojure codec on that codec’s home ground, and 5× ahead of Fressian. Put both behind a compressor, which every storage layer and transport does, and boring with zstd is 1.6× smaller than nippy’s default output.
hako is the one codec that leads boring on the JVM, and it is built for speed on java.lang.foreign. With both libraries reusing their readers, it is ahead on map-heavy payloads by 1.2 to 1.8×, boring is ahead on mixed data and on nested vectors by 2×, and boring allocates less heap on nearly every encode. No codec wins every cell, and the losses on either side are fractions of the other’s time, not multiples.
ClojureScript is the exception. transit-cljs writes JSON text, so JSON.parse does the byte walk in native code, and it decodes generic data faster than boring’s JavaScript parser. boring is smaller on every payload measured, and with :shapes it decodes the datom-shaped tables it was built for faster than transit.
Serialization rarely decides a workload. Where it does, a few percent deciding an application, you are past what any self-describing format offers and into a fixed-schema, zero-copy layout such as FlatBuffers, with a compiled schema and no open world. Everywhere else the reach costs little in these measurements, and what remains to choose on is what the format lets you do with the bytes.
Reading into a value without decoding it
A CBOR value is self-describing: every item carries its own type and count, so a reader can step over an item it does not want. boring adds an optional offset index at the end of a document, an ordinary named tag that any other CBOR reader skips, which turns a walk into a jump. boring.nav builds on both. It is a read-only cursor into the encoded bytes that supports get-in, count, nth, seq and reduce, and it decodes a value only when you ask for it.
hako’s read API is decode, decode-into! and decode-many; nippy’s is thaw. Neither has a partial read, a cursor or an early stop. To see one field of one row, both must build every row. That is the question a store asks, and answering it is what the cursor exists for. Measured on a 5,000-row table of datom-shaped maps, {:e :a :v :tx :added}, encoded with shapes and stringref, with clojure -M:bench -m capability:
| 5,000 rows, 5 fields | boring | hako | nippy |
|---|---|---|---|
| size | 134,465 B | 138,766 B | 273,865 B |
| one field of one row | 0.94 µs | 236 µs | 1,005 µs |
| sum one column | 110 µs | 267 µs | 1,070 µs |
| heap allocated, one field of one row | 4,048 B | 987,344 B |
The point read scales and the column scan does not. Against hako the point read is 16.8× at 200 rows, 75× at 1,000 and 251× at 5,000, because the cursor is O(log n) in the index and a decoder is O(n) in the rows. The column scan is a flat 2.4× at every size, because both sides visit every row; what boring saves there is the allocation, not the walk.
Most of that win is navigation, not the codec, and the harness says so: boring’s own full decode loses the column scan to hako. A table that left out that row would be claiming a codec advantage boring does not have.
One field from a stored value
With konserve’s file store the cursor reads through a memory mapping. store is an open file store holding a boring-encoded value under "customers":
require('[konserve.mmap :as kmm], '[boring.nav :as nav])
kmm/with-mmap-value([customers store "customers"],
nav/value(get-in(customers, ["customer-137" "name"])))
(require '[konserve.mmap :as kmm]
'[boring.nav :as nav])
(kmm/with-mmap-value [customers store "customers"]
(nav/value (get-in customers ["customer-137" "name"])))
The other customers are never built. The cursor is valid inside with-mmap-value; decode what you need before leaving the scope.
An indexed document is a CBOR sequence: the data, then the index items. A sequence-aware reader parses all of them; a reader that wants only the data reads the first item; code that insists on exactly one item must account for the trailing index. The index documentation explains when the jumps repay the index space, which is roughly when a document has more than a few hundred items before the fields you read.
One field from each of many values
konserve-lmdb stores keys in order, so a prefix names a set of records, and it exposes the cursor as project and project-reduce. The store’s range picks the rows; boring’s navigator picks the fields; neither decodes what the calculation does not use.
require('[konserve-lmdb.store :as s])
def store: s/connect-store("/path/to/customers", :navigable?, true, :index, 16)
s/project(store, {:prefix ["user"]}, [:profile :address :city])
s/project-reduce(store,
{:prefix ["user"]},
[[:profile :address :city] [:revenue]],
fn [totals _key [city revenue]]: update(totals, city, fnil(+, 0), revenue) end,
{})
(require '[konserve-lmdb.store :as s])
(def store
(s/connect-store "/path/to/customers" :navigable? true :index 16))
(s/project store {:prefix ["user"]} [:profile :address :city])
(s/project-reduce
store
{:prefix ["user"]}
[[:profile :address :city] [:revenue]]
(fn [totals _key [city revenue]]
(update totals city (fnil + 0.0) revenue))
{})
This still visits every record in the range. What it skips is the decoding, and the amount skipped grows with the record. In konserve-lmdb’s published measurement, 300 records whose probed field sits behind a padding array, on ZFS with the CPU governor pinned, projecting one field costs 3.61 ms at 400 padding elements and 1.48 ms at 3,000: the document grew and the read got cheaper, because the index stops the cost tracking document size at all. PostgreSQL’s JSONB, on the same records and machine, takes 24.34 ms and 73.07 ms, measured as the planner’s own execution time. JSONB’s navigation is structurally as good as ours or better; it loses because every path extraction detoasts the whole document first. The same README says where this does not apply: documents of a kilobyte or two read one field at a time, where JSONB is a reasonable choice and this store has no structural advantage.
Both projection calls refuse on a store that is not navigable rather than quietly decoding everything, because a silent fallback would turn the point of the call into a performance mystery.
Editing a field in place
The opening example is the third use of the same cursor. konserve’s file-store API locates a field, encodes its replacement and updates the stored bytes, choosing among three edits by what changed:
| Change | Work |
|---|---|
| Same encoded length | Overwrite that span; existing offsets stay valid |
| A leaf changes length | Encode the leaf, move the bytes after it, adjust the index |
| A key added or removed | Re-encode the parent container and update the surrounding bytes and index |
An increment does not always keep its length, since CBOR integers widen at encoding boundaries; the library picks the path. The default durability mode, :rename, writes a replacement file and renames it into place. That still copies the file, but it maintains the index rather than rebuilding it and never decodes the value. :durability :checked edits the live file, so a same-length edit costs one page write whatever the value’s size, provided the value is indexed. On a 33.8 MB value, konserve’s in-place editing guide reports the :checked update-in! at 70 to 360 times the ordinary update-in; the guide states the value size and the comparison but not yet the hardware, so read the range as the order of magnitude rather than a benchmark result.
The trade is recovery. :checked leaves a dirty marker if an edit is interrupted, and the application must check kmm/torn? and reconstruct the value; the marker neither repairs it nor blocks reads. Concurrent readers of the same key can observe half-written bytes, so coordinate them or keep the default. In-place writes need JDK 22 or later for java.lang.foreign; on older JDKs the operation rewrites the file, and a value that is not editable falls back to the ordinary konserve operation. Editing also requires a deterministic encoding without string references, which the :archival profile provides; the setup guide shows the configuration. This part of the stack is experimental and file-store only.
These edits apply to application-owned konserve values. They are not a way to mutate Datahike’s content-addressed index blocks, whose immutability is part of the database’s storage contract.
The columnar trick, still inside the format
One more thing a fast binary format usually makes you trade reach for is columnar packing. Protocol Buffers and FlatBuffers get their compactness by fixing a schema out of band: field names live in a compiled .proto or .fbs, and each record stores only values at known positions. The bytes are unreadable without the schema.
boring does the same packing without leaving self-describing CBOR. With :shapes true, an array of maps sharing a key set is written with the keys once and each row reduced to its values:
[{:e 1, :a :x} {:e 2, :a :y}]
;; tag 39649 [ [:e :a]
;; [[1 :x] [2 :y]] ]
[{:e 1 :a :x} {:e 2 :a :y}]
;; tag 39649 [ [:e :a]
;; [[1 :x] [2 :y]] ]
The key set rides inside the document, so a foreign reader still decodes it as a tagged array and no schema file or code generation is involved. On the published datom-maps-200 fixture, shapes cut the encoding from 9,952 to 4,982 bytes; with stringref on top, 5,063 bytes against hako’s 5,165 for the same table. The two options compose, and the shaped array is one of the three tags the cursor can descend into, so packing and navigation combine.
Two boundaries. A general-purpose compressor feeds on exactly the repetition shapes remove, so under zstd the size advantage largely returns to zstd; the gain is in raw size and decode speed. And tag 39649 is provisional: boring emits it, it is not yet in the IANA registry, and the map-of-maps generalisation is specified but not shipped.
Nothing here left the format
String deduplication is stringref, CBOR tags 25 and 256, a registered extension. The offset index is a named tag 27 item, CBOR’s registered extension point for named types, that every other reader skips. Shaped arrays are one tag number. A file boring writes is a file cbor2 reads. The reach was never spent to buy the speed or the navigation, which is the reason the two libraries above could build on it without a private format of their own.
Try boring
org.replikativ/boring
{:mvn/version "0.1.30"}
org.replikativ/boring {:mvn/version "0.1.30"}
Start with the encode and decode example. The boring repository links the type mappings, the compatibility tests and the reading APIs. If you already use konserve, the file-store guide and the LMDB projection guide apply the cursor to stored data.
Use CBOR by default, and reach for something else only for a reason you would still defend in five years, to whoever is holding your bytes then.