Serialization Considered Boring

August 2026

Open a 34-megabyte value in a konserve store. Change one number buried deep inside it:

require('[konserve.mmap :as kmm])

kmm/update-in!(store
  [:report :region "us-west" :views] inc
  {:durability :checked})

(require '[konserve.mmap :as kmm])

(kmm/update-in! store [:report :region "us-west" :views] inc
                {:durability :checked})

That call touches a page or two of memory and returns in microseconds. The value is 34 MB. The other 34 MB are never decoded, never rebuilt into objects, never re-encoded, and never rewritten. Against konserve’s ordinary update-in (which reads the whole value, reconstructs the object graph, applies your function, re-encodes, and writes it all back), this is 70 to 360× faster.

The reason is the serialization format, not a diffing trick layered on top of an opaque blob. This post is about that format, called boring, and about why making a serializer, of all things, boring turns out to unlock things a fast serializer usually cannot do.

What it is

boring is serialization for Clojure and ClojureScript in a format the rest of the world can already read. It is CBOR (IETF STD 94, a full Internet Standard since 2020), with implementations in 26 languages, an IANA tag registry, and a standard diagnostic notation. The same code runs on the JVM and in the browser, and it carries edn faithfully: keywords, symbols, sets, ratios, records, metadata all round-trip.

require('[boring.core :as boring])

boring/encode({:user/name "Ada", :scores [99 100], :tags #{:x :y}})
;; => #object["[B" ...]  58 bytes

boring/decode(*1)
;; => {:user/name "Ada", :scores [99 100], :tags #{:x :y}}

(require '[boring.core :as boring])

(boring/encode {:user/name "Ada" :scores [99 100] :tags #{:x :y}})
;; => #object["[B" ...]  58 bytes

(boring/decode *1)
;; => {:user/name "Ada", :scores [99 100], :tags #{:x :y}}

A Python, Rust, or Go program reads those bytes with its own CBOR library: cbor2, ciborium, fxamacker. A foreign reader that has never heard of a keyword still gets your data as ordinary CBOR.

The boring part is the point

Clojure is a hosted language on purpose. A language that only talks to itself is a silo no matter how good it is. That is why Clojure runs on the JVM, in the browser, and speaks to the libraries and platforms already there.

The community mostly made the opposite choice about serialization, and did not notice. nippy and hako are fast and JVM-only. fressian is portable across Clojure and speaks to nothing else. transit was designed for reach (no criticism there), but in practice its reach is Clojure, ClojureScript, and a short list of ports, several unmaintained, against a spec still at 0.8.

The argument for reach is stronger for data than it ever was for code, because code runs in a world you control and data does not. When you write bytes to storage you write to an open world: the reader may be another team, another language, a cache some later service inherits, or nobody at all for five years. And data outlives code. It outlives the application, usually the platform, and often the ability to run the program that wrote it. A format that can only be read by re-running your code has encoded the largest constraint of all.

transit’s own README is honest about which bet it makes:

Transit is intended primarily as a wire protocol for transferring data between applications. If storing Transit data durably, readers and writers are expected to use the same version of Transit and you are responsible for migrating/transforming/re-storing that data when and if the transit format changes.

That is a fine position for a wire protocol and a poor one for an archive, and an archive is what Datahike needed, which is why boring exists. A serialization format should be boring: unexciting, dependable, and readable by whoever is holding your bytes long after your build stops resolving.

Reach usually costs speed. Here it doesn’t.

The usual trade is that a portable format is a slow one. boring doesn’t pay it. On the JVM it beats nippy on every encode we measure and on the map-heavy decodes, and runs even with it on the small ones. Against hako (a codec engineered for raw speed and especially low allocation, staying off-heap via JDK FFM), the fair comparison is tier-matched, reusing both sides. Reused, hako’s decisive, consistent win is allocation (near-zero per call on its off-heap path), with a time lead on small payloads, primitive arrays, and larger-map decode. On most collection encode and decode the two trade roughly even, and boring keeps the nested-vector and small-map decodes. Out-running a dedicated speed codec was never the point. A portable format not being a slow one is. On ClojureScript boring is always smaller on the wire than transit, faster on the datom-shaped data it was built for, and slower on generic data. The performance notes say exactly where, in both tiers.

Getting there did not require changing a single byte of CBOR. Where boring needed more, it grew inside the format instead of around it, and that discipline is what makes the rest of this post possible.

The property everything depends on: navigability

A boring blob is self-describing CBOR (every value carries its own shape), and boring can lay an offset index at the end of the blob as an ordinary tagged item that any other CBOR reader skips. So a reader can walk straight to one field, or jump to it through the index, without materialising the rest.

A dumb blob store where every read means “decode the whole thing” becomes a store you can reach into. That single capability shows up three ways.

Read one field without decoding the value

require('[konserve.mmap :as kmm] '[boring.nav :as nav])

kmm/with-mmap-value([c store "customers"]
  nav/value(get-in(c ["customer-137" "name"])))
;; walks the memory-mapped wire format to that one key;
;; the other customers are never built

(require '[konserve.mmap :as kmm]
         '[boring.nav :as nav])

(kmm/with-mmap-value [c store "customers"]
  (nav/value (get-in c ["customer-137" "name"])))
;; walks the memory-mapped wire format to that one key;
;; the other customers are never built

Edit one field without decoding, or rewriting, the value

This is the opening example, and it comes in three shapes, chosen automatically:

• a same-length change is a poke: the bytes are overwritten in place, and any offset index stays valid.
• a size-changing leaf is a splice: only the altered bytes move.
• a structural change (a new key, a removed key) re-encodes only the parent container.

kmm/assoc-in!(store [:doc :section :field] 42)
kmm/update-in!(store [:doc :counter] inc)
kmm/dissoc-in!(store [:doc :section :stale])

(kmm/assoc-in!  store [:doc :section :field] 42)
(kmm/update-in! store [:doc :counter] inc)
(kmm/dissoc-in! store [:doc :section :stale])

You pick durability, not mechanism: :rename (the default, never mutates in place, crash-safe by construction), :checked (edits in place, guarded by a dirty marker a reader can detect after a crash), or :raw. On a 34 MB value the in-place :checked path is the 70–360× from the top of the post, because the value is never decoded or re-encoded. A field update costs a page write whether the value is a kilobyte or a gigabyte.

This part is experimental, filestore-only, and the in-place writes need JDK 22+ (java.lang.foreign); without it they fall back to a whole-file rewrite that needs nothing. It also asks the store to write boring’s deterministic, stringref-free :archival profile. Full detail in konserve’s in-place-editing guide.

Scan and aggregate a million blobs, decoding only the columns you touch

The same navigability turns konserve-lmdb into something close to a map-reduce engine. It stores keys in value order, so range and prefix scans mean something, and a scan is an ordinary reducible that folds each chunk as it arrives rather than building a seq:

require('[konserve-lmdb.store :as s])

s/scan(store {:from "a", :to "m"})
s/scan-keys(store {:prefix ["user"]})

(require '[konserve-lmdb.store :as s])

(s/scan store {:from "a" :to "m"})       ; ordered [key value] pairs
(s/scan-keys store {:prefix ["user"]})   ; keys only — reads no value pages

The potent operation is projection. project walks a key range and pulls named fields out of each stored value without materialising the value. The store’s Range picks the rows, boring’s navigator picks the columns, and neither decodes what it does not need:

s/project(store {:prefix ["user"]} [:profile :address :city])

(s/project store {:prefix ["user"]} [:profile :address :city])

Against a full decode of the same blob, projecting one path costs 0.8 µs versus 19.7 µs at 8 KB, and 129 µs versus 4.3 ms at 1.4 MB. The gap widens with size, because you pay per field, not per document. And project-reduce folds those fields as it goes, in one pass, without ever building a row, a GROUP BY … SUM over a plain key-value store:

s/project-reduce(store
  {:prefix ["user"]} [[:profile :city] [:revenue]]
  fn [acc _k [city revenue]]: update(acc city fnil(+ 0) revenue) end
  {})

(s/project-reduce store {:prefix ["user"]} [[:profile :city] [:revenue]]
                  (fn [acc _k [city revenue]]
                    (update acc city (fnil + 0.0) revenue))
                  {})

Grouping revenue by city over 20 000 rows costs 0.20 µs/row in one pass, against 0.97 doing it as two scans and a join, because the named fields are read together and no intermediate [key value] pair is ever allocated (a reduced return stops the cursor). Both refuse on a non-navigable store rather than silently falling back to a full decode. A silent fallback would turn the whole point of the call into a performance mystery.

Even the FlatBuffers trick stays in-band

Columnar packing is the last thing a fast binary format usually makes you trade reach for. protobuf and FlatBuffers get their compactness by fixing a schema out of band: field names live in a compiled .proto/.fbs, and each record stores only values at known positions. It is fast, but the bytes are unreadable without the schema.

boring does the same packing without leaving self-describing CBOR. Turn on :shapes, and an array of maps that share a key set is written with the keys hoisted out once and each row reduced to values in position:

[{:e 1, :a :x} {:e 2, :a :y}]
;; shaped:  tag 39649 [ [:e :a]            ← the key set, once
;;                      [[1 :x] [2 :y]] ]  ← rows: values only

[{:e 1 :a :x} {:e 2 :a :y}]
;; shaped:  tag 39649 [ [:e :a]            ← the key set, once
;;                      [[1 :x] [2 :y]] ]  ← rows: values only

On 200 datom-shaped maps that halves the wire size (9 952 → 4 982 bytes) and cuts decode from 21.0 to 11.4 µs (level with hako in the same per-call benchmark), because each key is interned once and threaded into every row with no per-row key work. But the key set rides inside the blob, so there is no schema file and no codegen: a foreign reader still decodes it as an ordinary tagged array. You get the FlatBuffers columnar win and keep the reach. (The zero-copy random field access FlatBuffers is famous for is boring’s separate offset index, and the two compose.)

Two honest edges: it’s a win for decode speed and raw size (a general-purpose compressor feeds on the repetition shapes remove, so under zstd the size advantage goes back to zstd), and today it covers arrays of same-shaped maps, on a still-provisional tag. The map-of-maps generalisation is specified, not yet shipped.

It grew inside the format

None of this is a private extension bolted onto CBOR. String deduplication is stringref, a registered CBOR extension. The offset index that makes a file navigable is an ordinary tagged item at the end that every other CBOR reader skips. A file boring writes stays a file cbor2 and cbor.me can read. The reach is never spent to buy the speed or the navigability. You keep all three.

Try it

org.replikativ/boring

{:mvn/version "0.1.27"}

org.replikativ/boring {:mvn/version "0.1.27"}

• boring: the serializer, with reading, interop, and performance docs.
• konserve.mmap: read and edit stored values without decoding them.
• konserve-lmdb: ordered scans and column projection over an LMDB keyspace.

Use CBOR by default, and reach for something else only when you have a reason you would still defend in five years, to whoever is holding your data then.