Datahike

Stratum: SQL that branches

A few years ago I hit a wall I suspect many data engineers know. I had a million-row analytical dataset and I wanted to run an experiment: modify a few pricing assumptions, re-run a set of aggregation queries, compare the results against the original. Simple enough - except in a mutable database, “compare against the original” means either keeping a copy of the data or hoping nothing changed. Neither scales.

Datahike solves this for entity-level data. Its storage is EAVT-indexed - like Datomic, tuned for entity traversal and point lookups. That’s the right structure for a system-of-record, but not for scanning 10M rows to compute a GROUP BY with SIMD. Stratum explores the columnar alternative: the same CoW branching semantics, but over column-oriented storage optimized for analytical scans. SQL is the natural interface for this access pattern - something Datahike doesn’t yet have. The longer-term plan is integration: Stratum’s columnar engine and SQL support as a query path within Datahike’s Datalog planner.

The core insight is that a columnar dataset is just a value. Make it immutable with structural sharing and you get git-like semantics for free: fork a dataset in O(1), modify branches independently, time-travel to any snapshot, persist named commits to storage. Then add SIMD execution via the Java Vector API, and it turns out you can beat DuckDB on most single-threaded analytical queries from pure JVM code - no native compilation, no JNI.

The SQL interface

Stratum speaks the PostgreSQL wire protocol. The quickest entry point is the standalone server:

java --add-modules jdk.incubator.vector \
     --enable-native-access=ALL-UNNAMED \
     - jar stratum-standalone.jar \
     --index orders:/data/orders.csv

Any PostgreSQL client connects immediately - psql, DBeaver, JDBC, psycopg2:

psql - h localhost - p 5432 - U stratum
-- Standard analytical SQL
SELECT region,
       SUM(amount * discount) AS revenue,
       COUNT(*)               AS orders
FROM orders
WHERE ship_date BETWEEN '2024-01-01' AND '2024-12-31'
GROUP BY region
ORDER BY revenue DESC;

-- Query CSV and Parquet files inline - auto-indexed on first access
SELECT payment_type,
       AVG(tip_amount),
       PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY tip_amount)
FROM read_csv('/data/taxi.csv')
GROUP BY payment_type;

Full DML: SELECT, INSERT, UPDATE, DELETE, UPSERT (INSERT ON CONFLICT). CTEs, correlated subqueries, window functions (ROW_NUMBER, RANK, LAG, LEAD, running aggregates), joins (INNER/LEFT/RIGHT/FULL with multi-column keys), set operations (UNION/INTERSECT/EXCEPT). Aggregates: SUM, COUNT, AVG, MIN, MAX, STDDEV, VARIANCE, CORR, MEDIAN, PERCENTILE_CONT, APPROX_QUANTILE, COUNT(DISTINCT). CASE WHEN, COALESCE, date functions, LIKE/ILIKE, FILTER clause. Full SQL reference →

How the engine works

Every column is split into fixed-size chunks. Each chunk carries pre-computed statistics: minimum, maximum, sum, count. This unlocks two significant optimizations.

Zone-map pruning. Each chunk carries pre-computed min, max, sum, and count statistics. DuckDB stores only min and max per segment, used for predicate filter pushdown - skipping segments that can’t contain rows matching a WHERE clause. Both engines do this. What DuckDB doesn’t pre-compute is per-segment SUM or COUNT, so unfiltered aggregates like COUNT(*), SUM(price), or AVG(price) require a full data scan in DuckDB. In Stratum, these are answered by traversing the pre-computed metadata at tree nodes - no row data touched. SELECT AVG(price) FROM orders on 10M rows: Stratum 0.1ms, DuckDB 7.1ms.

Fused SIMD execution. Most columnar engines evaluate predicates in one pass, then apply the result mask during a separate aggregation pass. Stratum fuses these into a single loop: predicates and accumulation run simultaneously via Java Vector API VectorMask chains, processing four doubles or longs per SIMD cycle. No intermediate arrays, no second pass, no extra allocation.

The Vector API (JDK 21+) provides DoubleVector and LongVector operations backed by AVX-512 on x86 and SVE on ARM. The bet was that the JVM incubator API had matured enough to compete with native code on analytical workloads without the deployment complexity of a native library. The benchmarks suggest that bet paid off.

Performance

Single-threaded comparison vs DuckDB v1.4.4 (JDBC in-process) on 10M rows, Intel Core Ultra 7 258V, JVM 25. Median of 10 iterations, 5 warmup:

Query Stratum DuckDB Ratio
TPC-H Q6 (filter + sum-product) 13ms 28ms 2.2x faster
Filtered COUNT (NEQ pred) 3ms 12ms 4.0x faster
TPC-H Q1 (7 aggs, 4 groups) 75ms 93ms 1.2x faster
H2O Q3 (100K string groups) 71ms 362ms 5.1x faster
H2O Q10 (10M groups, 6 cols) 832ms 7056ms 8.5x faster
LIKE '%search%' 47ms 240ms 5.1x faster
AVG(LENGTH(URL)) 38ms 170ms 4.5x faster
H2O Q6 (STDDEV group-by) 30ms 81ms 2.7x faster
H2O Q9 (CORR) 61ms 134ms 2.2x faster
MEDIAN(price) 68ms 158ms 2.3x faster
ROW_NUMBER window 316ms 426ms 1.3x faster

Stratum wins 35 of 46 queries at 10M rows (single-threaded, median of 10 runs). DuckDB wins on sparse-selectivity filters, window-based top-N, high-cardinality hash group-by at scale (1M+ unique groups where hash tables become DRAM-bound), and global COUNT(DISTINCT). Full methodology and raw results: benchmark docs.

DuckDB is an excellent system. The point is that pure JVM code can compete with a mature native engine on the workloads that matter most, while adding semantics DuckDB doesn’t have.

Branching: where it diverges

This is the part that doesn’t exist anywhere else.

Each column is backed by a chunked B-tree (PersistentColumnIndex) that implements Clojure’s IPersistentCollection and IEditableCollection protocols. When you call (st/fork ds), you get a new dataset that shares all unchanged chunks with the original. No data is copied - just a new root pointer into a shared tree. Mutations through the transient protocol only write-copy the chunks they touch. A billion-row dataset costs essentially nothing to fork.

(require '[stratum.api :as st]
         '[konserve.file-store :as fs]
         '[clojure.core.async :refer [<!!]])

;; Open storage, load the orders dataset (10M rows)
(def store  (<!! (fs/new-fs-store "/data/stratum")))
(def orders (<!! (st/load store "orders")))

;; Fork in O(1) - structural sharing, zero data copied
(def experiment (st/fork orders))

;; Persist the fork as a named branch
(<!! (st/sync! experiment store "experiment"))

;; Query both branches via SQL - pass column data as table map
(st/q "SELECT SUM(price * qty) FROM t" {"t" (st/columns orders)})
;; => {:SUM(price * qty) 4821903.40}   ← main branch

(st/q "SELECT SUM(price * qty) FROM t" {"t" (st/columns experiment)})
;; => {:SUM(price * qty) 4401238.66}   ← experiment branch

;; Time-travel: load any historical branch by name
(def baseline (<!! (st/load store "orders-baseline")))

From the server side, register-live-table! lets you expose named branches as separate SQL tables - query them with plain SQL over the PostgreSQL connection without touching the Clojure API.

The practical uses this unlocks:

For Clojure developers

If you’re coming from the Clojure ecosystem, Stratum datasets behave like ordinary Clojure values. They implement IPersistentCollection, ILookup, IEditableCollection - tablecloth and tech.ml.dataset work with them directly as column maps. You can query with SQL strings or a Clojure DSL that composes programmatically:

(require '[stratum.api :as st])

;; DSL - composable, no string manipulation
(st/q {:from   {:price prices :qty quantities :region regions}
       :where  [[:> :price 100]]
       :group  [:region]
       :agg    [[:sum [:* :price :qty]]
                [:count]]})

;; SQL string - same engine underneath
(st/q "SELECT region, SUM(price * qty), COUNT(*)
       FROM orders WHERE price > 100 GROUP BY region"
      {"orders" {:price prices :qty quantities :region regions}})

The DSL is useful when building queries programmatically - no string interpolation, no injection risk, results are plain Clojure maps.

The origin

This work started with Votorola, a collaborative democracy project that needed distributed state. The limitations of imperative systems led to Clojure, then to replikativ for distributed replication, then to Datahike for immutable entity-level storage. Each step sharpened the same conviction: mutability is the core problem. When data changes in place, you lose history, auditability, and the ability to reason about what a system knew at any point in time.

My PhD work on simulator-based inference at UBC’s PLAI lab reinforced this. Probabilistic systems need to fork hypotheses, accumulate evidence, and explain their reasoning - tracking not just the current state but the path that led to it. Stratum is the analytics piece of the infrastructure we’re building for that.

The ecosystem

Via Yggdrasil you can fork a Datahike database, a Stratum dataset, and a Proximum index together - consistent snapshots across SQL, Datalog, and vector search at the same point in time.

Getting started

;; deps.edn - check https://clojars.org/org.replikativ/stratum for latest
{:deps {org.replikativ/stratum {:mvn/version "0.1.7"}}}

:jvm-opts ["--add-modules=jdk.incubator.vector"
           "--enable-native-access=ALL-UNNAMED"]
# Standalone server with built-in demo tables (lineitem, taxi -100K rows each)
java --add-modules jdk.incubator.vector \
     --enable-native-access=ALL-UNNAMED \
     - jar stratum-standalone.jar --demo

Requirements: JDK 21+

Source: github.com/replikativ/stratum

Documentation: Query DSL · SQL Interface · Dataset API · Anomaly Detection · Benchmarks · Architecture

Source and documentation: github.com/replikativ/stratum. Feedback welcome on the Clojurians #datahike channel.