Data governance in versioned systems

May 2026

If a database remembers everything, how do you delete things?

The question arises in right-to-erasure requests, accidental ingestion of personal data and retention policies. An immutable system needs an explicit answer because overwriting the current value does not remove historical reachability.

Access control has the same historical dimension. Permissions must account for current data, branches and older snapshots rather than only the latest state.

This note describes what Datahike removes from its live store, what remains elsewhere and which controls are still application work.

Purge in Datahike

Datahike has a purge operation that removes datoms from a database’s indices, both the current and the history index of the resulting commit.

What is this syntax?

require('[datahike.api :as d])

;; Before purge
d/q('[:find ?n :where [?e :name ?n]], @conn)
;; => #{["Alice"] ["Bob"]}

;; Purge Alice (requires :keep-history? true)
d/transact(conn, [[:db.purge/entity [:name "Alice"]]])

;; After purge: gone from current state and the new commit's history
d/q('[:find ?n :where [?e :name ?n]], @conn)
;; => #{["Bob"]}

d/q('[:find ?n :where [?e :name ?n]], d/history(@conn))
;; => #{["Bob"]}

(require '[datahike.api :as d])

;; Before purge
(d/q '[:find ?n :where [?e :name ?n]] @conn)
;; => #{["Alice"] ["Bob"]}

;; Purge Alice (requires :keep-history? true)
(d/transact conn [[:db.purge/entity [:name "Alice"]]])

;; After purge: gone from current state and the new commit's history
(d/q '[:find ?n :where [?e :name ?n]] @conn)
;; => #{["Bob"]}

(d/q '[:find ?n :where [?e :name ?n]] (d/history @conn))
;; => #{["Bob"]}

purge is not a soft delete. It rewrites the affected path through the persistent sorted set, and the new commit’s index roots no longer reach a node containing Alice. It is also not retroactive across the commit graph. A pre-purge commit still points to the old nodes until d/gc-storage sweeps that commit. Before then, (d/commit-as-db conn <pre-purge-uuid>) can still see Alice.

How `DELETE` disposes of data, and how purge differs

A traditional DELETE leaves the bytes behind. In PostgreSQL, DELETE marks the row’s heap tuple dead, and the bytes live on in several places:

In the heap page as a dead tuple. Plain VACUUM marks the space reusable and may defragment within the page, but never returns space to the OS or zeroes bytes. VACUUM FULL rewrites the table into a new file. The old file is unlinked but its disk blocks are not zeroed.
In WAL segments until they’re recycled, and indefinitely in archived WAL if archive_mode=on.
In replicas, which received the original WAL and run their own VACUUM cycle.
In backups: base backups, pg_dump archives, PITR base + WAL.

Confirming erasure requires checking each of those layers under the deployment’s backup and retention policy.

Purge changes the live-store part of that picture. The persistent sorted set is a tree of content-addressed nodes, and the purge transaction rewrites the affected path. A later gc-storage with an appropriate cutoff sweeps unreachable intermediate commits. Within that commit graph, an operator can enumerate the snapshots reachable from known branch heads and determine which ones contain the datom.

Where Datahike does not differ from PostgreSQL: backups, storage backends with their own versioning, and replication targets. We come back to that in Backups and storage-layer history.

Garbage collection in Datahike

d/gc-storage physically reclaims unreachable storage. Branches as values, merges as queries covers cutoff dates, protected branch heads and online versus offline collection. For erasure, it composes with purge as follows:

What is this syntax?

require('[superv.async :refer [<?? S]])

;; 1. Purge produces a new commit whose roots no longer reach Alice.
d/transact(conn, [[:db.purge/entity [:name "Alice"]]])

;; 2. gc-storage WITHOUT a cutoff only reclaims storage on deleted branches.
;;    The pre-purge commit on :db is a live intermediate commit, so its
;;    tree nodes (containing Alice) survive this sweep.
<??(S, d/gc-storage(conn))

;; 3. gc-storage WITH a cutoff is what physically evicts those nodes.
;;    Pick a cutoff that exceeds your longest-running reader's lifetime.
let [seven-days-ago new java.util.Date(System/currentTimeMillis() - 7 * 24 * 60 * 60 * 1000)]:
  <??(S, d/gc-storage(conn, seven-days-ago))
end

(require '[superv.async :refer [<?? S]])

;; 1. Purge produces a new commit whose roots no longer reach Alice.
(d/transact conn [[:db.purge/entity [:name "Alice"]]])

;; 2. gc-storage WITHOUT a cutoff only reclaims storage on deleted branches.
;;    The pre-purge commit on :db is a live intermediate commit, so its
;;    tree nodes (containing Alice) survive this sweep.
(<?? S (d/gc-storage conn))

;; 3. gc-storage WITH a cutoff is what physically evicts those nodes.
;;    Pick a cutoff that exceeds your longest-running reader's lifetime.
(let [seven-days-ago (java.util.Date. (- (System/currentTimeMillis)
                                         (* 7 24 60 60 1000)))]
  (<?? S (d/gc-storage conn seven-days-ago)))

The required sequence is purge followed by cutoff GC, not purge alone. Plain gc-storage leaves intermediate commits intact. A pre-purge commit becomes eligible on the first cutoff-GC pass after it ages out of the grace window. The resulting erasure delay is determined by the chosen grace period and GC cadence; whether that meets a legal or contractual requirement depends on the deployment.

The cutoff has to comfortably exceed your longest-running reader’s lifetime. Datahike’s distributed readers walk storage directly without coordinating with a writer, so a snapshot vanishing mid-query is a real failure mode.

Branch heads are always kept regardless of cutoff. So the recipe assumes the post-purge state is the head you want to keep. If the datom also exists on another branch, you purge there too before sweeping.

Secondary indices

Datahike’s secondary indices are first-class versioned state: Scriptum (Lucene full-text), Proximum (HNSW vector), Stratum (columnar). Indices are CoW-forked on branch, persisted with each commit, and restored on connect.

For governance, the question is whether purge propagates. It does: a purge transaction routes a retraction event (-transact with :added? false) to every secondary index covering an affected attribute, the same way :db/retract does. After purging Alice on a database with a Scriptum index over :person/name and :person/bio, a full-text search for “Alice” returns nothing, a vector KNN over her embedding skips her, and any columnar aggregate excludes her row.

On storage reclamation, Stratum and Proximum are konserve-backed: d/gc-storage sweeps their unreachable blobs alongside the primary indices, following the same pattern (Stratum: columnar rewrite, Proximum: HNSW mark-delete).

Scriptum is the exception. Its Lucene segments live on the writer node’s local filesystem, outside konserve. Scriptum’s -sec-mark returns the empty set, so d/gc-storage can’t reach them, and Lucene’s own delete model is tombstones-until-segment-merge. The bytes linger inside a segment file until Lucene merges that segment away. For full erasure on Scriptum you may need to force a segment merge and make sure the writer’s filesystem snapshot policy doesn’t pin old segments.

Backups and storage-layer history

Datahike does not escape the backup and archive problem. A backup of the konserve store taken before purge+GC contains the pre-purge nodes. Storage backends with their own versioning hold them too:

S3 object versioning, if enabled, retains every overwritten key as a prior version.
ZFS / btrfs snapshots of the konserve directory retain the pre-purge tree.
A git-backed konserve backend retains the pre-purge commit graph in .git even after a purge rewrite on the working tree.
Logical replicas of the konserve store hold whatever they received.

Purge and cutoff GC reach the live store. Erasure across backups and storage-layer history is a separate procedure: identify the destinations that may contain the datom, then expire, purge or rewrite them according to their own capabilities. This is an operational retention problem in any system with independent backups.

One operational technique is crypto-shredding: encrypt sensitive values with a dedicated key and destroy the key when access must end. The ciphertext remains in backups but should become unreadable. Whether that satisfies a particular erasure requirement needs legal and security review.

Within Datahike’s live commit graph, history and branches are explicit. Starting from the known branch heads, an operator can enumerate reachable snapshots and test which ones reference a datom. WAL archives, backups, replicas and storage-level versions remain separate inventories in both Datahike and PostgreSQL deployments.

Multi-branch purge

Purge is a transaction on one branch. If the same datom is reachable from another branch head, or from a commit inside the GC window, that path remains. Structural sharing may keep one physical node in konserve, but each branch has an independent path to it.

The practical procedure: purge on every branch you control, delete branches you no longer need, then run cutoff-GC. Data reachable only from deleted branches gets reclaimed.

For databases with hundreds of agent-created branches, this gets expensive. Each branch’s purge is its own transaction and writes its own tree path. Optimization for the agent-fanout case is on the roadmap.

Access control

Today, Datahike access control works at the connection and storage level. Storage credentials gate database access. The :branch config field, or SET datahike.branch over SQL via pg-datahike, pins a connection to a branch. commit-as-db, or SET datahike.commit_id, selects a snapshot for audit-style access. Read versus write permission is configured per connection.

Row-level access (“user X can query this branch but shouldn’t see rows where :department = "HR"”) applied consistently across current state, history, branches, and secondary-index queries is the next layer. EACL-style ReBAC is one direction we’re exploring.

Why immutable is still useful for compliance

The claim that the database alone can prove global erasure does not survive independent backups and storage-level history. A narrower claim is supportable.

In a conventional mutable deployment, WAL archives, replicas, backup catalogues and page storage are separate places to inspect.

In Datahike’s live store, history and branches are explicit. You can walk the commit graph from known branch heads and identify reachable snapshots that hold the datom. Each storage class then has its own procedure:

Live store: purge + cutoff-gc-storage rewrites the index path and sweeps the intermediate commits. commit-as-db lookups for swept UUIDs fail cleanly.
Secondary indices: covered by purge propagation on the konserve-backed ones. Scriptum needs the segment-merge / filesystem step.
Backups, S3 versioning, ZFS snapshots, git-backed stores: each is a separate destination addressed by policy.

The benefit is not fewer copies. Versioning deliberately retains more states. The benefit is that states inside the commit graph are addressable, turning that part of an erasure procedure into a checklist. External copies still require a separate inventory.

What’s not solved yet

Multi-branch purge at scale is expensive. Hundreds of branches means hundreds of transactions, each rewriting its own tree path. Needs optimization work for agent-fanout workloads.
Scriptum filesystem GC. Lucene segments live outside konserve. d/gc-storage can’t reach them, and Lucene’s delete model is tombstones-until-merge. Filesystem cleanup is a separate operational step.
Row-level access control is the next layer. Connection / branch / commit-level access works today. Fine-grained ReBAC across current state, history, branches, and secondary indices is the work in flight.
No retention-policy automation. Datahike ships no built-in “delete everything older than 7 years.” :db.history.purge/before is the primitive. The policy loop is application code.

For the live Datahike store, the practical sequence is explicit: purge the relevant branches, wait out the reader grace period and run cutoff GC. Secondary indexes and every external backup destination must be included in the operating procedure. Immutability makes the live commit graph inspectable; it does not replace retention, access-control or backup policy.