A database that is just a bucket

Most databases are a process you run. Datahike is a value you store — an immutable, persistent index that lives in a key-value store — and that changes what “deploying a database” has to mean. dthk is Datahike compiled to a native binary for Linux, macOS and Windows. Give it a directory, or an S3 bucket, and that is the whole installation.

unzip datahike-<version>-linux-amd64.zip && chmod +x dthk

cat > db.edn <<'EDN'
{:store {:backend :file :path "./mydb" :id #uuid "a1b2c3d4-e5f6-7890-abcd-ef1234567890"}
 :keep-history? true
 :schema-flexibility :read}
EDN

./dthk create-database edn:db.edn
./dthk transact conn:db.edn '[{:name "Alice"} {:name "Bob"}]'
./dthk query '[:find ?n :where [?e :name ?n]]' db:db.edn

Every command is a fresh process. It opens the store, does its work, and exits. There is nothing to keep running between them.

What you get for that

The same database as the JVM library, because it is the JVM library, ahead-of-time compiled: Datalog queries, a full transaction history you can query with history:db.edn or asof:<tx>:db.edn, branches, and one file format shared with every other Datahike client. A Clojure service, a Python script through libdatahike, and a shell pipeline can all open the same directory.

Two writers, no server

The part that usually forces a server is writing. Two processes that both read the current state and both write a new one will lose one of the writes — unless something orders them. Datahike orders them in the store.

A commit is a new index value, written to storage, plus one pointer flip: the branch head. The head is written conditionally — it lands only if the head is still the one the writer read. If another process moved it in between, the commit is rejected and re-applied against the new head, up to three times with a jittered backoff. Nothing is lost and nothing is torn: the values a rejected commit wrote are content-addressed and immutable, so a losing writer leaves collectable garbage, never a dangling pointer.

What “conditionally” means depends on the store, and konserve — the storage layer — reports how far its guarantee reaches:

So two shells on one machine are safe on the filesystem, and any number of Lambdas or containers are safe on a bucket. The default :writer-ownership :shared re-reads the head before every batch, which is what makes writers that alternate safe on any store; the conditional write is what makes writers that overlap safe on a store that can compare-and-set.

If your deployment depends on that, say so, and the CLI will refuse to connect to a store that cannot provide it:

{:store {:backend :s3, :bucket "my-bucket", :region "eu-central-1",
         :id #uuid "7a3e8400-e29b-41d4-a716-4466554407d1"},
 :writer {:backend :self, :writer-ownership :shared, :require-fencing :global}}

{:store  {:backend :s3 :bucket "my-bucket" :region "eu-central-1"
          :id #uuid "7a3e8400-e29b-41d4-a716-4466554407d1"}
 :writer {:backend :self
          :writer-ownership :shared
          :require-fencing :global}}   ; :machine for one host, :global for many

What it costs

Shared ownership costs one branch-head read per batch of transactions — roughly 10–40 ms and $0.0000004 on S3. Transactions already queued when one commits are chained onto it and share its read, so a burst of 500 concurrent transactions costs on the order of nine head reads and twenty commits, not five hundred of each. A caller that awaits every transaction before issuing the next has nothing to batch and pays one read per commit; that is the honest worst case, and it is the cost of a stat.

Storage is the only bill. A bucket with a few thousand small objects, or a directory, is the entire footprint of a database with its complete history.

Where the writer runs

None of this needs Datahike’s server. It needs a store that can compare-and-set, and every writer to open it with shared ownership. That is a different shape from the usual “one writer process, many readers” — it is no writer process, and any process may write, ordered by the store.

The moment you want more than that — a long-lived writer that batches on your behalf, a WebSocket that streams commits to browsers, a place to put authorization — that is what the Datahike server and the Kabel writer are for, and a :writer {:backend :datahike-server ...} entry in the same config file sends transactions there instead. The store, and the data in it, do not change.

Honest edges

• The filesystem guarantee is per host, and per runtime: the JVM and native binaries take an OS lock, the Node build a lockfile, and the two do not see each other. Mixing Node and JVM writers on one directory is not fenced today; on S3 they are, because S3 does the comparing.
• A crashed writer leaves its staging file behind. Harmless, invisible to reads, and something the garbage collector will learn to sweep.
• Windows: shipped as of 0.8.1835, after a run of fixes that were mostly about what Windows will not let you do to an open file.

The CLI documentation has the full command set, including forking and merging databases from the shell; the configuration reference has the writer and fencing options in detail.