Architecture

Memlake has four moving parts, and one of them is your bucket: the bucket holds every byte of memory, query nodes answer queries from a cache of it, indexers turn new writes into indexes, and an optional archiver copies everything to a backup bucket. Hindsight's API talks to the query nodes over gRPC. All of Memlake's components are written in Rust.

The bucket

Each organization gets its own bucket, and each memory bank is a prefix inside it holding three kinds of objects:

• The log (wal/). Every write becomes one new, immutable file. The log holds recent memories until they are indexed.
• Segments (seg-*/). Small, self-contained indexes over a slice of the bank, written once and never modified. Inside a segment, each memory type (world facts, experiences, observations) has its own vector clusters, full-text index, entity index, time index and links.
• The manifest and the head pointer. The only two objects that ever change. The manifest lists the live segments and how far the log has been indexed; the head pointer marks how far the log has been committed.

Conditional writes are the whole coordination story

The two objects that change are only ever replaced with a conditional write: the store accepts the new version only if nobody changed it first (If-Match: <etag>). Log slots are claimed the same way, with a conditional create (If-None-Match: *). That is all the coordination Memlake has. There is no database, lock service or leader beside the bucket, and query nodes never talk to each other.

Every decision that keeps data safe (who claims the next slot in the log, when an index is published, what garbage collection may delete) comes down to a conditional write that the bucket either accepts or rejects. Because nothing is overwritten in place, a reader always sees a complete version of a bank or the previous one, never something in between. This small protocol is what we verify with a model checker; see Guarantees.

Writes

A query node writes each Hindsight retain to the log with a conditional create, so two writers can never claim the same position. It then advances the head pointer with a conditional replace, and only then acknowledges. A retain (new facts, replaced old ones, newly seen entities) is a single log entry, so it lands entirely or not at all.

Writes don't wait for indexing. Every recall scans the part of the log between the index cursor and the head pointer and merges it with the segments, so a write is visible to the next query from any query node as soon as it is acknowledged.

A write commits in about 250 ms at p50, bound by the round trip to object storage. Concurrent writes to one bank can optionally share one head-pointer update (group commit, off by default); each write is still its own log object. In Hindsight, every retain goes through an LLM first, so how fast memories land is normally bound by the LLM, not by Memlake.

Query nodes

A query node's only state is a cache, in memory and on local NVMe, of objects it has read from the bucket. Data objects never change and live under unique paths, so the cache never needs invalidating; dropping it changes latency, never results.

The load balancer sends each bank to the same query node so its cache stays warm (consistent hashing, so a scale event moves only about 1/N of banks), but any query node can serve any bank. An optional balancer can place large or busy banks explicitly, and serve a hot bank from up to three query nodes.

On a recall, the query node:

1. reads the head pointer, to see the latest writes (concurrent queries share this read),
2. reads only the blocks it needs from the segments, from cache when it can,
3. scans the un-indexed tail of the log,
4. runs the four searches over the same memories,
5. applies Hindsight's ranking policy next to the data, and returns only the memories that make the cut.

Hindsight owns the ranking policy (fusion, cross-encoder reranking, boosts, score floors and the token budget) and sends it with every request. Policy changes take effect on the next request, without touching stored data. Memlake can also return each search's raw scores for Hindsight to rank.

A bank's index lives on one query node at a time, and Memlake grows by spreading banks across query nodes, which is how agent memory splits anyway: per agent, user or project. When a query node disappears, another picks up its banks with no handover and warms its cache as it serves:

Indexers

Indexers run as a separate workload, scheduled away from the query nodes that answer queries, so building indexes never takes CPU from recall.

An indexer folds new log entries into a fresh segment, deriving links between memories as it goes, and publishes it by swapping the manifest with a conditional replace. Indexing cost follows what is new, not what is stored: adding 100 memories to a 100M-memory bank builds a 100-memory segment. Background compaction merges small segments so their count stays bounded, with memory bounded by configuration rather than bank size.

A work queue assigns banks to indexers, but correctness does not depend on it. Two indexers can work on the same bank with no lock: one wins the manifest swap, and the other's files are never referenced. Garbage collection deletes only what no manifest points to, after a safety window (15 minutes by default) measured on the object store's own clock.

If writes arrive faster than indexing can fold them, the un-indexed tail grows and every recall has to scan it. Memlake caps that tail in bytes: past the limit, new writes get a retryable error until indexing catches up. A rejected write was never acknowledged, so nothing is lost.

Archiver

When backups are on, the archiver continuously copies every bank's log, plus a daily snapshot of the rest, to a separate bucket, ideally in another account with a credential that cannot delete. The backup trails live writes by about one indexing interval plus one archive pass. A bank can be restored to a log position or a point in time, and every restore is verified before traffic moves to it.

The four searches

All four searches read the same stored memories:

• By meaning (vector). Vectors are grouped into clusters (an IVF index), and a query scans only the clusters closest to it. Each vector is kept twice: compressed with RaBitQ to about 60 bytes (for 384-dimension embeddings, 25× smaller than the original) for a fast first pass, and at full precision to rerank the best candidates exactly.
• By keyword (BM25). BM25 over a tantivy index stored as one immutable object per memory type in each segment, loaded when the bank is opened, with word segmentation for Chinese.
• Through relationships (graph). One step out from the best vector matches: to memories that share an entity, are close in meaning, caused or were caused by them, or happened right before or after. The indexer derives the "close in meaning" links from the clusters it already builds, and links travel with each memory rather than living in a separate links table.
• Through time. Each segment has a time index; a query with a time window reads just that range, then takes one step out to the nearest neighbours in time.

Each memory type is its own independent index, so one type's statistics never skew another's ranking. Inside a segment, each memory type has its centroids, vector clusters (compressed codes), exact vectors for reranking, record payloads, a BM25 index, entity postings, a time index and reverse links, each stored as its own object so a query reads only what it needs.

What is exact, and what is approximate

• Guaranteed by design: exact scores are comparable across segments, so merging each segment's reranked top results gives the same top results as reranking the same candidates in one place.
• Approximate, as with any cluster index: finding candidates within a segment. A query that probes only the nearest clusters can miss a match in a far one.
• Tested: a search that probes every cluster matches brute force exactly, and on BEIR the compressed first pass finds the same top 10 as 8-bit vectors.
• Expected, and close to what we measure: a segment's vector search cost grows with the square root of its size. In our benchmarks, a bank 30× larger made recall about 6× slower under heavy load.

Every search reads every segment, so the bytes a recall reads grow with segment count. For large banks, optional segment routing has the vector search score segments by their centroids and read only the nearest few. It is off by default.

Query latency and cold starts

On a cached bank, the only object-storage request a recall makes is reading the head pointer. Everything else comes from memory or NVMe. On a 3.08M-memory bank, one request at a time on one 6-vCPU query node against GCS in the same region, warm recall measured p50 71 ms and p99 90 ms at the engine.

On a cold query, each search issues its reads across all segments at once, so the number of sequential round trips to object storage is bounded and does not grow with bank size or segment count. The first time a query node opens a bank, it loads the bank's index metadata: 8.6 s at p95 for a 10M-memory bank, once per query node rather than per query. Until its data blocks are cached, a query can take around a second at p99.

Query nodes don't wait for traffic to warm a bank. Whenever a query node opens a bank or picks up a new index, a warm-up pass loads what recall needs in the background. If you know a bank is about to get busy (a user logging in, an agent starting a job), send it one recall ahead of time: that opens the bank and starts the warm-up, so the real queries find it ready.

For more, see Guarantees, Tradeoffs, Performance and cost and the Memlake whitepaper.