Hindsight Memlake

Hindsight Memlake is a storage and retrieval engine for memories, written in Rust and built on object storage. It is designed for three things together: scale, low, consistent latency, and lower cost as you grow. It is part of Hindsight Enterprise, and most of Hindsight Cloud already runs on it. This page is for architects evaluating Hindsight Enterprise and for the teams who operate an Enterprise deployment. It covers how Memlake stores your data, when to use it, how it is turned on, and how it is backed up and secured.

Enterprise feature

Memlake is part of Hindsight Enterprise. On Hindsight Cloud, Vectorize runs and manages it for you, so there is nothing to turn on. To discuss it for a dedicated deployment, book a call or contact your Hindsight representative.

Why agent memory needs its own store

Agent memory only grows. Every retain adds facts, links and observations, and nothing ages out on its own. In our models, an employee's agent writes about 68,000 memories a year, an autonomous agent running around the clock about 5 million, and a 10,000-person company with a thousand autonomous agents about 5.5 billion.

The workload also has an unusual shape: tiny records that only accumulate, four kinds of search (vector, keyword, graph and time) over the same data on every recall, and every write visible the moment it lands. A general-purpose relational database pays for each memory when it is stored and again on every query. On Postgres, we measured where that stops keeping up:

• recall slows as a bank grows (7.0 queries per second at 33k memories, 1.9 at 164k, on a 4-vCPU primary);
• built-in keyword search scores every matching row, so on a 210k-memory bank one recall took about 5 seconds;
• each memory produced about 190 KB of write-ahead log across its indexes, capping ingest at 55–80 memories per second;
• one organization with 1,000 small banks sustained only about 2 recalls per second.

The database isn't doing anything wrong here. Memlake is a store built for this shape of data.

Design goals

Every change to Memlake is reviewed against these properties:

• Object storage is the only source of truth for memory. Memlake has no database, consensus cluster or second stateful system of its own. Durability, replication and availability come from the object store, and every byte of an organization's memory lives in its own bucket.
• Durable before acknowledged, visible once acknowledged. When Hindsight confirms a memory is saved, it is already in object storage, and the next recall from any query node finds it.
• Everything is incremental. Nothing on the write, indexing or recall path reprocesses a whole bank. Indexing works on new data only, and a recall makes a bounded number of round trips to object storage however large the bank.
• One copy, four kinds of search. Each memory is stored once and serves vector, keyword, graph and time search. There is no separate vector database, search engine or graph store to keep in sync.
• Compute is disposable. Query nodes hold only a cache. Losing one costs latency, never data, and capacity grows by adding nodes.
• Bounded by design. Caches and reads of not-yet-indexed writes run on fixed memory budgets. When writes outrun indexing, Memlake slows writers with retryable errors instead of growing memory.

What Memlake is

In a standard Hindsight deployment, memories live in the database. With Memlake, object storage is the system of record for memories. Everything Memlake needs to keep is stored as objects in a bucket.

Memlake has three parts:

Component	What it does
Query nodes	Serve reads and writes for retain, recall and reflect. They are stateless: their local disk and memory are only a cache of data held in object storage.
Indexer	Builds search indexes (vector, full-text, graph and time) from new writes and stores them back in object storage as immutable files.
Archiver	Continuously copies changes to a separate backup bucket. See Durability and backups.

Because the query nodes hold no data of their own, any node can serve any request, and a node can be replaced or rescheduled without losing anything. Caching keeps frequently used banks fast; the object store remains the durable copy.

A write is acknowledged only after it is durable in object storage, and an acknowledged write is visible to the next recall.

One bucket per organization

Each organization's memories are stored in its own bucket. Inside that bucket, each memory bank is a separate prefix (a folder-like path). A separate system bucket holds the engine's own coordination data and never holds memories.

This gives you a simple isolation boundary to reason about and audit:

• One organization's memories never share a bucket with another organization's.
• Access policies, encryption settings, logging and lifecycle rules can be applied at the bucket level for an organization.
• When an organization is created, Memlake creates its bucket. When an organization is deleted, Memlake deletes its bucket.

When to use Memlake

Memlake is a good fit when:

• You need to scale. Storage grows in object storage rather than in a database, and query nodes scale out horizontally because they are stateless. There is no single database writer to outgrow: any query node accepts writes, so you add capacity by adding nodes.
• You need low, consistent latency. Query nodes keep frequently used banks hot in memory and on local disk, and each bank's search indexes are purpose-built for retain and recall. Because load spreads across nodes instead of queuing on one database, latency stays low and predictable as memory volume and traffic grow, rather than degrading with them.
• You want cost to stay under control as you grow. Object storage is typically the lowest-cost durable storage in a cloud account or data center, and compute scales independently of storage, so you pay for query capacity you use rather than for an ever-larger database.
• You want durable storage with independent backups. Object stores are built for durability, and Memlake adds continuous archiving to a separate bucket.
• You want memories in storage you control. In a BYOC or on-premises deployment, memories sit in buckets in your own cloud account or your own object store.

Memlake's advantage shows up as banks get big, and as there are more of them. Small banks cost about the same on either store, and a small deployment can be cheaper on the database. As a rule of thumb, if your banks stay under roughly 50,000 memories each, the default database is a good fit. See Performance and cost for the numbers and Tradeoffs for a side-by-side comparison.

What stays the same

Memlake changes where memories are stored, not how you use Hindsight.

• The same API, Python SDK, TypeScript SDK and MCP server.
• The same operations: retain, recall, reflect and mental models.
• No client changes. Your applications and agents don't need to know whether an organization uses Memlake.
• Memory bank configuration, directives, webhooks, API keys and team membership are still managed by Hindsight in its database, exactly as before.

Memlake takes over only where memories live: memories, their links and indexes, the documents behind them, and mental models and knowledge pages (whose text is indexed for search) move to the bucket. The operational side (banks and their settings, the queue of background jobs, the audit log) stays in the database. That data is small and relational, and it changes the way databases handle well, with updates, joins and transactions, so each system does what it is best at.

Supported object stores

Object store	Typical deployment
Amazon S3	Private cloud or BYOC on AWS
Google Cloud Storage	Private cloud or BYOC on GCP
Azure Blob Storage	Private cloud or BYOC on Azure
S3-compatible object stores (such as MinIO and enterprise S3 storage appliances)	On-premises

For S3-compatible stores, the store must support conditional writes (If-None-Match and If-Match on PUT), which Memlake uses to coordinate writers safely, and must be reachable with path-style addressing. The Hindsight team confirms compatibility with your store during planning.

Availability by deployment option

Deployment option	Memlake	Where the buckets live
Hindsight Cloud with Enterprise features	Managed by Vectorize	Hindsight Cloud, one bucket per organization
Private cloud	Available	Vectorize's cloud account, dedicated to your deployment
Bring your own cloud (BYOC)	Available	Your AWS, GCP or Azure account
On-premises	Available	Your S3-compatible object store

Memlake runs as its own set of workloads alongside Hindsight in the same Kubernetes cluster.

Responsibilities

You provide	Vectorize provides and operates
For BYOC: the cloud account where the buckets are created	Installing, configuring and upgrading Memlake
For on-premises: an S3-compatible object store, an endpoint, and credentials for Memlake	Creating and deleting each organization's bucket
For on-premises: fast local storage for the query-node cache and indexer scratch space	Sizing the cache and indexer for your workload
A bucket location for backups (for BYOC and on-premises)	Configuring backups and running restores

Enabling Memlake

The Hindsight team enables Memlake in two steps:

1. Per deployment. Memlake is installed into your deployment. Installing it does not move any data on its own.
2. Per organization. Each organization is then switched to Memlake individually. All banks in an organization move together.

New organizations can start on Memlake from the beginning, before they hold any memories.

Migrating an existing organization

The Hindsight team can migrate an existing organization from the database to Memlake. The migration is designed to run against a live organization:

• Copy while live. Memories are copied to Memlake while your applications keep reading and writing as normal. Memory IDs are preserved, so any IDs your applications have stored keep working.
• Verify by content. Before cutover, every bank is compared record by record against the source, so changes made during the copy are caught and copied too.
• Brief write pause at cutover. For a short window, writes to that organization are paused while the final comparison runs and the organization is switched over. Reads continue throughout. Writes during the pause receive a retryable 503 response with a Retry-After header, so clients that retry will succeed once the pause ends. The pause is always lifted, even if the cutover is stopped.
• Safe to roll back. The migration does not delete the original copy in the database. If needed, the Hindsight team can move the organization back.

If the final verification does not pass, the organization is not switched over. The Hindsight team schedules migrations with you in advance so you can plan around the write pause.

Durability and backups

Memlake backs up every bank to a separate backup bucket, kept apart from the live data:

• Continuous archiving. The archiver continuously copies each bank's write-ahead log (the record of every change) to the backup bucket.
• Daily snapshots. A full base snapshot of every bank is taken daily.
• Separate failure domain. The backup bucket is in a different region from the live data where the platform supports it, and uses soft delete so deleted backup objects can be recovered within the retention window.

Together, a snapshot plus the archived log let the Hindsight team restore a bank to a specific point in time. A restore is written to a new location first and verified before anything is pointed at it, so the live bank is never overwritten by a restore in progress.

A Memlake restore brings back a bank's memories. A bank's settings (its configuration, directives, webhooks and other bank metadata) live in the Hindsight database and are restored from the database backup.

For deployment-wide backup and disaster recovery, see Operations.

Security

• Encryption at rest. Memories are encrypted at rest by the object store (Amazon S3, Google Cloud Storage, Azure Blob Storage, or your on-premises store's encryption).
• Workload identity. In the cloud, Memlake authenticates to the object store with workload identity or IAM roles, so no long-lived storage keys are stored in the deployment. On-premises, Memlake uses credentials you issue for your object store, held as a Kubernetes secret.
• Scoped access. Only Memlake's own storage identity can read and write the memory buckets; the rest of Hindsight reaches memories through Memlake.
• Per-organization buckets. Each organization's memories are in their own bucket, which keeps the isolation boundary visible in your cloud or storage console.

Next steps

• Read how Memlake works: Architecture, Guarantees, Tradeoffs, Performance and cost and the Concepts glossary.
• Compare deployment options in the deployment overview.
• Review on-premises requirements if you plan to run Memlake on your own object store.
• Read the launch post for the story behind Memlake, or get the 35-page technical whitepaper.
• Book a call or contact your Hindsight representative to plan Memlake for your deployment.