MMDBBench

Agent memory is becoming an important workload. An agent keeps what it learns as facts and events (documents and tables), the relations between them (a graph), their meaning (vectors), their words (text), and their time. It asks one database to serve all of these at once, usually in many small collections that change on every turn: a few inserts, then a query right away, with other agents writing next to it. Most benchmarks measure one data model at a time, so the cost of putting them in one engine is not visible.

MMDBBench is an open benchmark of multi-model databases: databases that serve tables, time series, documents, graphs, key-values, and vectors from one storage engine. Specialist databases built for a single model run the same tasks as per-model baselines, so a reader can see what one engine gives up or gains against the engine built for that job.

Every engine in a lane answers the same tasks over the same data on the same x86 Linux machine, in SQL, Cypher, or its native language. The settings are published, and every result is tied to the exact version of the engine that produced it. The raw result rows are open.

Results are kept. When an engine ships a new version, new rows are added next to the old ones. When a setting is corrected, the corrected run is added and the earlier result stays visible.

MMDBBench does not rank databases in general. It measures the tasks listed in each lane, on the stated machine and settings. A different workload, scale or tuning can change the order. That is why the settings are published, and why anyone can tell us when one is wrong.

There are six lanes. An engine appears only in the lanes it is built for.

• Tables: Documents and structured query language (SQL) analytics.
• Graphs: Interactive and analytical queries on a social network graph.
• Vectors: Dense and sparse vector search, with recall beside every latency.
• Time series: Monitoring-style queries over timestamped readings.
• Cross-model: One operation across a vector, a graph edge, and a document.
• Lifecycle: Opening, closing, deployment boundaries, and what waiting for the disk costs.

The first results are published in November 2026: ArcadeDB 26.11.1 against a multi-model subset of engines, measured on the November instrument. Until then MMDBBench shows its method and its lanes, and no numbers.

Each lane below says what it measures and which tables it will carry. The method above is what every result will follow.

These points summarise the rules every published result follows. A result that breaks one is published as a declared exception, or not at all.

1. Same task, same data, same machine. Every engine in a lane answers the same queries over the same corpus on the same dedicated machine.
2. Answers are checked. Each query's answer is reduced to a digest and compared between engines. A fast wrong answer is a failure, not a result.
3. Settings are published. Each engine runs at its documented defaults unless the task needs a change. Every change is listed with its reason, and an index added for one engine is added in its equivalent form for every engine that has one.
4. Equal resources. Every engine gets the same memory cap and processor allotment inside a lane, and the same durability settings are measured for every engine that offers them.
5. Repeated runs. Each cell is run several times and the page prints the median with its spread, never a single run. Whether a timing includes the first run, a warm-up or an index build is stated in the table.
6. Limits and failures are results. A query that cannot finish within its time budget is recorded as censored, with the budget. An engine that runs out of memory, times out or refuses a task has that recorded for the cell, with the error it gave.
7. Versions are exact. Every row records the engine version and, for containers, the image digest. A cell never mixes rows from different versions.
8. History is kept. A correction or a new version adds rows. The earlier rows stay visible.

The full rules are in the methodology. The exact settings for each engine, and the reason for each one, are on the engines page. The code is at github.com/humemai/mmdbbench.

Most people arrive with tables, so this lane comes first. The tasks are TPC-C and TPC-H from the Transaction Processing Performance Council, the standard transaction and analytics workloads. TPC-C is many small reads and writes. TPC-H is a few large scanning queries.

The tables lane, with the tables it will publish.

The tasks come from the Linked Data Benchmark Council's Social Network Benchmark. The interactive half reads a graph at several depths: a point lookup, one hop, two hops, and a three-hop read with a property filter. It also inserts, updates and deletes.

The graphs lane, with the tables it will publish.

Dense and sparse vectors are different problems with different engines behind them, so they get separate tables. Both report recall beside every latency. An approximate index can be made fast by returning worse answers, so a latency without the share of true neighbours it found is not a comparable number.

The vectors lane, with the tables it will publish.

The task is the Time Series Benchmark Suite on its CPU data set. The queries are the ones a monitoring stack asks: the newest reading per sensor, a twelve-hour aggregate, a double group-by over host and hour, a high-usage filter, and a grouped, ordered, limited query of the shape a dashboard issues.

The time series lane, with the tables it will publish.

Most lanes compare engines that do one thing. This lane asks what a single engine buys when one operation touches more than one data model. The comparators are other engines that hold several models and, beside them, a composed stack: separate stores with application code between them.

The cross-model lane, with the tables it will publish.

These are the costs that nobody benchmarks and everybody pays. A session table opens a database, does one thing and closes it, for each kind of data the engine can hold, embedded and served. It is the cold measurement in its purest form.

The lifecycle lane, with the tables it will publish.

If a setting we use for your engine differs from what you recommend, open an issue. Say which engine and version, what we use, what you recommend, and where your documentation says so.

We re-run the affected cells with your configuration on the same machine, and publish them beside the earlier result with the reason. The earlier result stays in the history.

Open an issue on GitHub. The contribution guide also covers adding an engine, a query or a lane, and reporting a problem with a result.

Four academic benchmarks already address multi-model databases.

• UniBench, from the University of Helsinki: relational, JSON, XML, key-value, and graph data in one social commerce scenario, with multi-model queries and transactions.
• M2Bench, from Seoul National University: analytic tasks over relational, document, property graph, and array data, run on polyglot persistence (MySQL, MongoDB, Neo4j, and SciDB), ArangoDB, and AgensGraph.
• MMSBench-Net, from the University of Basel: a scenario-based benchmark over relational, document, and graph data, used to compare Polypheny and SurrealDB.
• GRBench, by Liu and colleagues: relational and graph data together, evaluated on relational extensions, multi-model databases, and dedicated graph-relational systems.

None of the four lists vectors or time series among its data models. MMDBBench adds both, and it runs a specialist database for each model as a baseline, so the question is what one engine gains or gives up against a specialist, not only how two multi-model systems compare. Tasks are written in SQL, Cypher, and native languages, on one x86 Linux platform with pinned versions, and the result rows are open.

MMDBBench is run by HumemAI. HumemAI also builds the ArcadeDB Python package. It is measured by the same rules as every other engine: the same tasks, the same machine, published settings, and a result for every failure.

The code and the settings are public, so anyone can check a result or reproduce it. When a result is wrong, the fix is a new row, not an edit.

@misc{kim2026mmdbbench,
      title={{MMDBBench}: an open benchmark of multi-model databases},
      author={Taewoon Kim},
      year={2026},
      publisher={Zenodo},
      doi={10.5281/zenodo.23481288},
      url={https://github.com/humemai/mmdbbench},
}

Cite a release of the repository rather than this page, because the page changes with every measurement. The DOI above covers every release; the Zenodo record lists the DOI of each release, for when the exact numbers matter.