Measurements are added as each run completes. A table that is not finished says so where its numbers will go.

Project

DBBench

An independent, open benchmark of database engines. The same tasks on the same machine, with published settings and every result tied to the exact engine version.

Documentation GitHub

DBBench

The same tasks, on the same machine, for every engine.

DBBench is an independent, open benchmark of database engines. Every engine in a lane answers the same tasks over the same data on the same machine. The settings are published, and every result is tied to the exact version of the engine that produced it.

Results are kept. When an engine ships a new version, new rows are added next to the old ones. When a setting is corrected, the corrected run is added and the earlier result stays visible.

DBBench does not rank databases in general. It measures the tasks listed in each lane, on the stated machine and settings. A different workload, scale or tuning can change the order. That is why the settings are published, and why anyone can tell us when one is wrong.

There are six lanes. An engine appears only in the lanes it is built for.

Tables: Documents and structured query language (SQL) analytics.
Graphs: Interactive and analytical queries on a social network graph.
Vectors: Dense and sparse vector search, with recall beside every latency.
Time series: Monitoring-style queries over timestamped readings.
Cross-model: One operation across a vector, a graph edge, and a document.
Lifecycle: Opening, closing, deployment boundaries, and what waiting for the disk costs.

Method

How a result is produced.

These points summarise the rules every published result follows. A result that breaks one is published as a declared exception, or not at all.

Same task, same data, same machine. Every engine in a lane answers the same queries over the same corpus on the same dedicated machine.
Answers are checked. Each query's answer is reduced to a digest and compared between engines. A fast wrong answer is a failure, not a result.
Settings are published. Each engine runs at its documented defaults unless the task needs a change. Every change is listed with its reason, and an index added for one engine is added in its equivalent form for every engine that has one.
Equal resources. Every engine gets the same memory cap and processor allotment inside a lane, and the same durability settings are measured for every engine that offers them.
Repeated runs. Each cell is run several times and the page prints the median with its spread, never a single run. Whether a timing includes the first run, a warm-up or an index build is stated in the table.
Limits and failures are results. A query that cannot finish within its time budget is recorded as censored, with the budget. An engine that runs out of memory, times out or refuses a task has that recorded for the cell, with the error it gave.
Versions are exact. Every row records the engine version and, for containers, the image digest. A cell never mixes rows from different versions.
History is kept. A correction or a new version adds rows. The earlier rows stay visible.

The full rules are in the methodology. The exact settings for each engine, and the reason for each one, are on the engines page. The code is at github.com/humemai/dbbench.

Every table was measured on the machine below. A table lists its own conditions beneath its caption.

Processor (benchmark machine): Intel Core i9-12900HK, 14 cores (6 performance, 8 efficient), 20 threads, 24 MiB L3
Memory: 64 GiB
Storage: Samsung 980 PRO 2 TB NVMe
Software: Ubuntu 26.04 LTS, Linux 7.0, Docker 29
Every engine is pinned to the same cores (cpuset 0-11).

Lane

Tables.

Most people arrive with tables, so this lane comes first. The tasks are TPC-C and TPC-H from the Transaction Processing Performance Council, the standard transaction and analytics workloads. TPC-C is many small reads and writes. TPC-H is a few large scanning queries.

Online transaction processing: TPC-C's new-order and payment transactions on the TPC-H tables, and the single-record insert, read, update, and delete beside them. Each column is a warm median; the ninety-ninth percentile is printed for new-order, the transaction this table leads with. There is no cold column of its own because every operation runs against an already-open, already-warm database by construction.

Document OLTP TPC-C new-order on the TPC-H tables (SF1, SF10)
Engine	Mode	Size	new-order p50 ms ↓	new-order p99 ms ↓	payment p50 ms ↓	insert p50 ms ↓	read p50 ms ↓	update p50 ms ↓	delete p50 ms ↓	OLTP ops/s ↑	ingest documents/s ↑	ingest total s ↓	ingest s ↓	index s ↓	peak memory GiB ↓	disk GiB ↓
ArcadeDB	embedded	TPC-H SF1 (6.0M line items)	0.46	0.90	0.20	0.10	0.030	0.086	0.19	2,586	47k	132.5	126.3	6.11	6.37	0.67
ArcadeDB	server	TPC-H SF1 (6.0M line items)	1.21	2.38	0.89	0.69	0.60	0.64	0.78	888	35k	178.7	172.6	6.21	6.42	0.67
ArangoDB	server	TPC-H SF1 (6.0M line items)	2.55	3.75	2.57	0.50	0.46	0.53	0.51	384	91k	67.8	67.8	0	1.22	1.04
DuckDB	embedded	TPC-H SF1 (6.0M line items)	9.89	21.9	10.5	9	0.16	8.89	8.94	94	512k	12.1	12.1	0.010	0.58	0.073
MongoDB	server	TPC-H SF1 (6.0M line items)	0.48	0.83	0.45	0.080	0.092	0.11	0.11	2,001	98k	63.1	54.9	7.78	4.50	0.88
PostgreSQL	server	TPC-H SF1 (6.0M line items)	0.084	0.13	0.080	0.043	0.040	0.046	0.039	12k	222k	28.0	26.5	1.42	0.52	0.93
SQLite	embedded	TPC-H SF1 (6.0M line items)	0.012	0.027	0.010	0.0060	0.0020	0.0060	0.0050	21k	225k	27.5	27.5	0	0.42	0.31
SurrealDB	embedded	TPC-H SF1 (6.0M line items)	0.16	0.20	0.14	0.080	0.049	0.077	0.12	6,518	10k	599.5	599.5	0	6.89	2.62
SurrealDB	server	TPC-H SF1 (6.0M line items)	re-run	re-run	re-run	re-run	0.13	re-run	re-run	re-run	20k	305.6	305.4	0.14	6.14	0.48
ArcadeDB	embedded	TPC-H SF10 (60.0M line items)	0.45	0.92	0.18	0.096	0.032	0.083	0.18	2,734	48k	1,288	1,227	59.8	18.0	6.84
ArcadeDB	server	TPC-H SF10 (60.0M line items)	1.32	2.44	0.85	0.73	0.62	0.64	0.82	873	37k	1,685	1,623	61.1	18.1	6.88
ArangoDB	server	TPC-H SF10 (60.0M line items)	2.88	4.18	2.63	0.51	0.47	0.54	0.51	351	89k	699.7	699.7	0	6.64	4.74
DuckDB	embedded	TPC-H SF10 (60.0M line items)	11.4	20.9	10.5	9.14	0.16	8.95	8.97	88	502k	123.4	123.4	0.010	2.66	0.84
MongoDB	server	TPC-H SF10 (60.0M line items)	0.59	0.94	0.45	0.081	0.093	0.12	0.11	1,869	101k	614.1	547.6	66.6	16.6	8.87
PostgreSQL	server	TPC-H SF10 (60.0M line items)	0.090	0.16	0.085	0.042	0.040	0.044	0.039	11k	224k	277.0	265.1	11.9	1.54	6.23
SQLite	embedded	TPC-H SF10 (60.0M line items)	0.012	0.019	0.010	0.0060	0.0020	0.0050	0.0050	33k	229k	271.1	271.1	0	1.65	3.21
SurrealDB	embedded	TPC-H SF10 (60.0M line items)	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM
SurrealDB	server	TPC-H SF10 (60.0M line items)	re-run	re-run	re-run	re-run	0.14	re-run	re-run	re-run	19k	3,295	3,294	0.14	30.9	4.42

↑ higher is better, ↓ lower is better; the best value in each column, within one size, is bold

Measured rows: benchmarks/experiments/results/runs_paper_oct.csv

Exact builds measured

arcadedb 26.10.1-dev · 417314c18 (in-process)
arcadedb 26.10.1-dev · 417314c18 arcadedb-c25:417314c18
arangodb 3.12.11 arangodb@563cb2c07af0…
duckdb 1.5.4
mongodb 8.2.12 mongo@41afd6e1183f…
postgresql 18.6 postgres@7341002d2b8c…
sqlite 3.46.1
surrealdb 2.3.10
surrealdb 3.2.4 surrealdb/surrealdb@6a5002363ff5…

Six transactional operations. New-order: TPC-C's checkout transaction: it reads a customer and a warehouse, inserts an order with its line items, and updates stock, all in one transaction. Payment: TPC-C's payment against one of the orders new-order just placed, chosen at random: the order is read, marked paid, and the payment inserted, in one transaction. Insert: one record inserted by key. Read: one record read back by key. Update: one record's quantity updated by key. Delete: one record deleted by key. OLTP ops/s is the rate of the two TPC-C transactions together; the four single-record operations are timed one at a time.
Disk is what the workload left on disk, in GiB: the engine's writable layer plus its volumes after the cell, minus the same engine's empty footprint.
SurrealDB (server)'s write cells are marked `re-run`: the server ran at its default, a sync to disk at every commit, although it has a setting that does not wait, and every other engine on this table that has such a setting ran at it. Our harness missed that setting, and the next measurement runs its writes at it. Its read cells commit nothing and stay.
The operations on this table are timed from the first one after the data is loaded, so its columns include each engine's warm-up. On ArcadeDB (embedded) and ArcadeDB (server), which run on a JVM, that warm-up includes the JVM compiling the statement path. The next measurement gives every engine an untimed warm-up on separate keys first, prints the warm median, and adds the first operation as the cold column.
Durability is matched at the relaxed end: on every engine that has the setting, a commit returns without waiting for the disk and the log is flushed by the engine's own background policy, which is ArcadeDB's engine default and a documented production mode for each of the others. Each row records what it ran as `durability`, and the setting was read out of the engine rather than assumed on every engine except ArcadeDB (server) and SurrealDB (server). ArcadeDB (server) is given the setting when the server starts, and the row records it, because the server offers no way to read it back. The exception on this table is DuckDB, which has no setting to relax and waits for the disk at every commit; its write and transaction cells are paying for that. SurrealDB (server)'s rows record that no setting was found, and its server log shows what it ran: its default in both runs, a sync to disk at every commit (the log reads "Sync mode: every transaction commit" in every run), although it has a setting that does not wait; the next measurement runs both.
PostgreSQL's memory cell is not comparable to the others: this column counts memory an engine holds in its own address space, and PostgreSQL holds its data in shared memory and the kernel's file cache instead. At TPC-H SF10 (60.0M line items), of the 1.54 GiB shown, 1.464 GiB is our Python client and 0.070 GiB is the database.
ArcadeDB (embedded) and ArcadeDB (server) run on a JVM, and for them this column is close to the heap they were given rather than the memory the work needed: ArcadeDB (server)'s 6.42 GiB here is about 80 per cent of its 8g heap. Every engine on a table gets the same memory envelope; how much of it a JVM takes is a property of the runtime, so read these cells against each other rather than against an engine that manages its own memory.
Each repetition runs 1,000 new-order transactions and 1,000 payments, then 1,000 each of the single-record insert, read, update, and delete; the p50 and p99 are over those, and OLTP ops/s is the rate of the two transactions together.
`ingest total s` is the whole of building this corpus. `ingest s` and `index s` are the two phases timed inside it -- writing the data, then building whatever index the engine was given -- and they are not a division of it: each engine's ingest clock starts after the setup it does first, creating a schema or a collection, which neither phase counts. How much is left over depends on the engine: nothing on some of the ones here, up to about 1 per cent on others. Read the two as what each phase cost, and the total as what the whole of it cost.
SurrealDB (embedded) at TPC-H SF10 (60.0M line items): the transaction cell reached the 32g memory envelope (peak 32,768 MiB) and was killed by the kernel, the same envelope every engine on this table had; it did not run out of time, and there is no row.
In the cells: `OOM` means killed at the cell's memory envelope rather than running out of time. `re-run` means the cell was measured before a fix to our harness and stays off the table until it is measured again; the note says what was wrong. A dash is an operation the engine cannot express, never a slow one.
Ingest paths: ArcadeDB embedded loads through the Python package's insert_many in 10,000-row batches, one JSON payload per batch; served sends INSERT statements as sqlscript batches over HTTP; PostgreSQL COPY FROM STDIN; DuckDB CREATE TABLE AS SELECT from in-memory frames; SQLite executemany; MongoDB insert_many; ArangoDB import_bulk; SurrealDB inserts through its Python SDK.

Every tables table, with the rest of the tasks in this lane.

Lane

Graphs.

The tasks come from the Linked Data Benchmark Council's Social Network Benchmark. The interactive half reads a graph at several depths: a point lookup, one hop, two hops, and a three-hop read with a property filter. It also inserts, updates and deletes.

Graph interactive queries: point lookup, one hop, two hops, and a three-hop read with a property filter, with insert, update, and delete beside them. Warm medians, with the ninety-ninth percentile on the point lookup.

Graph OLTP LDBC-SNB Interactive (SF1, SF10)
Engine	Mode	Size	point p50 ms ↓	point p99 ms ↓	1-hop p50 ms ↓	2-hop p50 ms ↓	3-hop filtered p50 ms ↓	insert p50 ms ↓	update p50 ms ↓	delete p50 ms ↓	ingest vertices+edges/s ↑	ingest total s ↓	peak memory GiB ↓	disk GiB ↓
ArcadeDB	embedded	SF1 (10k people, 181k friendships)	0.48	0.91	0.45	0.71	5.57	0.49	0.20	0.49	87k	2.18	2.79	0.013
ArcadeDB	server	SF1 (10k people, 181k friendships)	1.23	1.84	1.18	1.47	6.61	1.26	0.86	1.07	25k	7.54	3.36	0.012
ArangoDB	server	SF1 (10k people, 181k friendships)	0.72	0.96	1.00	2.28	17.0	0.80	0.68	0.86	98k	1.94	0.50	0.048
DuckPGQ	embedded	SF1 (10k people, 181k friendships)	0.25	0.29	0.93	2.24	3.30	10.5	9.06	10.5	187k	1.02	0.19	0.041
FalkorDB	server	SF1 (10k people, 181k friendships)	0.13	0.28	0.22	0.34	0.61	0.16	0.12	0.76	41k	4.70	0.18	0
LadybugDB	embedded	SF1 (10k people, 181k friendships)	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run
Memgraph	server	SF1 (10k people, 181k friendships)	0.15	0.35	0.15	0.21	1.17	0.15	0.14	0.14	72k	2.66	0.46	0.024
MongoDB	server	SF1 (10k people, 181k friendships)	0.090	0.28	0.27	0.90	78.9	0.23	0.086	0.32	131k	1.45	0.25	0.022
Neo4j	server	SF1 (10k people, 181k friendships)	6.11	12.7	5.66	4.66	6.29	7.14	5.89	6.06	37k	5.21	4.74	0.015
SurrealDB	embedded	SF1 (10k people, 181k friendships)	0.044	0.051	0.69	15.2	190.0	0.18	0.083	0.15	14k	13.8	0.80	0.10
SurrealDB	server	SF1 (10k people, 181k friendships)	0.11	0.61	0.17	0.57	28.8	re-run	re-run	re-run	20k	9.51	0.31	0.047
ArcadeDB	embedded	SF10 (66k people, 1.9M friendships)	0.51	1.05	0.62	1.55	23.4	0.52	0.20	0.46	87k	23	8.19	0.13
ArcadeDB	server	SF10 (66k people, 1.9M friendships)	1.13	1.67	1.30	2.36	24.9	1.32	0.83	1.15	28k	71.9	7.99	0.14
ArangoDB	server	SF10 (66k people, 1.9M friendships)	0.90	1.12	1.06	6.00	92.4	0.91	0.74	0.93	91k	21.9	0.78	0.41
DuckPGQ	embedded	SF10 (66k people, 1.9M friendships)	0.26	0.33	1.33	4.27	6.96	10.6	9.02	10.8	275k	7.28	0.31	0.16
FalkorDB	server	SF10 (66k people, 1.9M friendships)	0.13	0.42	0.24	0.66	1.51	0.17	0.10	0.21	24k	85.3	0.37	0
LadybugDB	embedded	SF10 (66k people, 1.9M friendships)	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run
Memgraph	server	SF10 (66k people, 1.9M friendships)	0.15	0.40	0.15	0.30	3.71	0.15	0.14	0.14	71k	28.1	0.66	0.24
MongoDB	server	SF10 (66k people, 1.9M friendships)	0.089	0.33	0.40	1.71	345.2	0.22	0.086	0.32	161k	12.5	1.10	0.19
Neo4j	server	SF10 (66k people, 1.9M friendships)	7.18	12.3	7.07	6.23	13.2	7.54	6.23	6.45	44k	45.4	13.1	0.66
SurrealDB	embedded	SF10 (66k people, 1.9M friendships)	0.047	0.058	1.38	60.7	1,217	0.20	0.089	0.18	11k	176.2	5.67	1.09
SurrealDB	server	SF10 (66k people, 1.9M friendships)	0.17	0.60	0.40	1.24	210.5	re-run	re-run	re-run	18k	111.3	2.14	0.26

↑ higher is better, ↓ lower is better; the best value in each column, within one size, is bold

Measured rows: benchmarks/experiments/results/runs_paper_oct.csv

Exact builds measured

arcadedb 26.10.1-dev · 417314c18 (in-process)
arcadedb 26.10.1-dev · 417314c18 arcadedb-c25:417314c18
arangodb 3.12.11 arangodb@563cb2c07af0…
duckdb 1.5.4 + duckpgq f386a6c
falkordb 4.20.6 falkordb/falkordb@0a9fe4d1ee0b…
memgraph 3.13.1 memgraph/memgraph@4710bee1ab5b…
mongodb 8.2.12 mongo@41afd6e1183f…
neo4j 2026.08.1 neo4j@e702d6b535d9…
surrealdb 2.3.10
surrealdb 3.2.4 surrealdb/surrealdb@6a5002363ff5…

Disk is what the workload left on disk, in GiB: the engine's writable layer plus its volumes after the cell, minus the same engine's empty footprint.
LadybugDB's row is marked `re-run`: it sized its thread count and buffer pool from the whole machine rather than from the cores and memory its cell was given, while every other engine here that sizes itself from the machine was fitted to its cell. It is being re-measured with both fitted.
SurrealDB (server)'s write cells are marked `re-run`: the server ran at its default, a sync to disk at every commit, although it has a setting that does not wait, and every other engine on this table that has such a setting ran at it. Our harness missed that setting, and the next measurement runs its writes at it. Its read cells commit nothing and stay.
In this measurement every person's age loaded as zero, through a date-parsing error in our data loader, so the 3-hop filtered read (age above 30) matched no one on any engine: its times are three hops plus a check on every end node. The next measurement parses the ages correctly.
Each friendship is one edge, pointing from the person with the smaller id to the person with the larger. The engines can follow an edge from either end, but the 1-hop, 2-hop, and 3-hop reads follow it only the way it points, so 1-hop counts only the friends with a larger id than the start person, and 2-hop and 3-hop reach only people along a chain of rising ids. The next measurement asks the same questions in both directions of a friendship.
Each read on this table is timed twice over the same start persons, and the table prints the first pass, so its read columns include each engine's warm-up. On ArcadeDB (embedded), ArcadeDB (server), and Neo4j, which run on a JVM, that warm-up includes the JVM compiling the query path. The next measurement gives every engine an untimed warm-up on other start persons first, prints the warm median, and adds the first query of a session as the cold column.
Durability is matched at the relaxed end: on every engine that has the setting, a commit returns without waiting for the disk and the log is flushed by the engine's own background policy, which is ArcadeDB's engine default and a documented production mode for each of the others. Each row records what it ran as `durability`, and the setting was read out of the engine rather than assumed on every engine except ArcadeDB (server) and SurrealDB (server). ArcadeDB (server) is given the setting when the server starts, and the row records it, because the server offers no way to read it back. The exceptions on this table are DuckPGQ and Neo4j, which have no setting to relax and wait for the disk at every commit; their write and transaction cells are paying for that. SurrealDB (server)'s rows record that no setting was found, and its server log shows what it ran: its default in both runs, a sync to disk at every commit (the log reads "Sync mode: every transaction commit" in every run), although it has a setting that does not wait; the next measurement runs both.
Every engine traverses the same persons-and-KNOWS projection, with edges stored in both directions.
ArcadeDB (embedded), ArcadeDB (server), and Neo4j run on a JVM, and for them this column is close to the heap they were given rather than the memory the work needed: Neo4j's 4.74 GiB here is about 119 per cent of its 4g heap. Every engine on a table gets the same memory envelope; how much of it a JVM takes is a property of the runtime, so read these cells against each other rather than against an engine that manages its own memory.
Every repetition, on every engine, runs each read against the same start persons, 500 at SF1 (10k people, 181k friendships) and 200 at SF10 (66k people, 1.9M friendships), plus 1,000 each of the insert, update, and delete; the p50 and p99 are over those. The start persons are picked at random with a fixed seed and with replacement, so one person can be picked more than once.
FalkorDB at SF1 (10k people, 181k friendships) and SF10 (66k people, 1.9M friendships): its disk cells are 0.0 because the server container did not grow over its empty footprint during the run: at the relaxed setting FalkorDB persists only by timed snapshots, with no append-only file, and none had been taken when disk was measured, so the graph was held in memory only; read them as a floor under 0.01 GiB, not as a size.
In the cells: `re-run` means the cell was measured before a fix to our harness and stays off the table until it is measured again; the note says what was wrong. A dash is an operation the engine cannot express, never a slow one.
Ingest paths: ArcadeDB embedded loads through the Java API (newVertex, newEdge) in 5,000-record transactions; served sends CREATE VERTEX and CREATE EDGE statements as sqlscript batches over HTTP; Neo4j and Memgraph UNWIND batches over bolt; FalkorDB the same UNWIND batches over the Redis protocol; LadybugDB COPY from CSV, its native bulk path; DuckPGQ registers each batch as an Arrow table and INSERT ... SELECTs from it into the Person and knows tables, then defines the property graph over them; ArangoDB import_bulk; MongoDB insert_many; SurrealDB inserts the persons through its Python SDK and the KNOWS edges as bulk relation inserts.

Every graphs table, with the rest of the tasks in this lane.

Lane

Vectors.

Dense and sparse vectors are different problems with different engines behind them, so they get separate tables. Both report recall beside every latency. An approximate index can be made fast by returning worse answers, so a latency without the share of true neighbours it found is not a comparable number.

Dense search, and search after an insert and after a delete into the same built index, with recall beside each. Every row states what it stores, full precision or quantized, because a quantized index and a full-precision one are not the same measurement.

Dense vector search DEEP-10M (deep-image-96-angular) and SIFT-1M
Engine	Mode	Precision	Size	cold p50 ms ↓	cold p99 ms ↓	warm p50 ms ↓	warm p99 ms ↓	after insert p50 ms ↓	after delete p50 ms ↓	insert into index ms/vector	recall@10 after insert	delete from index ms/vector	recall@10 after delete	recall@10 ↑	ingest+index vectors/s ↑	ingest s ↓	index s ↓	ingest+index total s ↓	peak memory GiB ↓	disk GiB ↓
ArcadeDB	embedded	int8	1M vectors	1.36	2.26	0.75	1.12	0.95	0.78	0.044	0.98	4.64	0.98	0.98	6,874	16.2	129.3	145.5	7.79	0.93
ArcadeDB	embedded	fp32	1M vectors	1.96	4.20	0.74	1.12	0.91	0.78	0.037	0.99	4.72	0.99	0.99	6,320	16.0	142.0	158.2	7.63	0.80
ArcadeDB	server	int8	1M vectors	2.45	3.62	1.74	2.29	1.74	1.79	0.16	0.98	6.95	0.98	0.98	4,868	87.4	117.4	205.4	8.37	0.94
ArcadeDB	server	fp32	1M vectors	2.94	5.54	1.78	2.23	1.86	1.81	0.10	0.99	6.93	0.99	0.99	4,626	87.1	128.8	216.2	7.65	0.81
ArangoDB	server	fp32	1M vectors	5.85	7.55	5.67	7.52	5.62	5.63	0.069	0.99	0.056	0.99	0.99	11k	—	—	92.7	1.83	0.98
Chroma	embedded	fp32	1M vectors	0.65	0.85	0.66	0.89	0.66	0.67	0.52	0.98	0.47	0.98	0.97	6,820	—	—	146.6	1.15	0.78
DuckDB VSS	embedded	fp32	1M vectors	1.77	3.42	1.77	2.06	2.03	2.02	10.7	0.97	1.78	0.98	0.98	17k	—	—	58.4	2.83	1.38
LanceDB	embedded	int8	1M vectors	1.55	2.81	1.58	2.33	20.5	19.8	0.0040	0.97	0.025	0.97	0.97	35k	0.86	27.4	28.4	2.02	0.89
Milvus	server	int8	1M vectors	1.13	2.40	1.09	1.78	1.14	1.13	8.56	0.97	2.53	0.97	0.97	7,497	13.6	117.8	133.4	2.93	3.13
Milvus	server	fp32	1M vectors	1.17	2.01	1.10	1.93	1.15	1.12	8.55	0.98	1.53	0.98	0.98	7,738	13.6	113.9	129.2	4.24	4.14
MongoDB	server	fp32	1M vectors	6.37	19.6	4.01	4.98	4.30	4.54	1.07	0.98	0.023	0.98	0.98	17k	17.3	41.2	58.8	5.10	2.54
Neo4j	server	fp32	1M vectors	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run
pgvector	server	fp32	1M vectors	1.82	2.24	1.81	2.21	1.81	1.74	1.08	0.99	0.079	0.99	0.99	6,506	21.8	132.1	153.7	0.055	2.56
Qdrant	server	int8	1M vectors	0.93	1.17	0.93	1.16	0.93	0.92	0.054	1.00	0.013	1.00	1.00	32k	—	—	31.4	1.23	2.44
Qdrant	server	fp32	1M vectors	1.09	1.45	1.11	1.44	1.10	1.08	0.058	1.00	0.013	1.00	1.00	23k	—	—	43.1	0.92	0.69
sqlite-vec	embedded	fp32	1M vectors	124.9	127.4	124.8	127.1	125.5	124.9	0.038	1.00	0.13	1.00	1.00	156k	—	—	6.39	0.042	0.50
SurrealDB	embedded	—	1M vectors	>2h	>2h	>2h	>2h	>2h	>2h	>2h	>2h	>2h	>2h	>2h	>2h	>2h	>2h	>2h	>2h	>2h
SurrealDB	server	fp32	1M vectors	6.25	8.57	4.00	5.43	4.44	4.52	0.17	0.98	8.03	0.98	0.98	551	159.8	1,654	1,814	13.6	0.74
ArcadeDB	embedded	int8	9.99M vectors	2.77	5.55	0.87	1.37	—	—	—	—	—	—	0.94	5,047	143.0	1,843	1,979	28.3	8.57
ArcadeDB	embedded	fp32	9.99M vectors	18.9	126.2	1.01	1.50	—	—	—	—	—	—	0.95	4,664	148.8	1,991	2,142	28.3	7.56
ArcadeDB	server	int8	9.99M vectors	3.77	7.83	2.00	2.45	—	—	—	—	—	—	0.94	3,622	1,004	1,755	2,758	28.4	8.59
ArcadeDB	server	fp32	9.99M vectors	23.8	133.5	2.13	2.68	—	—	—	—	—	—	0.95	3,387	1,001	1,961	2,949	28.5	7.60
ArangoDB	server	fp32	9.99M vectors	n/c	n/c	n/c	n/c	n/c	n/c	n/c	n/c	n/c	n/c	n/c	n/c	n/c	n/c	n/c	n/c	n/c
Chroma	embedded	fp32	9.99M vectors	0.70	0.91	0.71	0.89	—	—	—	—	—	—	0.93	2,650	—	—	3,770	12.7	6.59
DuckDB VSS	embedded	fp32	9.99M vectors	2.58	7.32	2.58	3.05	—	—	—	—	—	—	0.93	12k	—	—	833	25.2	8.08
LanceDB	embedded	int8	9.99M vectors	1.92	2.37	1.94	2.39	—	—	—	—	—	—	0.94	34k	6.59	287.9	295.0	11.0	7.11
Milvus	server	int8	9.99M vectors	1.49	1.82	1.46	1.77	—	—	—	—	—	—	0.96	7,654	193.2	1,108	1,305	13.6	25.2
Milvus	server	fp32	9.99M vectors	1.56	1.84	1.50	1.74	—	—	—	—	—	—	0.98	8,056	195.9	1,043	1,240	16.9	33.2
MongoDB	server	fp32	9.99M vectors	13.7	39.0	6.76	9.89	—	—	—	—	—	—	0.95	13k	135.5	642.0	777.2	22.0	22.9
Neo4j	server	fp32	9.99M vectors	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run
pgvector	server	fp32	9.99M vectors	2.84	3.78	2.52	3.33	—	—	—	—	—	—	0.95	5,153	272.6	1,665	1,939	3.73	17.1
Qdrant	server	int8	9.99M vectors	1.06	1.49	1.08	1.39	—	—	—	—	—	—	0.98	24k	—	—	412.6	7.10	5.34
Qdrant	server	fp32	9.99M vectors	1.35	1.88	1.26	1.75	—	—	—	—	—	—	0.98	18k	—	—	546.5	6.02	4.42
sqlite-vec	embedded	int8	9.99M vectors	671.0	680.6	671.0	678.6	—	—	—	—	—	—	0.92	174k	—	—	57.3	3.77	1.12
sqlite-vec	embedded	fp32	9.99M vectors	897.7	915.7	896.5	912.9	—	—	—	—	—	—	1	169k	—	—	59.0	3.61	3.80
SurrealDB	embedded	—	9.99M vectors	>8h	>8h	>8h	>8h	>8h	>8h	>8h	>8h	>8h	>8h	>8h	>8h	>8h	>8h	>8h	>8h	>8h
SurrealDB	server	—	9.99M vectors	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM

↑ higher is better, ↓ lower is better; the best value in each column, within one size, is bold

Measured rows: benchmarks/experiments/results/runs_paper_oct.csv, benchmarks/experiments/results/dense_mp5_<pin or 2681>

Exact builds measured

arcadedb 26.10.1-dev · 417314c18
arangodb 3.12.11
chroma 1.5.9
duckdb vss 1.5.4
lancedb 0.39.0
milvus 3.0.1
mongodb 8.2.12 + MongoDb Search Community Version 1.70.4
pgvector 0.8.6
qdrant 1.19.1
sqlite-vec 0.1.9
surrealdb 3.2.4

ingest s and index s are two timers where the engine has the boundary: inserting the vectors, then building the index. ArangoDB (fp32), Chroma (fp32), DuckDB VSS (fp32), Qdrant (fp32), Qdrant (int8), sqlite-vec (fp32), and sqlite-vec (int8) record one timer only, because the index is built while ingesting or the adapter times no boundary, so their ingest s and index s are blank. ingest+index total s is one timer around both, and ingest+index vectors/s divides the vector count by it.
Disk is what the workload left on disk, in GiB: the engine's writable layer plus its volumes after the cell, minus the same engine's empty footprint.
Neo4j's row is marked `re-run`: its vector index ran at Neo4j's default quantization, binary with a three-fold search expansion, which our index definition did not override, while the row recorded it as unquantized. The next measurement sets the quantization explicitly, unquantized here and as a separate scalar-quantized row.
The insert and delete into a built index, the search after each, and the recall after each are measured at 1M vectors only. At 9.99M vectors those columns are blank because the pass is not run there, not because an engine failed it. These engines do not all maintain an index the same way: ArcadeDB (embedded, fp32) and ArcadeDB (embedded, int8) left 1,000 vectors in their delta buffers, to be merged by a later full rebuild this cell does not pay for. A full rebuild touches every vector in the index and an incremental insert touches only the new ones, so the per-vector costs here price different amounts of work and are not a straight speed comparison.
Cold p50 and p99 are the first timed pass over the query set after the index is built; a short untimed warm-up on held-out queries runs before it, so cold means an index that has not yet answered the timed queries, not a process that has done nothing. Warm columns, where present, are the passes after it.
Every engine searches this corpus with the same distance, squared Euclidean, although the corpus is one whose published neighbours were computed by angle. The two agree here: every vector and every query is scaled to unit length before anything is indexed, and once they are, ordering by Euclidean distance and ordering by angle produce the same list. So the recall column is measured against the dataset's own published neighbours, unchanged, while every engine runs one metric rather than each running its own -- which is what makes the latencies comparable at all.
ArcadeDB's maxConnections is a Vamana per-layer degree, not hnswlib's M. Matching the parameter names would compare a half-degree graph against a full-degree one, so the graphs are matched by effect instead.
ArangoDB's vector index is FAISS IVF (inverted lists over trained centroids), not HNSW, so the degree match above does not apply to it, and its rows record its operating point instead. Its number of lists is four times the square root of the vector count, the low end of the range FAISS's own guidelines give, which makes 4,000 lists at 1M vectors. The number of lists each query probes is calibrated, not set: after the build and before any timed pass, the benchmark picks the smallest number whose recall@10 on 200 queries held out from the timed ones reaches the median recall@10 of ArcadeDB (embedded, fp32) at the same size in the previous published measurement. At 1M vectors that target is 0.9886 and the calibration chose 90 to 119 lists across five builds.
Milvus's dense rows run with segments sealed at 50% of the maximum segment size, where the image default is 12%, so a large ingest lands in the few-large-segments layout that Milvus's own compaction otherwise reaches at an unpredictable moment. One line changed from the image's configuration; sparse rows are at the default.
ArcadeDB (embedded, fp32), ArcadeDB (embedded, int8), ArcadeDB (server, fp32), and ArcadeDB (server, int8) run on a JVM, and for them this column is close to the heap they were given rather than the memory the work needed: ArcadeDB (server, fp32)'s 28.50 GiB here is about 119 per cent of its 24g heap. Every engine on a table gets the same memory envelope; how much of it a JVM takes is a property of the runtime, so read these cells against each other rather than against an engine that manages its own memory.
Each pass answers 1,000 queries; cold is the first pass after the build and warm pools the four passes after it, over five builds.
`ingest+index total s` is the whole of building this corpus. `ingest s` and `index s` are the two phases timed inside it -- writing the data, then building whatever index the engine was given -- and they are not a division of it: each engine's ingest clock starts after the setup it does first, creating a schema or a collection, which neither phase counts. How much is left over depends on the engine: nothing on some of the ones here, up to about 1 per cent on others. Read the two as what each phase cost, and the total as what the whole of it cost.
SurrealDB (embedded) at 1M vectors: the search cell exceeded its 2 hour budget, the same budget every engine on this table had, on its first attempt and was not retried; there is no row. It was still in the index build when the budget ran out.
SurrealDB (embedded) at 9.99M vectors: the search cell exceeded its 8 hour budget, the same budget every engine on this table had, on its first attempt and was not retried; there is no row. It was still in the index build when the budget ran out.
SurrealDB (server) at 9.99M vectors: the search cell reached the 36g memory envelope (peak 36,864 MiB) and was killed by the kernel, the same envelope every engine on this table had; it did not run out of time, and there is no row.
ArangoDB at 9.99M vectors is not comparable. On the release measured here, 3.12.11, an IVF index with 10,000 or more lists answers every query from one fixed list. The index at this size has 12,643 lists, so the search returned none of the true neighbours, and the cell is not a measurement of ArangoDB's search. The next measurement runs ArangoDB at its latest stable release.
sqlite-vec (int8) at 1M vectors is withheld: its search answered with a recall@10 of 0.0229 across 5 repetition(s), which is not a measurement of search but of a broken index, so its latency is not printed beside engines answering correctly. The cause is investigated on the benchmark machine before anything is claimed about it.
In the cells: `>2h` and `>8h` mean the cell ran past its tier's cap and was not retried; each tier has its own cap, the same for every engine on it. `OOM` means killed at the cell's memory envelope rather than running out of time. `n/c` means not comparable, because a defect in the engine's release decided the cell's result; none of its numbers is printed, and the note names the defect. `re-run` means the cell was measured before a fix to our harness and stays off the table until it is measured again; the note says what was wrong. A dash is an operation the engine cannot express, never a slow one.
Ingest paths: ArcadeDB embedded issues INSERT per vector in 10,000-row transactions through the Python package, then CREATE INDEX ... LSM_VECTOR; served sends 500-statement sqlscript batches over HTTP with each vector spelled out as text, then the same CREATE INDEX; Chroma add() in batches of 5,000; LanceDB an Arrow table then create_index; Qdrant and Milvus upsert in batches; DuckDB VSS and sqlite-vec executemany; pgvector COPY FROM STDIN then CREATE INDEX; Neo4j loads then builds its vector index; MongoDB insert_many then its vector search index; ArangoDB import_bulk then its FAISS IVF index; SurrealDB inserts through its Python SDK under the HNSW index it defines first; the server builds that index in the background, so the served build time includes waiting until a query probe shows the index has caught up.

Every vectors table, with the rest of the tasks in this lane.

Lane

Time series.

The task is the Time Series Benchmark Suite on its CPU data set. The queries are the ones a monitoring stack asks: the newest reading per sensor, a twelve-hour aggregate, a double group-by over host and hour, a high-usage filter, and a grouped, ordered, limited query of the shape a dashboard issues.

Still being measured at this version. This table appears here when its run finishes.

Every time series table, with the rest of the tasks in this lane.

Lane

Cross-model.

Most lanes compare engines that do one thing. This lane asks what a single engine buys when one operation touches more than one data model. The comparators are other engines that hold several models and, beside them, a composed stack: separate stores with application code between them.

The composed write, the retrieval path, and the graph-filtered vector search, with recall for the two read paths. The write is a transaction on the engines that have one and a sequence of calls on the stack that does not.

Cross-model transaction Vector hit to graph traversal to document update, in one transaction
Engine	Mode	Size	transaction p50 ms ↓	transaction p99 ms ↓	retrieval p50 ms ↓	graph-filtered search p50 ms ↓	retrieval recall@10	filtered recall@10	ingest+index vertices+edges/s ↑	ingest+index total s ↓	peak memory GiB ↓	disk GiB ↓
ArcadeDB (one transaction)	embedded	50k products	1.71	3.19	2.62	6.81	0.77	0.069	21k	9.50	2.41	0.038
ArcadeDB (one transaction)	server	50k products	7.52	9.87	5.29	9.74	0.79	0.069	15k	13.6	4.09	0.098
ArangoDB	server	50k products	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run
MongoDB	server	50k products	5.68	9.07	7.62	6.70	0.84	1	6,993	28.6	1.18	0.070
Neo4j (vector index)	server	50k products	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run
PostgreSQL + pgvector + AGE	server	50k products	4.93	7.32	2.41	1.25	0.78	1	24k	8.23	0.12	0.16
Qdrant + Neo4j (no shared transaction)	server	50k products	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run
SurrealDB	embedded	50k products	7.88	8.87	9.30	re-run	0.70	1	620	322.7	0.84	0.24
SurrealDB	server	50k products	re-run	re-run	5.17	re-run	0.73	1	16k	12.2	0.90	0.47
ArcadeDB (one transaction)	embedded	500k products	2.15	4.41	6.46	9.33	0.42	0.013	17k	117.4	14.1	0.39
ArcadeDB (one transaction)	server	500k products	7.92	10.3	8.47	11.9	0.42	0.012	11k	190.2	11.6	0.42
ArangoDB	server	500k products	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run
MongoDB	server	500k products	6.14	7.75	8.70	7.09	0.69	1	35k	57.6	3.92	0.85
Neo4j (vector index)	server	500k products	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run
PostgreSQL + pgvector + AGE	server	500k products	13.1	15.2	6.11	1.37	0.41	1	18k	113.6	0.67	1.51
Qdrant + Neo4j (no shared transaction)	server	500k products	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run	re-run
SurrealDB	embedded	500k products	>2h	>2h	>2h	>2h	>2h	>2h	>2h	>2h	>2h	>2h
SurrealDB	server	500k products	re-run	re-run	re-run	re-run	re-run	1	15k	137.4	6.49	0.69

↑ higher is better, ↓ lower is better; the best value in each column, within one size, is bold

Measured rows: benchmarks/experiments/results/runs_paper_oct.csv

Exact builds measured

arcadedb 26.10.1-dev · 417314c18 (in-process)
arcadedb 26.10.1-dev · 417314c18 arcadedb-c25:417314c18
ArangoDB arangodb@563cb2c07af0…
mongodb 8.2.12 + MongoDb Search Community Version 1.70.4 dbbench:mongo-search
Neo4j (vector index), Qdrant + Neo4j (no shared transaction) neo4j@e702d6b535d9…
PostgreSQL 18.6 + pgvector 0.8.6 + age 1.8.0 dbbench:pg-age
surrealdb 2.3.10
surrealdb 3.2.4 surrealdb/surrealdb@6a5002363ff5…

The engines do not all run the same SHAPE of filtered search, and it shows in the recall column as much as the time one. ArangoDB, MongoDB, Neo4j (vector index), PostgreSQL + pgvector + AGE, Qdrant + Neo4j (no shared transaction), SurrealDB (embedded), and SurrealDB (server) restrict the candidate set first and rank what is left by distance, so they return that set's true nearest matches. ArcadeDB (one transaction) and ArcadeDB (server, one transaction) instead take an over-large global top-k and keep whichever candidates happen to fall inside it: more work, and a partial answer, because a global list of that size usually does not contain enough of the candidates. Read both columns for those arms as what this query shape costs rather than as what the engine can do.
The transaction is the update, and every engine here commits it all or nothing. ArcadeDB (one transaction), ArcadeDB (server, one transaction), Neo4j (vector index), and PostgreSQL + pgvector + AGE also run the vector search and the hop inside it; ArangoDB, MongoDB, SurrealDB (embedded), and SurrealDB (server) run those two reads first and wrap only the update (MongoDB cannot do otherwise: it refuses a vector search inside a multi-document transaction, and its rows record the refusal). One client runs at a time, so the answers are the same either way. Qdrant + Neo4j (no shared transaction) has no transaction spanning its two systems, which is what that row exists to show.
Disk is what the workload left on disk, in GiB: the engine's writable layer plus its volumes after the cell, minus the same engine's empty footprint.
ArangoDB's row is marked `re-run`: its rows on this table were measured before the index on the product id that the protocol builds for it existed, so its reads scanned every product. They are withheld until it is measured with the index.
SurrealDB's graph-filtered search is marked `re-run`: our query scanned every product instead of reading the candidates by record id, the access path every other engine on this table was given. It is being re-measured with the fix.
SurrealDB (server) retrieval at the 500k tier is marked `re-run`: the server was still building its vector index in the background when these queries ran. It is being re-measured with the build waiting for the index to catch up.
Qdrant + Neo4j's row is marked `re-run`. Its vector half ran in the Qdrant client's in-memory local mode, a pure-Python reimplementation rather than the Qdrant server, so every time, recall, and disk value it produced described that reimplementation. It is being re-run against the Qdrant server the vector table uses. Its all-or-nothing result on the atomicity table below does not depend on this.
Neo4j's row is marked `re-run`: its vector index ran at Neo4j's default quantization, binary with a three-fold search expansion, which our index definition did not override, so its vector search was not the unquantized search the other engines ran. The next measurement sets the quantization explicitly. Its all-or-nothing result on the atomicity table does not depend on the index.
SurrealDB (server)'s transaction cells are marked `re-run`: the server ran at its default, a sync to disk at every commit, although it has a setting that does not wait, and every other engine on this table that has such a setting ran at it. Our harness missed that setting, and the next measurement runs the transaction at it.
There is no cold column on this table, and the reason is recorded on the rows themselves: no cold/warm split: each operation runs against an already-built, already-warm database by construction
Durability is matched at the relaxed end: on every engine that has the setting, a commit returns without waiting for the disk and the log is flushed by the engine's own background policy, which is ArcadeDB's engine default and a documented production mode for each of the others. Each row records what it ran as `durability`, and the setting was read out of the engine rather than assumed on every engine except ArcadeDB (server, one transaction) and SurrealDB (server). ArcadeDB (server, one transaction) is given the setting when the server starts, and the row records it, because the server offers no way to read it back. SurrealDB (server)'s rows record that no setting was found, and its server log shows what it ran: its default in both runs, a sync to disk at every commit (the log reads "Sync mode: every transaction commit" in every run), although it has a setting that does not wait; the next measurement runs both. ArcadeDB (one transaction) loaded its catalog with the write-ahead log off, which the bulk-load helper it used does by default, so its ingest figures had less durability than the relaxed setting; its timed operations ran with the log on. The next measurement loads with the log on.
Atomic means all or nothing: the whole update happens, or none of it does, with no state in between that anyone can observe. One engine can promise that across a vector, a graph edge, and a document because they share a transaction. Qdrant and Neo4j cannot promise it to each other, because nothing spans the two.
So the interesting result here is not the speed. It is what a crash halfway through leaves behind, and the atomicity table below shows it: for each engine, how many interrupted writes left the store holding some but not all of the write after it was reopened. That is what this comparison exists to show.
SurrealDB embedded runs on the SDK's SurrealKV store on disk and SurrealDB server on RocksDB, and each has its own disk reading.
ArcadeDB (one transaction) and ArcadeDB (server, one transaction) run on a JVM, and for them this column is close to the heap they were given rather than the memory the work needed: ArcadeDB (server, one transaction)'s 4.09 GiB here is about 68 per cent of its 6g heap. Every engine on a table gets the same memory envelope; how much of it a JVM takes is a property of the runtime, so read these cells against each other rather than against an engine that manages its own memory.
Each repetition runs the transaction 300 times; the p50 and p99 are over those.
Indexes on this table are matched by effect rather than by rule: ArcadeDB (one transaction), ArcadeDB (server, one transaction), MongoDB, PostgreSQL + pgvector + AGE, SurrealDB (embedded), and SurrealDB (server) build one. Where building one was a choice, the engine was timed with it and without it and the faster configuration kept; where an engine reaches these queries through its own primary key or record id, that is its equivalent. This table has no index column: the time an index took to build is inside the ingest+index total.
SurrealDB (embedded) at 500k products: the transaction cell exceeded its 2 hour budget, the same budget every engine on this table had, on its first attempt and was not retried; there is no row. It was still in the index build when the budget ran out.
In the cells: `>2h` means the cell ran past its tier's cap and was not retried. `re-run` means the cell was measured before a fix to our harness and stays off the table until it is measured again; the note says what was wrong. A dash is an operation the engine cannot express, never a slow one.
ingest+index total s is one timer around loading the vertices and edges and creating the vector index. Ingest paths: ArcadeDB embedded loads with the Python package's graph_batch (5,000 records per commit, vertices then edges) and then CREATE INDEX ... LSM_VECTOR; served sends CREATE VERTEX and CREATE EDGE batches as sqlscript over HTTP, then the same CREATE INDEX; SurrealDB embedded inserts through its Python SDK into the SDK's SurrealKV store on disk, and SurrealDB server through the same SDK over WebSocket onto RocksDB; Neo4j and the composed stack load the graph with UNWIND over bolt, and the composed stack upserts its vectors into Qdrant; PostgreSQL + pgvector + AGE loads the products with COPY under a pgvector HNSW index and creates the graph with UNWIND inside Cypher; ArangoDB import_bulk; MongoDB insert_many.

Every cross-model table, with the rest of the tasks in this lane.

Lane

Lifecycle.

These are the costs that nobody benchmarks and everybody pays. A session table opens a database, does one thing and closes it, for each kind of data the engine can hold, embedded and served. It is the cold measurement in its purest form.

A session for each kind of data, embedded and served. The embedded rows also carry what starting the process costs before the first database call, because an open measured inside an already-running process is not what someone launching a script pays.

Session cost, open to close synthetic, one structure per row
Engine	Mode	Size	JVM start ms ↓	first open ms ↓	cold process ms ↓	open and close ms ↓	one query ms ↓	one write ms ↓	write, then query ms ↓	peak memory GiB ↓
Dense vectors	embedded	10k	182.1	222.7	500.2	14.3	26.7	24.4	33.3	1.12
Dense vectors	server	10k	—	—	—	16.3	34.3	27.8	38.5	1.37
Documents	embedded	10k	177.8	185.5	483.3	12.3	11.1	20.8	22.0	0.36
Documents	server	10k	—	—	—	14.8	15.1	24.0	28.6	0.40
Documents, ten indexes	embedded	10k	179.8	195.6	471.0	18.7	17.5	48.7	48.9	0.66
Documents, ten indexes	server	10k	—	—	—	20.4	23.5	53.2	56.4	0.88
Empty database	embedded	10k	179.1	184.3	455.1	11.3	11.6	20.3	21.1	0.26
Empty database	server	10k	—	—	—	13.1	14.9	22.4	23.8	0.30
Graph	embedded	10k	177.8	186.2	475.9	13.7	15.2	21.1	23.3	2.11
Graph	server	10k	—	—	—	15.2	18.6	30.2	26.7	2.01
Sparse vectors	embedded	10k	183.7	195.1	468.5	12.9	11.9	32.5	29.8	0.98
Sparse vectors	server	10k	—	—	—	14.9	15.7	33.6	33.7	1.15
Time series	embedded	10k	180.2	198.4	472.9	18.2	17.0	27.9	28.7	2.08
Time series	server	10k	—	—	—	21.2	21.0	29.3	32.9	2.01
Dense vectors (SurrealDB)	embedded	10k	—	79.1	207.3	54.7	59.4	53.6	61.2	re-run
Documents (SurrealDB)	embedded	10k	—	29.8	158.8	8.47	11.8	9.75	14.0	re-run
Documents, ten indexes (SurrealDB)	embedded	10k	—	129.7	259.5	114.0	116.4	116.4	117.7	re-run
Empty database (SurrealDB)	embedded	10k	—	22.9	150.7	2.60	2.55	4.08	3.96	re-run
Graph (SurrealDB)	embedded	10k	—	255.7	384.3	205.9	218.1	279.1	213.8	re-run
Time series (SurrealDB)	embedded	10k	—	34.1	163.0	9.82	13.8	10.9	14.2	re-run
Dense vectors	embedded	100k	186.2	230.7	507.6	22.4	103.5	32.6	111.1	2.88
Dense vectors	server	100k	—	—	—	17.4	107.9	34.9	117.7	4.16
Documents	embedded	100k	179.9	180.6	449.4	11.9	11.4	19.5	22.2	0.88
Documents	server	100k	—	—	—	14.0	15.8	27.8	27.9	0.94
Documents, ten indexes	embedded	100k	177.8	193.4	465.4	18.7	17.9	50.1	49.1	2.09
Documents, ten indexes	server	100k	—	—	—	20.7	21.6	53.6	57.0	2.06
Empty database	embedded	100k	180.3	184.1	454.7	12.5	11.8	20.9	21.5	0.25
Empty database	server	100k	—	—	—	12.4	13.9	22.9	25.5	0.30
Graph	embedded	100k	180.3	181.7	453.5	13.8	15.0	24.2	22.0	2.76
Graph	server	100k	—	—	—	16.1	17.9	30.1	27.9	2.75
Sparse vectors	embedded	100k	179.4	195.7	464.5	12.6	12.2	29.8	27.5	2.74
Sparse vectors	server	100k	—	—	—	14.1	16.1	35.1	33.6	4.11
Time series	embedded	100k	181.6	197.1	472.9	17.9	16.8	24.9	29.2	2.70
Time series	server	100k	—	—	—	20.4	21.0	32.7	30.1	2.69
Dense vectors (SurrealDB)	embedded	100k	—	666.9	796.8	730.3	621.6	623.4	632.8	re-run
Documents (SurrealDB)	embedded	100k	—	106.4	237.5	83.0	110.0	82.9	110.7	re-run
Documents, ten indexes (SurrealDB)	embedded	100k	—	1,033	1,162	1,247	1,251	1,257	1,249	re-run
Empty database (SurrealDB)	embedded	100k	—	20.7	149.3	2.55	2.42	3.86	4.10	re-run
Graph (SurrealDB)	embedded	100k	—	2,544	2,676	3,156	3,169	3,160	3,166	re-run
Time series (SurrealDB)	embedded	100k	—	124.9	253.9	118.0	170.2	120.6	171.0	re-run
Dense vectors	embedded	1M	178.6	209.1	482.6	98.0	853.1	98.1	872.3	12.2
Dense vectors	server	1M	—	—	—	16.2	897.3	103.9	902.0	12.5
Documents	embedded	1M	178.2	171.8	436.8	12.1	11.5	22.4	22.7	4.19
Documents	server	1M	—	—	—	14.0	13.8	21.5	30.1	4.22
Documents, ten indexes	embedded	1M	178.8	194.9	484.4	30.3	30.3	64.5	64.4	10.4
Documents, ten indexes	server	1M	—	—	—	35.4	37.3	68.3	69.1	10.3
Empty database	embedded	1M	186.9	175.2	447.4	12.1	12.1	20.7	21.0	0.47
Empty database	server	1M	—	—	—	13.1	14.5	24.5	24.7	0.52
Graph	embedded	1M	179.9	172.1	441.5	12.7	14.8	23.2	22.8	10.5
Graph	server	1M	—	—	—	15.6	17.0	27.8	28.2	10.5
Sparse vectors	embedded	1M	180.3	182.7	451.0	11.8	12.1	29.7	28.6	10.5
Sparse vectors	server	1M	—	—	—	13.3	14.8	30.1	28.9	10.5
Time series	embedded	1M	180.9	190.7	469.6	18.6	17.0	29.6	27.0	10.1
Time series	server	1M	—	—	—	21.4	21.0	33.0	27.7	10.2
Dense vectors (SurrealDB)	embedded	1M	>8h	>8h	>8h	>8h	>8h	>8h	>8h	—
Dense vectors	embedded	10M	179.3	206.9	480.4	1,412	18k	1,420	18k	24.7
Dense vectors	server	10M	—	—	—	15.2	18k	1,489	18k	24.7
Documents	embedded	10M	178.8	171.9	437.0	11.0	10.7	21.1	21.4	15.3
Documents	server	10M	—	—	—	13.9	13.8	27.2	28.1	15.3
Documents, ten indexes	embedded	10M	180.1	201.9	494.2	24.7	26.9	64.6	63.5	22.7
Documents, ten indexes	server	10M	—	—	—	28.4	32.4	65.8	62.9	22.1
Empty database	embedded	10M	179.4	174.1	441.7	11.3	11.4	21.4	20.8	0.60
Empty database	server	10M	—	—	—	14.1	14.8	26.7	24.9	0.68
Graph	embedded	10M	180.1	172.4	441.5	12.7	14.5	20.8	23.0	19.1
Graph	server	10M	—	—	—	14.9	16.8	28.9	24.0	19.1
Sparse vectors	embedded	10M	179.5	179.8	449.2	11.9	12.0	29.8	28.3	19.1
Sparse vectors	server	10M	—	—	—	13.0	14.8	33.8	27.5	19.1
Time series	embedded	10M	180.2	193.3	472.1	18.6	17.4	28.0	28.1	15.2
Time series	server	10M	—	—	—	21.0	21.0	26.5	31.2	15.3
Documents, ten indexes (SurrealDB)	embedded	10M	OOM	OOM	OOM	OOM	OOM	OOM	OOM	—
Graph (SurrealDB)	embedded	10M	OOM	OOM	OOM	OOM	OOM	OOM	OOM	—

↑ higher is better, ↓ lower is better; the best value in each column, within one size, is bold

Measured rows: benchmarks/experiments/results/runs_paper_oct.csv

The SESSION is open + action + close. Reporting open and close alone hides work triggered between them.
Cold start is measured in a fresh subprocess and reported beside every session number, because a millisecond open inside a process that takes half a second to reach its first database call is not a millisecond to whoever launched it.
A clean close should be O(what was written), not O(what is stored): write nothing and closing should cost the same at 10k documents and 10M.
Server rows have no JVM start, first open, or cold process: the server is already running when the probe connects, so those three columns describe the embedded process only. The session columns are measured for both.
Graph with the analytical view is withheld: its query grew from a bounded set of seeds to an unbounded 2-hop, so it stays off this table until the next measurement bounds the query again.
SurrealDB's peak memory on this table is withheld, and its 10M dense-vector cell is not shown: our build fed SurrealDB from one in-memory Python list while the ArcadeDB arm streamed, so its memory reading was partly ours, and the 10M vector build was killed for our allocation, not SurrealDB's. Its session timings ran after the build and stand. Both are re-measured in the next measurement, with the input streamed.
An engine with no row for a situation cannot build it, and its adapter says so rather than leaving the row blank; the declared row keeps the statement the adapter tried and the engine's own answer.
SurrealDB (embedded) has no row for the graph with the analytical view (no graph analytical view) or sparse vectors (no sparse-vector index type).
The dense-vector rows' one-query and write-then-query sessions, embedded and server alike, carry a defect in the engine build measured here, ArcadeDB issue #8852 (https://github.com/ArcadeData/arcadedb/issues/8852): the first vector search after a database is reopened re-reads every vector's record on one thread, so from a million vectors up those sessions are mostly that re-read, and ArcadeDB 26.10.1 fixes it.
The rows each session reads back are hashed, and for documents, documents with ten indexes, the graph, and time series every embedded engine that builds the situation must return the same hash as ArcadeDB embedded, or nothing is published. Only ArcadeDB (embedded) builds sparse vectors, so there is no second answer to compare. Not compared: the empty database, which reads nothing back; and dense vectors, whose read goes through an approximate index and is checked by recall on the dense vector table instead. The server rows are compared only with other server rows, because a server session runs a different set of modes and its record counts differ by construction; ArcadeDB (server) is the only engine served here, so they are compared with nothing.
ArcadeDB's embedded dense-vector rows read the index's statistics just before close, to record whether a rebuild ran. The read is outside the timers, but on this engine build it raises the cost of the close that follows it, so that row's open and close figure is an upper bound for a session that touches nothing.
SurrealDB rows have no JVM start: the SurrealDB core is a compiled extension that the Python import loads, so there is no runtime to start apart from the import. Their first open is the first open with the SDK already imported, and their cold process is interpreter start, import, open, and close, the same span the ArcadeDB embedded rows time. SurrealDB's time-series row is a plain table of timestamped records, the footing its time-series rows elsewhere on this page run on, because the engine has no time-series type.
SurrealDB (embedded) at 1M: the cell for dense vectors exceeded its 8 hour budget, the same budget every engine on this table had, on its first attempt and was not retried; there is no row.
SurrealDB (embedded) at 10M: the cell for documents with ten indexes reached the 28g memory envelope (peak 28,672 MiB) and was killed by the kernel, the same envelope every engine on this table had; it did not run out of time, and there is no row.
SurrealDB (embedded) at 10M: the cell for the graph reached the 28g memory envelope (peak 28,672 MiB) and was killed by the kernel, the same envelope every engine on this table had; it did not run out of time, and there is no row.
In the cells: `>8h` means the cell ran past its tier's cap and was not retried. `OOM` means killed at the cell's memory envelope rather than running out of time. `re-run` means the cell was measured before a fix to our harness and stays off the table until it is measured again; the note says what was wrong. A dash is an operation the engine cannot express, never a slow one.

Every lifecycle table, with the rest of the tasks in this lane.

Corrections

Tell us a setting is wrong.

If a setting we use for your engine differs from what you recommend, open an issue. Say which engine and version, what we use, what you recommend, and where your documentation says so.

We re-run the affected cells with your configuration on the same machine, and publish them beside the earlier result with the reason. The earlier result stays in the history.

Open an issue on GitHub. The contribution guide also covers adding an engine, a query or a lane, and reporting a problem with a result.

Independence

Who runs DBBench.

DBBench is run by HumemAI. HumemAI also builds the ArcadeDB Python package. It is measured by the same rules as every other engine: the same tasks, the same machine, published settings, and a result for every failure.

The code and the settings are public, so anyone can check a result or reproduce it. When a result is wrong, the fix is a new row, not an edit.

Resources

Rules and code.

DocumentationOpen GitHubOpen Report a wrong settingOpen