Lane
Tables
Documents and structured query language (SQL) analytics.
Results
Tasks and results.
Most people arrive with tables, so this lane comes first. The tasks are TPC-C and TPC-H from the Transaction Processing Performance Council, the standard transaction and analytics workloads. TPC-C is many small reads and writes. TPC-H is a few large scanning queries.
Each engine answers the same question in its own query language: SQL where it has one, and its native language where it does not. The answers are hashed and compared before any time is published.
Online transaction processing: TPC-C's new-order and payment transactions on the TPC-H tables, and the single-record insert, read, update, and delete beside them. Each column is a warm median; the ninety-ninth percentile is printed for new-order, the transaction this table leads with. There is no cold column of its own because every operation runs against an already-open, already-warm database by construction.
| Engine | Mode | Size | new-order p50 ms ↓ | new-order p99 ms ↓ | payment p50 ms ↓ | insert p50 ms ↓ | read p50 ms ↓ | update p50 ms ↓ | delete p50 ms ↓ | OLTP ops/s ↑ | ingest documents/s ↑ | ingest total s ↓ | ingest s ↓ | index s ↓ | peak memory GiB ↓ | disk GiB ↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ArcadeDB | embedded | TPC-H SF1 (6.0M line items) | 0.46 | 0.90 | 0.20 | 0.10 | 0.030 | 0.086 | 0.19 | 2,586 | 47k | 132.5 | 126.3 | 6.11 | 6.37 | 0.67 |
| ArcadeDB | server | TPC-H SF1 (6.0M line items) | 1.21 | 2.38 | 0.89 | 0.69 | 0.60 | 0.64 | 0.78 | 888 | 35k | 178.7 | 172.6 | 6.21 | 6.42 | 0.67 |
| ArangoDB | server | TPC-H SF1 (6.0M line items) | 2.55 | 3.75 | 2.57 | 0.50 | 0.46 | 0.53 | 0.51 | 384 | 91k | 67.8 | 67.8 | 0 | 1.22 | 1.04 |
| DuckDB | embedded | TPC-H SF1 (6.0M line items) | 9.89 | 21.9 | 10.5 | 9 | 0.16 | 8.89 | 8.94 | 94 | 512k | 12.1 | 12.1 | 0.010 | 0.58 | 0.073 |
| MongoDB | server | TPC-H SF1 (6.0M line items) | 0.48 | 0.83 | 0.45 | 0.080 | 0.092 | 0.11 | 0.11 | 2,001 | 98k | 63.1 | 54.9 | 7.78 | 4.50 | 0.88 |
| PostgreSQL | server | TPC-H SF1 (6.0M line items) | 0.084 | 0.13 | 0.080 | 0.043 | 0.040 | 0.046 | 0.039 | 12k | 222k | 28.0 | 26.5 | 1.42 | 0.52 | 0.93 |
| SQLite | embedded | TPC-H SF1 (6.0M line items) | 0.012 | 0.027 | 0.010 | 0.0060 | 0.0020 | 0.0060 | 0.0050 | 21k | 225k | 27.5 | 27.5 | 0 | 0.42 | 0.31 |
| SurrealDB | embedded | TPC-H SF1 (6.0M line items) | 0.16 | 0.20 | 0.14 | 0.080 | 0.049 | 0.077 | 0.12 | 6,518 | 10k | 599.5 | 599.5 | 0 | 6.89 | 2.62 |
| SurrealDB | server | TPC-H SF1 (6.0M line items) | re-run | re-run | re-run | re-run | 0.13 | re-run | re-run | re-run | 20k | 305.6 | 305.4 | 0.14 | 6.14 | 0.48 |
| ArcadeDB | embedded | TPC-H SF10 (60.0M line items) | 0.45 | 0.92 | 0.18 | 0.096 | 0.032 | 0.083 | 0.18 | 2,734 | 48k | 1,288 | 1,227 | 59.8 | 18.0 | 6.84 |
| ArcadeDB | server | TPC-H SF10 (60.0M line items) | 1.32 | 2.44 | 0.85 | 0.73 | 0.62 | 0.64 | 0.82 | 873 | 37k | 1,685 | 1,623 | 61.1 | 18.1 | 6.88 |
| ArangoDB | server | TPC-H SF10 (60.0M line items) | 2.88 | 4.18 | 2.63 | 0.51 | 0.47 | 0.54 | 0.51 | 351 | 89k | 699.7 | 699.7 | 0 | 6.64 | 4.74 |
| DuckDB | embedded | TPC-H SF10 (60.0M line items) | 11.4 | 20.9 | 10.5 | 9.14 | 0.16 | 8.95 | 8.97 | 88 | 502k | 123.4 | 123.4 | 0.010 | 2.66 | 0.84 |
| MongoDB | server | TPC-H SF10 (60.0M line items) | 0.59 | 0.94 | 0.45 | 0.081 | 0.093 | 0.12 | 0.11 | 1,869 | 101k | 614.1 | 547.6 | 66.6 | 16.6 | 8.87 |
| PostgreSQL | server | TPC-H SF10 (60.0M line items) | 0.090 | 0.16 | 0.085 | 0.042 | 0.040 | 0.044 | 0.039 | 11k | 224k | 277.0 | 265.1 | 11.9 | 1.54 | 6.23 |
| SQLite | embedded | TPC-H SF10 (60.0M line items) | 0.012 | 0.019 | 0.010 | 0.0060 | 0.0020 | 0.0050 | 0.0050 | 33k | 229k | 271.1 | 271.1 | 0 | 1.65 | 3.21 |
| SurrealDB | embedded | TPC-H SF10 (60.0M line items) | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM |
| SurrealDB | server | TPC-H SF10 (60.0M line items) | re-run | re-run | re-run | re-run | 0.14 | re-run | re-run | re-run | 19k | 3,295 | 3,294 | 0.14 | 30.9 | 4.42 |
↑ higher is better, ↓ lower is better; the best value in each column, within one size, is bold
Measured rows: benchmarks/experiments/results/runs_paper_oct.csv
Exact builds measured
- arcadedb 26.10.1-dev · 417314c18 (in-process)
- arcadedb 26.10.1-dev · 417314c18
arcadedb-c25:417314c18 - arangodb 3.12.11
arangodb@563cb2c07af0… - duckdb 1.5.4
- mongodb 8.2.12
mongo@41afd6e1183f… - postgresql 18.6
postgres@7341002d2b8c… - sqlite 3.46.1
- surrealdb 2.3.10
- surrealdb 3.2.4
surrealdb/surrealdb@6a5002363ff5…
- Six transactional operations. New-order: TPC-C's checkout transaction: it reads a customer and a warehouse, inserts an order with its line items, and updates stock, all in one transaction. Payment: TPC-C's payment against one of the orders new-order just placed, chosen at random: the order is read, marked paid, and the payment inserted, in one transaction. Insert: one record inserted by key. Read: one record read back by key. Update: one record's quantity updated by key. Delete: one record deleted by key. OLTP ops/s is the rate of the two TPC-C transactions together; the four single-record operations are timed one at a time.
- Disk is what the workload left on disk, in GiB: the engine's writable layer plus its volumes after the cell, minus the same engine's empty footprint.
- SurrealDB (server)'s write cells are marked `re-run`: the server ran at its default, a sync to disk at every commit, although it has a setting that does not wait, and every other engine on this table that has such a setting ran at it. Our harness missed that setting, and the next measurement runs its writes at it. Its read cells commit nothing and stay.
- The operations on this table are timed from the first one after the data is loaded, so its columns include each engine's warm-up. On ArcadeDB (embedded) and ArcadeDB (server), which run on a JVM, that warm-up includes the JVM compiling the statement path. The next measurement gives every engine an untimed warm-up on separate keys first, prints the warm median, and adds the first operation as the cold column.
- Durability is matched at the relaxed end: on every engine that has the setting, a commit returns without waiting for the disk and the log is flushed by the engine's own background policy, which is ArcadeDB's engine default and a documented production mode for each of the others. Each row records what it ran as `durability`, and the setting was read out of the engine rather than assumed on every engine except ArcadeDB (server) and SurrealDB (server). ArcadeDB (server) is given the setting when the server starts, and the row records it, because the server offers no way to read it back. The exception on this table is DuckDB, which has no setting to relax and waits for the disk at every commit; its write and transaction cells are paying for that. SurrealDB (server)'s rows record that no setting was found, and its server log shows what it ran: its default in both runs, a sync to disk at every commit (the log reads "Sync mode: every transaction commit" in every run), although it has a setting that does not wait; the next measurement runs both.
- PostgreSQL's memory cell is not comparable to the others: this column counts memory an engine holds in its own address space, and PostgreSQL holds its data in shared memory and the kernel's file cache instead. At TPC-H SF10 (60.0M line items), of the 1.54 GiB shown, 1.464 GiB is our Python client and 0.070 GiB is the database.
- ArcadeDB (embedded) and ArcadeDB (server) run on a JVM, and for them this column is close to the heap they were given rather than the memory the work needed: ArcadeDB (server)'s 6.42 GiB here is about 80 per cent of its 8g heap. Every engine on a table gets the same memory envelope; how much of it a JVM takes is a property of the runtime, so read these cells against each other rather than against an engine that manages its own memory.
- Each repetition runs 1,000 new-order transactions and 1,000 payments, then 1,000 each of the single-record insert, read, update, and delete; the p50 and p99 are over those, and OLTP ops/s is the rate of the two transactions together.
- `ingest total s` is the whole of building this corpus. `ingest s` and `index s` are the two phases timed inside it -- writing the data, then building whatever index the engine was given -- and they are not a division of it: each engine's ingest clock starts after the setup it does first, creating a schema or a collection, which neither phase counts. How much is left over depends on the engine: nothing on some of the ones here, up to about 1 per cent on others. Read the two as what each phase cost, and the total as what the whole of it cost.
- SurrealDB (embedded) at TPC-H SF10 (60.0M line items): the transaction cell reached the 32g memory envelope (peak 32,768 MiB) and was killed by the kernel, the same envelope every engine on this table had; it did not run out of time, and there is no row.
- In the cells: `OOM` means killed at the cell's memory envelope rather than running out of time. `re-run` means the cell was measured before a fix to our harness and stays off the table until it is measured again; the note says what was wrong. A dash is an operation the engine cannot express, never a slow one.
- Ingest paths: ArcadeDB embedded loads through the Python package's insert_many in 10,000-row batches, one JSON payload per batch; served sends INSERT statements as sqlscript batches over HTTP; PostgreSQL COPY FROM STDIN; DuckDB CREATE TABLE AS SELECT from in-memory frames; SQLite executemany; MongoDB insert_many; ArangoDB import_bulk; SurrealDB inserts through its Python SDK.
Online analytical processing: five analytical queries over the same documents. The cold column is the first query of a session on a database that has just been opened, and every other latency column is the warm median of the iterations that follow it.
| Engine | Mode | Size | Q1 p50 ms ↓ | Q1 p99 ms ↓ | Q6 p50 ms ↓ | top parts p50 ms ↓ | ship mode p50 ms ↓ | by month p50 ms ↓ | cold first query ms ↓ | ingest documents/s ↑ | ingest total s ↓ | ingest s ↓ | index s ↓ | peak memory GiB ↓ | disk GiB ↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ArcadeDB | embedded | TPC-H SF1 (6.0M line items) | 12k | 12k | 745.5 | 8,138 | 4,041 | 6,299 | 6,517 | 47k | 132.5 | 126.3 | 6.11 | 6.37 | 0.67 |
| ArcadeDB | server | TPC-H SF1 (6.0M line items) | 12k | 13k | 746.3 | 8,459 | 4,261 | 6,647 | 6,658 | 35k | 178.7 | 172.6 | 6.21 | 6.42 | 0.67 |
| ArangoDB | server | TPC-H SF1 (6.0M line items) | 1,756 | 1,924 | 1,298 | 1,979 | 733.3 | 1,109 | 1,189 | 91k | 67.8 | 67.8 | 0 | 1.22 | 1.04 |
| DuckDB | embedded | TPC-H SF1 (6.0M line items) | 16.7 | 17.6 | 6.66 | 52.4 | 5.63 | 43.3 | 12.3 | 512k | 12.1 | 12.1 | 0.010 | 0.58 | 0.073 |
| MongoDB | server | TPC-H SF1 (6.0M line items) | 10k | 10k | 1,209 | 3,377 | 1,310 | 2,179 | 4,987 | 98k | 63.1 | 54.9 | 7.78 | 4.50 | 0.88 |
| PostgreSQL | server | TPC-H SF1 (6.0M line items) | 298.4 | 327.7 | 94.0 | 594.6 | 204.4 | 397.2 | 163.9 | 222k | 28.0 | 26.5 | 1.42 | 0.52 | 0.93 |
| SQLite | embedded | TPC-H SF1 (6.0M line items) | 3,221 | 3,278 | 367.7 | 2,322 | 1,574 | 2,474 | 1,594 | 225k | 27.5 | 27.5 | 0 | 0.42 | 0.31 |
| SurrealDB | embedded | TPC-H SF1 (6.0M line items) | 105k | 105k | 11k | 77k | 44k | 77k | 62k | 10k | 599.5 | 599.5 | 0 | 6.89 | 2.62 |
| SurrealDB | server | TPC-H SF1 (6.0M line items) | 46k | 46k | 3,907 | 5,579 | 5,545 | 6,273 | 22k | 20k | 305.6 | 305.4 | 0.14 | 6.14 | 0.48 |
| ArcadeDB | embedded | TPC-H SF10 (60.0M line items) | 563k | 563k | 74k | 97k | 42k | 65k | 281k | 48k | 1,288 | 1,227 | 59.8 | 18.0 | 6.84 |
| ArcadeDB | server | TPC-H SF10 (60.0M line items) | 571k | 571k | 75k | 100k | 44k | 68k | 281k | 37k | 1,685 | 1,623 | 61.1 | 18.1 | 6.88 |
| ArangoDB | server | TPC-H SF10 (60.0M line items) | 18k | 18k | 13k | 25k | 7,522 | 11k | 12k | 89k | 699.7 | 699.7 | 0 | 6.64 | 4.74 |
| DuckDB | embedded | TPC-H SF10 (60.0M line items) | 140.3 | 145.2 | 54.3 | 550.4 | 25.4 | 364.2 | 74.6 | 502k | 123.4 | 123.4 | 0.010 | 2.66 | 0.84 |
| MongoDB | server | TPC-H SF10 (60.0M line items) | 304k | 304k | 40k | 2.17M | 13k | 22k | 150k | 101k | 614.1 | 547.6 | 66.6 | 16.6 | 8.87 |
| PostgreSQL | server | TPC-H SF10 (60.0M line items) | 2,311 | 2,919 | 666.4 | 6,616 | 1,850 | 3,301 | 4,785 | 224k | 277.0 | 265.1 | 11.9 | 1.54 | 6.23 |
| SQLite | embedded | TPC-H SF10 (60.0M line items) | 35k | 35k | 3,536 | 26k | 17k | 27k | 18k | 229k | 271.1 | 271.1 | 0 | 1.65 | 3.21 |
| SurrealDB | embedded | TPC-H SF10 (60.0M line items) | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM |
| SurrealDB | server | TPC-H SF10 (60.0M line items) | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM |
↑ higher is better, ↓ lower is better; the best value in each column, within one size, is bold
Measured rows: benchmarks/experiments/results/runs_paper_oct.csv
Exact builds measured
- arcadedb 26.10.1-dev · 417314c18 (in-process)
- arcadedb 26.10.1-dev · 417314c18
arcadedb-c25:417314c18 - arangodb 3.12.11
arangodb@563cb2c07af0… - duckdb 1.5.4
- mongodb 8.2.12
mongo@41afd6e1183f… - postgresql 18.6
postgres@7341002d2b8c… - sqlite 3.46.1
- surrealdb 2.3.10
- surrealdb 3.2.4
surrealdb/surrealdb@6a5002363ff5…
- Five analytical queries, each over the whole line-item table. Q1: TPC-H Q1, the pricing summary, with seven of its ten output columns. It groups and aggregates nearly every line item, so it measures a full scan. It leaves out the average price, the average discount, and the total charge with tax, which needs the tax column this measurement did not load. The next measurement runs the full Q1. Q6: TPC-H Q6, the forecasting revenue change, with a row count added: it sums one column under a narrow filter, so it measures how well an engine skips what it does not need. Top parts: the ten parts with the highest revenue, a group-by over every line item with a sort and a limit. Ship mode: how many line items went by each ship mode, a group-by over the whole table. By month: revenue by month of shipment, a group-by on a date expression. All times are milliseconds.
- Disk is what the workload left on disk, in GiB: the engine's writable layer plus its volumes after the cell, minus the same engine's empty footprint.
- The cold column is the very first query of a session, Q1, timed once on a database that has just been opened; every other latency column is the warm median of the iterations that follow it. It answers what the first query of a session costs against the hundredth, which is a different question from the steady-state one and is the one an application that opens a database per request actually asks.
- Durability is matched at the relaxed end: on every engine that has the setting, a commit returns without waiting for the disk and the log is flushed by the engine's own background policy, which is ArcadeDB's engine default and a documented production mode for each of the others. Each row records what it ran as `durability`, and the setting was read out of the engine rather than assumed on every engine except ArcadeDB (server) and SurrealDB (server). ArcadeDB (server) is given the setting when the server starts, and the row records it, because the server offers no way to read it back. The exception on this table is DuckDB, which has no setting to relax and waits for the disk at every commit; its write and transaction cells are paying for that. SurrealDB (server)'s rows record that no setting was found, and its server log shows what it ran: its default in both runs, a sync to disk at every commit (the log reads "Sync mode: every transaction commit" in every run), although it has a setting that does not wait; the next measurement runs both.
- PostgreSQL's memory cell is not comparable to the others: this column counts memory an engine holds in its own address space, and PostgreSQL holds its data in shared memory and the kernel's file cache instead. At TPC-H SF10 (60.0M line items), of the 1.54 GiB shown, 1.464 GiB is our Python client and 0.070 GiB is the database.
- ArcadeDB (embedded) and ArcadeDB (server) run on a JVM, and for them this column is close to the heap they were given rather than the memory the work needed: ArcadeDB (server)'s 6.42 GiB here is about 80 per cent of its 8g heap. Every engine on a table gets the same memory envelope; how much of it a JVM takes is a property of the runtime, so read these cells against each other rather than against an engine that manages its own memory.
- Each repetition runs every query 100 times; the p50 and p99 are over those runs.
- Indexes on this table are matched by effect rather than by rule: ArcadeDB (embedded), ArcadeDB (server), MongoDB, PostgreSQL, SurrealDB (embedded), and SurrealDB (server) build one; ArangoDB, DuckDB, and SQLite build none, because measuring it showed the index costs them or changes nothing. Where building one was a choice, the engine was timed with it and without it and the faster configuration kept; where an engine reaches these queries through its own primary key or record id, that is its equivalent. The time an index took to build is the index column. DuckDB's index cell is not zero because it times a different index: the order-key index of the table new orders are written into, which the transactional workload needs and which is built while that table is still empty.
- `ingest total s` is the whole of building this corpus. `ingest s` and `index s` are the two phases timed inside it -- writing the data, then building whatever index the engine was given -- and they are not a division of it: each engine's ingest clock starts after the setup it does first, creating a schema or a collection, which neither phase counts. How much is left over depends on the engine: nothing on some of the ones here, up to about 1 per cent on others. Read the two as what each phase cost, and the total as what the whole of it cost.
- SurrealDB (embedded) and SurrealDB (server) at TPC-H SF10 (60.0M line items): the analytics cell reached the 32g memory envelope (peak 32,768 MiB) and was killed by the kernel, the same envelope every engine on this table had; it did not run out of time, and there is no row.
- 30 query cells on this table stopped at their per-query budget, the same for every engine here; each one's p50 and p99 are over those iterations, the first of which is the cold pass, and every other query in those cells keeps its numbers.
- ArangoDB at TPC-H SF10 (60.0M line items): Q1 (49 of 100 iterations, reaching 867 to 871 s), Q6 (66 to 67, reaching 866 to 875 s), by month (76 to 78, reaching 864 to 874 s), and top parts (35 to 36, reaching 864 to 888 s).
- ArcadeDB (embedded) at TPC-H SF10 (60.0M line items): Q1 (1 of 100 iterations, reaching 562 to 569 s), Q6 (12, reaching 888 to 899 s), by month (13 to 14, reaching 866 to 924 s), ship mode (21, reaching 869 to 887 s), and top parts (9, reaching 868 to 890 s).
- ArcadeDB (server) at TPC-H SF10 (60.0M line items): Q1 (1 of 100 iterations, reaching 562 to 574 s), Q6 (12, reaching 882 to 906 s), by month (13, reaching 876 to 907 s), ship mode (19 to 21, reaching 864 to 904 s), and top parts (9, reaching 892 to 906 s).
- MongoDB at TPC-H SF10 (60.0M line items): Q1 (1 of 100 iterations, reaching 300 to 309 s), Q6 (22, reaching 873 to 882 s), by month (40 to 41, reaching 871 to 884 s), ship mode (67 to 69, reaching 866 to 876 s), and top parts (1, reaching 2148 to 2186 s).
- SQLite at TPC-H SF10 (60.0M line items): Q1 (25 of 100 iterations, reaching 878 to 879 s), by month (32, reaching 866 to 868 s), ship mode (50 to 51, reaching 865 to 881 s), and top parts (33 to 34, reaching 864 to 889 s).
- SurrealDB (embedded) at TPC-H SF1 (6.0M line items): Q1 (29 of 100 iterations, reaching 3017 to 3057 s), Q6 (25, reaching 276 to 278 s), by month (17 to 18, reaching 1304 to 1368 s), ship mode (30, reaching 1315 to 1334 s), and top parts (17, reaching 1301 to 1312 s).
- SurrealDB (server) at TPC-H SF1 (6.0M line items): Q1 (66 to 67 of 100 iterations, reaching 3012 to 3050 s) and Q6 (69 to 71, reaching 270 to 274 s).
- In the cells: `OOM` means killed at the cell's memory envelope rather than running out of time. A dash is an operation the engine cannot express, never a slow one.
- Ingest paths: ArcadeDB embedded loads through the Python package's insert_many in 10,000-row batches, one JSON payload per batch; served sends INSERT statements as sqlscript batches over HTTP; PostgreSQL COPY FROM STDIN; DuckDB CREATE TABLE AS SELECT from in-memory frames; SQLite executemany; MongoDB insert_many; ArangoDB import_bulk; SurrealDB inserts through its Python SDK.
The rules every result follows are in the methodology. The settings for each engine are on the engines page. If one is wrong, tell us.
DBBench
The other lanes.
- Graphs: Interactive and analytical queries on a social network graph.
- Vectors: Dense and sparse vector search, with recall beside every latency.
- Time series: Monitoring-style queries over timestamped readings.
- Cross-model: One operation across a vector, a graph edge, and a document.
- Lifecycle: Opening, closing, deployment boundaries, and what waiting for the disk costs.
Resources