Lane

Vectors

Dense and sparse vector search, with recall beside every latency.

Illustration for DBBench, an open benchmark of database engines

Results

Tasks and results.

Dense and sparse vectors are different problems with different engines behind them, so they get separate tables. Both report recall beside every latency. An approximate index can be made fast by returning worse answers, so a latency without the share of true neighbours it found is not a comparable number.

Recall is measured against an exact brute-force answer over the same corpus. For an approximate index, agreeing with the exact answer is a stronger check than agreeing with another engine.

Where an engine has the boundary, loading the data and building the index are timed separately. A search after inserting into a built index, and a search after deleting from one, are timed as well.

Dense search, and search after an insert and after a delete into the same built index, with recall beside each. Every row states what it stores, full precision or quantized, because a quantized index and a full-precision one are not the same measurement.

Dense vector search DEEP-10M (deep-image-96-angular) and SIFT-1M
EngineModePrecisionSizecold p50 ms ↓cold p99 ms ↓warm p50 ms ↓warm p99 ms ↓after insert p50 ms ↓after delete p50 ms ↓insert into index ms/vectorrecall@10 after insertdelete from index ms/vectorrecall@10 after deleterecall@10 ↑ingest+index vectors/s ↑ingest s ↓index s ↓ingest+index total s ↓peak memory GiB ↓disk GiB ↓
ArcadeDBembeddedint81M vectors1.362.260.751.120.950.780.0440.984.640.980.986,87416.2129.3145.57.790.93
ArcadeDBembeddedfp321M vectors1.964.200.741.120.910.780.0370.994.720.990.996,32016.0142.0158.27.630.80
ArcadeDBserverint81M vectors2.453.621.742.291.741.790.160.986.950.980.984,86887.4117.4205.48.370.94
ArcadeDBserverfp321M vectors2.945.541.782.231.861.810.100.996.930.990.994,62687.1128.8216.27.650.81
ArangoDBserverfp321M vectors5.857.555.677.525.625.630.0690.990.0560.990.9911k——92.71.830.98
Chromaembeddedfp321M vectors0.650.850.660.890.660.670.520.980.470.980.976,820——146.61.150.78
DuckDB VSSembeddedfp321M vectors1.773.421.772.062.032.0210.70.971.780.980.9817k——58.42.831.38
LanceDBembeddedint81M vectors1.552.811.582.3320.519.80.00400.970.0250.970.9735k0.8627.428.42.020.89
Milvusserverint81M vectors1.132.401.091.781.141.138.560.972.530.970.977,49713.6117.8133.42.933.13
Milvusserverfp321M vectors1.172.011.101.931.151.128.550.981.530.980.987,73813.6113.9129.24.244.14
MongoDBserverfp321M vectors6.3719.64.014.984.304.541.070.980.0230.980.9817k17.341.258.85.102.54
Neo4jserverfp321M vectorsre-runre-runre-runre-runre-runre-runre-runre-runre-runre-runre-runre-runre-runre-runre-runre-runre-run
pgvectorserverfp321M vectors1.822.241.812.211.811.741.080.990.0790.990.996,50621.8132.1153.70.0552.56
Qdrantserverint81M vectors0.931.170.931.160.930.920.0541.000.0131.001.0032k——31.41.232.44
Qdrantserverfp321M vectors1.091.451.111.441.101.080.0581.000.0131.001.0023k——43.10.920.69
sqlite-vecembeddedfp321M vectors124.9127.4124.8127.1125.5124.90.0381.000.131.001.00156k——6.390.0420.50
SurrealDBembedded—1M vectors>2h>2h>2h>2h>2h>2h>2h>2h>2h>2h>2h>2h>2h>2h>2h>2h>2h
SurrealDBserverfp321M vectors6.258.574.005.434.444.520.170.988.030.980.98551159.81,6541,81413.60.74
ArcadeDBembeddedint89.99M vectors2.775.550.871.37——————0.945,047143.01,8431,97928.38.57
ArcadeDBembeddedfp329.99M vectors18.9126.21.011.50——————0.954,664148.81,9912,14228.37.56
ArcadeDBserverint89.99M vectors3.777.832.002.45——————0.943,6221,0041,7552,75828.48.59
ArcadeDBserverfp329.99M vectors23.8133.52.132.68——————0.953,3871,0011,9612,94928.57.60
ArangoDBserverfp329.99M vectorsn/cn/cn/cn/cn/cn/cn/cn/cn/cn/cn/cn/cn/cn/cn/cn/cn/c
Chromaembeddedfp329.99M vectors0.700.910.710.89——————0.932,650——3,77012.76.59
DuckDB VSSembeddedfp329.99M vectors2.587.322.583.05——————0.9312k——83325.28.08
LanceDBembeddedint89.99M vectors1.922.371.942.39——————0.9434k6.59287.9295.011.07.11
Milvusserverint89.99M vectors1.491.821.461.77——————0.967,654193.21,1081,30513.625.2
Milvusserverfp329.99M vectors1.561.841.501.74——————0.988,056195.91,0431,24016.933.2
MongoDBserverfp329.99M vectors13.739.06.769.89——————0.9513k135.5642.0777.222.022.9
Neo4jserverfp329.99M vectorsre-runre-runre-runre-runre-runre-runre-runre-runre-runre-runre-runre-runre-runre-runre-runre-runre-run
pgvectorserverfp329.99M vectors2.843.782.523.33——————0.955,153272.61,6651,9393.7317.1
Qdrantserverint89.99M vectors1.061.491.081.39——————0.9824k——412.67.105.34
Qdrantserverfp329.99M vectors1.351.881.261.75——————0.9818k——546.56.024.42
sqlite-vecembeddedint89.99M vectors671.0680.6671.0678.6——————0.92174k——57.33.771.12
sqlite-vecembeddedfp329.99M vectors897.7915.7896.5912.9——————1169k——59.03.613.80
SurrealDBembedded—9.99M vectors>8h>8h>8h>8h>8h>8h>8h>8h>8h>8h>8h>8h>8h>8h>8h>8h>8h
SurrealDBserver—9.99M vectorsOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM

↑ higher is better, ↓ lower is better; the best value in each column, within one size, is bold

Measured rows: benchmarks/experiments/results/runs_paper_oct.csv, benchmarks/experiments/results/dense_mp5_<pin or 2681>

Exact builds measured
  • arcadedb 26.10.1-dev · 417314c18
  • arangodb 3.12.11
  • chroma 1.5.9
  • duckdb vss 1.5.4
  • lancedb 0.39.0
  • milvus 3.0.1
  • mongodb 8.2.12 + MongoDb Search Community Version 1.70.4
  • pgvector 0.8.6
  • qdrant 1.19.1
  • sqlite-vec 0.1.9
  • surrealdb 3.2.4
  • ingest s and index s are two timers where the engine has the boundary: inserting the vectors, then building the index. ArangoDB (fp32), Chroma (fp32), DuckDB VSS (fp32), Qdrant (fp32), Qdrant (int8), sqlite-vec (fp32), and sqlite-vec (int8) record one timer only, because the index is built while ingesting or the adapter times no boundary, so their ingest s and index s are blank. ingest+index total s is one timer around both, and ingest+index vectors/s divides the vector count by it.
  • Disk is what the workload left on disk, in GiB: the engine's writable layer plus its volumes after the cell, minus the same engine's empty footprint.
  • Neo4j's row is marked `re-run`: its vector index ran at Neo4j's default quantization, binary with a three-fold search expansion, which our index definition did not override, while the row recorded it as unquantized. The next measurement sets the quantization explicitly, unquantized here and as a separate scalar-quantized row.
  • The insert and delete into a built index, the search after each, and the recall after each are measured at 1M vectors only. At 9.99M vectors those columns are blank because the pass is not run there, not because an engine failed it. These engines do not all maintain an index the same way: ArcadeDB (embedded, fp32) and ArcadeDB (embedded, int8) left 1,000 vectors in their delta buffers, to be merged by a later full rebuild this cell does not pay for. A full rebuild touches every vector in the index and an incremental insert touches only the new ones, so the per-vector costs here price different amounts of work and are not a straight speed comparison.
  • Cold p50 and p99 are the first timed pass over the query set after the index is built; a short untimed warm-up on held-out queries runs before it, so cold means an index that has not yet answered the timed queries, not a process that has done nothing. Warm columns, where present, are the passes after it.
  • Every engine searches this corpus with the same distance, squared Euclidean, although the corpus is one whose published neighbours were computed by angle. The two agree here: every vector and every query is scaled to unit length before anything is indexed, and once they are, ordering by Euclidean distance and ordering by angle produce the same list. So the recall column is measured against the dataset's own published neighbours, unchanged, while every engine runs one metric rather than each running its own -- which is what makes the latencies comparable at all.
  • ArcadeDB's maxConnections is a Vamana per-layer degree, not hnswlib's M. Matching the parameter names would compare a half-degree graph against a full-degree one, so the graphs are matched by effect instead.
  • ArangoDB's vector index is FAISS IVF (inverted lists over trained centroids), not HNSW, so the degree match above does not apply to it, and its rows record its operating point instead. Its number of lists is four times the square root of the vector count, the low end of the range FAISS's own guidelines give, which makes 4,000 lists at 1M vectors. The number of lists each query probes is calibrated, not set: after the build and before any timed pass, the benchmark picks the smallest number whose recall@10 on 200 queries held out from the timed ones reaches the median recall@10 of ArcadeDB (embedded, fp32) at the same size in the previous published measurement. At 1M vectors that target is 0.9886 and the calibration chose 90 to 119 lists across five builds.
  • Milvus's dense rows run with segments sealed at 50% of the maximum segment size, where the image default is 12%, so a large ingest lands in the few-large-segments layout that Milvus's own compaction otherwise reaches at an unpredictable moment. One line changed from the image's configuration; sparse rows are at the default.
  • ArcadeDB (embedded, fp32), ArcadeDB (embedded, int8), ArcadeDB (server, fp32), and ArcadeDB (server, int8) run on a JVM, and for them this column is close to the heap they were given rather than the memory the work needed: ArcadeDB (server, fp32)'s 28.50 GiB here is about 119 per cent of its 24g heap. Every engine on a table gets the same memory envelope; how much of it a JVM takes is a property of the runtime, so read these cells against each other rather than against an engine that manages its own memory.
  • Each pass answers 1,000 queries; cold is the first pass after the build and warm pools the four passes after it, over five builds.
  • `ingest+index total s` is the whole of building this corpus. `ingest s` and `index s` are the two phases timed inside it -- writing the data, then building whatever index the engine was given -- and they are not a division of it: each engine's ingest clock starts after the setup it does first, creating a schema or a collection, which neither phase counts. How much is left over depends on the engine: nothing on some of the ones here, up to about 1 per cent on others. Read the two as what each phase cost, and the total as what the whole of it cost.
  • SurrealDB (embedded) at 1M vectors: the search cell exceeded its 2 hour budget, the same budget every engine on this table had, on its first attempt and was not retried; there is no row. It was still in the index build when the budget ran out.
  • SurrealDB (embedded) at 9.99M vectors: the search cell exceeded its 8 hour budget, the same budget every engine on this table had, on its first attempt and was not retried; there is no row. It was still in the index build when the budget ran out.
  • SurrealDB (server) at 9.99M vectors: the search cell reached the 36g memory envelope (peak 36,864 MiB) and was killed by the kernel, the same envelope every engine on this table had; it did not run out of time, and there is no row.
  • ArangoDB at 9.99M vectors is not comparable. On the release measured here, 3.12.11, an IVF index with 10,000 or more lists answers every query from one fixed list. The index at this size has 12,643 lists, so the search returned none of the true neighbours, and the cell is not a measurement of ArangoDB's search. The next measurement runs ArangoDB at its latest stable release.
  • sqlite-vec (int8) at 1M vectors is withheld: its search answered with a recall@10 of 0.0229 across 5 repetition(s), which is not a measurement of search but of a broken index, so its latency is not printed beside engines answering correctly. The cause is investigated on the benchmark machine before anything is claimed about it.
  • In the cells: `>2h` and `>8h` mean the cell ran past its tier's cap and was not retried; each tier has its own cap, the same for every engine on it. `OOM` means killed at the cell's memory envelope rather than running out of time. `n/c` means not comparable, because a defect in the engine's release decided the cell's result; none of its numbers is printed, and the note names the defect. `re-run` means the cell was measured before a fix to our harness and stays off the table until it is measured again; the note says what was wrong. A dash is an operation the engine cannot express, never a slow one.
  • Ingest paths: ArcadeDB embedded issues INSERT per vector in 10,000-row transactions through the Python package, then CREATE INDEX ... LSM_VECTOR; served sends 500-statement sqlscript batches over HTTP with each vector spelled out as text, then the same CREATE INDEX; Chroma add() in batches of 5,000; LanceDB an Arrow table then create_index; Qdrant and Milvus upsert in batches; DuckDB VSS and sqlite-vec executemany; pgvector COPY FROM STDIN then CREATE INDEX; Neo4j loads then builds its vector index; MongoDB insert_many then its vector search index; ArangoDB import_bulk then its FAISS IVF index; SurrealDB inserts through its Python SDK under the HNSW index it defines first; the server builds that index in the background, so the served build time includes waiting until a query probe shows the index has caught up.

Sparse search on real learned sparse vectors, with recall beside every latency. Loading and indexing are one timer on this table: the engines here build the index while ingesting, so the boundary the dense table splits does not exist.

Sparse vector search Big-ANN'23 Sparse (real SPLADE over MS MARCO)
EngineModePrecisionSizecold p50 ms ↓cold p99 ms ↓warm p50 ms ↓warm p99 ms ↓gain ↑recall@10 ↑ingest+index vectors/s ↑ingest+index total s ↓peak memory GiB ↓disk GiB ↓
ArcadeDBembeddedint8100k vectors3.949.573.637.591.12x0.9912k8.613.030.15
ArcadeDBembeddedfp32100k vectors4.0311.93.977.941.11x111k8.833.200.19
ArcadeDBserverint8100k vectors4.9310.54.439.051.20x0.994,01424.94.380.19
ArcadeDBserverfp32100k vectors4.9213.04.509.111.13x14,04424.74.380.23
Elasticsearchserver~9-bit100k vectors3.516.972.835.411.38x1.002,67237.44.790.094
Milvusserverfp32100k vectors1.132.511.061.401.08x1.008,53011.71.300.30
pgvectorserverfp32100k vectors0.981.760.961.701.06x0.821,54464.80.140.48
Qdrantserverfp32100k vectors0.891.200.871.111.07x112k8.140.860.44
ArcadeDBembeddedint81M vectors11.335.911.536.91.06x0.9911k89.08.351.48
ArcadeDBembeddedfp321M vectors11.135.710.736.01.12x111k89.88.421.83
ArcadeDBserverint81M vectors12.740.311.937.11.14x0.994,288233.28.651.50
ArcadeDBserverfp321M vectors12.638.812.436.81.09x14,287233.38.591.86
Elasticsearchserver~9-bit1M vectors9.9321.19.0919.31.13x1.003,142318.29.030.90
Milvusserverfp321M vectors8.0623.28.3720.81.02x1.009,770102.36.786.08
pgvectorserverfp321M vectors1.292.421.082.401.22x0.771,510662.40.254.22
Qdrantserverfp321M vectors2.854.132.794.060.99x111k88.58.091.45
ArcadeDBembeddedint88.84M vectors84.3305.283.8306.51.02x0.999,994884.716.813.1
ArcadeDBembeddedfp328.84M vectors86.5316.083.8309.31.03x1.0010k874.116.816.2
ArcadeDBserverint88.84M vectors86.9310.185.8310.51.04x0.994,0812,16716.913.1
ArcadeDBserverfp328.84M vectors87.7312.685.6311.81.04x1.004,1022,15616.916.3
Elasticsearchserver~9-bit8.84M vectors54.4128.352.3125.51.04x1.002,5673,44517.48.01
Milvusserverfp328.84M vectors20.752.321.652.70.96x1.009,740907.816.265.4
pgvectorserverfp328.84M vectors13.134.61.282.627.83x0.721,5015,8930.4029.2
Qdrantserverfp328.84M vectors15.624.816.525.30.96x1.005,3461,65419.612.3

↑ higher is better, ↓ lower is better; the best value in each column, within one size, is bold

Measured rows: benchmarks/experiments/results/runs_paper_oct.csv

Exact builds measured
  • arcadedb 26.10.1-dev · 417314c18 (in-process)
  • arcadedb 26.10.1-dev · 417314c18 arcadedb-c25:417314c18
  • elasticsearch 9.5.4 docker.elastic.co/elasticsearch/elasticsearch@33178ff49e06…
  • milvus 3.0.1 milvusdb/milvus@2b2fc2cf499a…
  • pgvector 0.8.6 pgvector/pgvector@1d50c689b0a6…
  • qdrant 1.19.1 qdrant/qdrant@0699e7733a6f…
  • Cold p50 and p99 are the first timed pass after the build, median of five builds. Warm and gain come from a separate run of the same arms: one build per engine, then five more passes over a different half of the query set, so a warm number cannot be explained by the engine having already answered that exact query; gain is that run's cold over its warm.
  • Disk is what the workload left on disk, in GiB: the engine's writable layer plus its volumes after the cell, minus the same engine's empty footprint.
  • On this table each ArcadeDB search returns every hit's whole record, while every other engine returns the ids of its hits, not the records. The next measurement has ArcadeDB return ids as well.
  • Durability is matched at the relaxed end: on every engine that has the setting, a commit returns without waiting for the disk and the log is flushed by the engine's own background policy, which is ArcadeDB's engine default and a documented production mode for each of the others. Each row records what it ran as `durability`, and the setting was read out of the engine rather than assumed on every engine except Elasticsearch, Milvus, and Qdrant. Elasticsearch, Milvus, and Qdrant run at their defaults, and their rows say so without reading a setting back, because this table times no transactional write.
  • Recall is reported beside every latency: ArcadeDB quantizes posting weights to int8 by default, so a latency number without its recall is not comparable.
  • ingest+index total s is one timer around inserting the documents and building the index; on this lane every engine builds its sparse index as it ingests, so there is no boundary to time separately. ingest+index vectors/s divides the document count by it.
  • Elasticsearch runs with index-time token pruning disabled. Its 9.x default prunes on thresholds tuned for a different model's vectors and costs recall on this corpus, which would have printed a quality gap belonging to that default rather than to the engine, and printed it in our favour.
  • ArangoDB has no row on this table. Its index types are the primary, edge, vertex-centric, persistent, inverted, TTL, geo, and vector indexes and the deprecated fulltext index (https://docs.arango.ai/arangodb/stable/indexes-and-search/indexing/basics/), and the vector index is FAISS IVF over one attribute that holds an array of numbers of a fixed dimension (https://docs.arango.ai/arangodb/stable/indexes-and-search/indexing/working-with-indexes/vector-indexes/); its sparse option means an index that skips documents missing the attribute. Nothing indexes a sparse vector, so the sparse nearest-neighbour search this table times cannot be expressed.
  • MongoDB has no row on this table. Its index types are single field, compound, multikey, wildcard, geospatial, hashed, text, clustered, and unique (https://www.mongodb.com/docs/manual/indexes/), and a Vector Search index's vector field must hold an array of numbers, as doubles or as a BinData vector of floats or small integers, with a fixed number of dimensions (https://www.mongodb.com/docs/vector-search/indexes/vector-search-type/); the field types an index accepts are vector, filter, and autoEmbed, and none is a sparse or token-and-weight form. Nothing indexes a sparse vector, so the sparse nearest-neighbour search this table times cannot be expressed.
  • SurrealDB has no row on this table. Its index types are the standard, unique, composite, count, full-text, HNSW, and DISKANN indexes, with brute force for an unindexed field and the MTREE index the embedded core also accepts, and every vector form is declared with a DIMENSION over one field that holds an array of numbers (https://surrealdb.com/docs/surrealql/statements/define/indexes); the nearest-neighbour operator takes one dense array (https://surrealdb.com/docs/surrealql/operators). Nothing indexes a sparse vector, so the sparse nearest-neighbour search this table times cannot be expressed.
  • ArcadeDB (embedded, fp32), ArcadeDB (embedded, int8), ArcadeDB (server, fp32), ArcadeDB (server, int8), and Elasticsearch run on a JVM, and for them this column is close to the heap they were given rather than the memory the work needed: Elasticsearch's 17.37 GiB here is about 109 per cent of its 16g heap. Every engine on a table gets the same memory envelope; how much of it a JVM takes is a property of the runtime, so read these cells against each other rather than against an engine that manages its own memory.
  • Each timed pass on this table answers 1,000 queries, and its p50 and p99 leave out the first 5 as warm-up; the second-pass run answers 500 of them in its cold pass and the other 500 in each warm pass.
  • Ingest paths: ArcadeDB embedded loads through the Java API (newDocument with int and float arrays) in 500-record transactions, then COMPACT INDEX; served sends INSERT statements as sqlscript batches over HTTP; Qdrant, Milvus, and Elasticsearch upsert or bulk-index in batches, then settle (Elasticsearch refresh and force-merge, Milvus flush and load); pgvector COPY FROM STDIN in sparsevec text form, then CREATE INDEX.

The rules every result follows are in the methodology. The settings for each engine are on the engines page. If one is wrong, tell us.

DBBench

The other lanes.

  • Tables: Documents and structured query language (SQL) analytics.
  • Graphs: Interactive and analytical queries on a social network graph.
  • Time series: Monitoring-style queries over timestamped readings.
  • Cross-model: One operation across a vector, a graph edge, and a document.
  • Lifecycle: Opening, closing, deployment boundaries, and what waiting for the disk costs.