Why vector search benchmarks are meaningless without recall
Benchmarking Vector Indexes
Percona introduces vector-bench, an open benchmark that measures vector indexes fairly. It explains the two main index types, HNSW and IVF, and why recall and throughput must be reported together. The post details the harness, including build methods, concurrency, and filtered search, and warns against common pitfalls like TOAST in PostgreSQL.
Which is why “our database does 3,678 vector queries a second” tells you nothing.
- stephantul
This is a super long ai generated slab of text with very little actual info. It doesn’t include any analysis of results.
It does include the phrase “worth listing”.
- BenoitP
What's wrong with https://ann-benchmarks.com/index.html
?
- wood_spirit
Really look forward to seeing the results.
It would be good if it includes as many things as possible, such as redid, annoy, voyager and lancedb etc and not just oltp sql dbs.