The largest open vector benchmark yet
Qdrant published a new open source benchmark on September 1, 2026 called Qdrant-FineWeb-10B, describing it as the largest open vector search benchmark released to date. The dataset totals 24.47 terabytes of vector data and 28.66 terabytes of source text and metadata, comprising 10.07 billion dense vectors and 10.07 billion sparse vectors generated from the gte-multilingual-base embedding model. Qdrant computed exact top-1000 ground truth results for 100,000 queries against the full dataset, a task the company says required more than one quadrillion distance calculations.
That scale is deliberate. Most vector database benchmarks in circulation today run in the tens or low hundreds of millions of vectors, several orders of magnitude below what large enterprises actually index once they move past a pilot into production search, recommendation, or retrieval-augmented generation across a real product catalog or document corpus. Qdrant built this release specifically to close that gap between what benchmarks measure and what production actually demands, and the choice to publish the full dataset and ground truth openly, rather than keeping the test set proprietary, is itself a statement about what a trustworthy benchmark should look like.
The complaint: benchmarks built to flatter, not to inform
Qdrant's blog post accompanying the release was pointed about why this mattered enough to build. The company criticized prevailing benchmarks for depending on gated, proprietary managed services running against synthetic data with hidden query sets, a setup that makes results difficult to independently verify or reproduce. It singled out a common marketing pattern: claiming strong recall, for example 90 percent recall at 10, on a purely synthetic retrieval-augmented generation benchmark, when real e-commerce and enterprise search workloads need results across the top 1,000 or more candidates, sustained above 95 percent recall, at high throughput and under 100 milliseconds of p99 latency.
That gap between a marketing benchmark and a production requirement is exactly where procurement decisions go wrong. A vendor can post an impressive recall number that is technically true and still be irrelevant to whether their system will hold up once real query volume, real document counts, and real latency service-level agreements enter the picture. Qdrant's framing is a direct challenge to every vendor, including itself, to be evaluated against harder, more realistic conditions, and it puts pressure on the entire category to publish reproducible numbers rather than curated highlight results.
What else came with the release
Beyond the headline dataset, Qdrant published two additional benchmarks aimed at retrieval patterns that are increasingly common in production but poorly covered by existing tests: PubMed-Multi-Vector for hybrid retrieval scenarios and Coyo-Vector-Embeddings for multimodal search. Both address a real blind spot, since most public vector benchmarks still test single-modality, single-vector retrieval even though production systems increasingly combine dense and sparse signals or search across text and image embeddings together.
Qdrant also released Supernova, an open source benchmarking framework structured around four phases: embedding generation, ground-truth calculation, database loading, and evaluation. The company built the release with infrastructure partners Vultr, SkyPilot, and Hugging Face, which matters practically, since reproducing a 24-terabyte benchmark requires real compute and storage partnerships, not just a dataset dump on a download page. Publishing the tooling alongside the data means any team, not just Qdrant's own engineers, can rerun the full pipeline against a competing database and generate comparable numbers rather than trusting a single vendor's report.
Why this changes how you should shop for a vector database
Vector database selection has largely run on trust in vendor-published numbers, because independently reproducing a rigorous benchmark at meaningful scale has been expensive and technically demanding for any buyer to do alone. An open, reproducible benchmark at 10 billion vectors removes that excuse. Any vendor claiming production readiness at enterprise scale can now be asked directly whether they have published or would accept evaluation against Qdrant-FineWeb-10B or an equivalent independently verifiable test.
This is true even for teams not currently considering Qdrant as a vendor. The benchmark's value is in the methodology and the honesty of scale, not in favoring any single database. Procurement and platform teams evaluating any vector search product, whether Qdrant, Pinecone, Weaviate, Milvus, or a cloud-native option, now have a credible yardstick to demand vendors run against, rather than relying on a comparison chart the vendor produced itself. A vendor's willingness to engage with an open, adversarial benchmark at all is itself a useful signal about how confident they are in their own numbers.
The decision to make before your next renewal
If your organization has a vector database contract coming up for renewal, or is mid-evaluation for a new one, add a specific requirement to the vendor scorecard: results, ideally reproducible by your own team, at retrieval depth and latency targets that match your actual production requirements, not a vendor's chosen demonstration scenario. Ask specifically whether the vendor has run against Qdrant-FineWeb-10B or a comparably scaled open dataset, and treat a refusal or a redirection to a proprietary benchmark as useful information in itself.
The broader lesson extends past vector databases. Every category of infrastructure software eventually accumulates a set of benchmark numbers optimized for marketing rather than for buyer decision-making. Qdrant's release is a reminder that the fix is to insist on tests you or an independent party can reproduce at the scale you actually plan to operate at, before the contract is signed rather than after performance problems show up in production, rather than to distrust every number a vendor publishes. Build that requirement into your next RFP template now, while the memory of this release is fresh, rather than relearning the lesson during an outage.
