Skip to content

Benchmark evidence

The retrieval-quality measurements that have been published, and what they apply to.

Cortrix has published one set of retrieval-quality measurements. Here are the conditions they were taken under, and what the numbers do and don’t tell you.

Measurement bundle

The measurements accepted for public use are the pinned four-corpus CPU measurement bundle, which holds results for 16 retrieval configurations.

Measurement conditions

ItemDetails
HardwareA 32-core Xeon Silver 4110 with 192 GB of RAM and no GPU
Modelsbge-m3 embedding and bge-reranker-v2-m3 reranking, both ONNX fp32 on CPU
Query shapeSerial, one query at a time, top_k=10
ScoringnDCG@10 and Recall@10 after de-duplicating by doc_id
CorpusDocumentsQueries
SciFact5,183300
NFCorpus3,633323
FiQA57,638648
Quora522,931The first 2,000 of 10,000 judged queries

SciFact, NFCorpus, and FiQA use every judged test query.

Results

The table shows nDCG@10:

DatasetTaskDense onlyWith cross-encoder reranking
SciFactScientific claim verification0.59420.6184
NFCorpusNutrition and medical QA0.29910.3121
FiQAFinancial forum QA0.25400.3125
QuoraDuplicate-question detection0.50030.2031

The three question-answering datasets gain 0.013 to 0.059 with reranking on. Quora loses 0.297.

Reranking helps question-answering tasks and hurts duplicate-detection tasks. For why, and what to do about it, see Performance tuning.

Comparability

  • The two Quora configurations query the same 8 namespaces, so the rerank flag is the only variable. This is the bundle’s only strictly controlled comparison.
  • The SciFact, NFCorpus, and FiQA configurations use independently ingested namespaces. Small differences between them fall within the variation floor the bundle discloses, roughly 0.001 to 0.003 nDCG.

What these numbers don’t show

The bundle measures retrieval quality at top_k=10. It isn’t evidence of:

  • End-to-end answer quality
  • Production latency or capacity under concurrency
  • Security or compliance properties
  • How Cortrix ranks against other products
  • Business outcomes

The latency figures quoted in this documentation are means under serial queries and don’t represent behavior under concurrent load.

Important

Act on the direction these measurements show, not on the absolute values. Verify that direction on your own corpus before you choose.

Next steps