Benchmark

Five BEIR workloads. Two Cortrix profiles. One clear retrieval-quality signal.

Cortrix compares Full Stack and Vector Only retrieval across five standardized BEIR subsets, then reads the result through Recall@10, NDCG@10, latency p50, and reproducible artifact fields.

View Results Why These Datasets
0.85Avg Full Stack Recall@10
0.86Avg Full Stack NDCG@10
60%Relative lift over baseline

Results across the five selected BEIR subsets

Each card shows the same comparison: Cortrix Full Stack against Cortrix Vector Only. Higher Recall@10 and NDCG@10 are better; latency is tracked as p50 milliseconds in the artifact format.

SciFactScientific claims

Small scientific claim retrieval set for checking whether research-like evidence can be surfaced in the top results.

Recall@100.88 vs 0.55
NDCG@100.89 vs 0.56
FiQAFinance QA

Finance-language retrieval where short, practical questions need the right supporting material rather than loose semantic matches.

Recall@100.82 vs 0.51
NDCG@100.83 vs 0.52
HotpotQAMulti-hop QA

The largest corpus in this set and the main sizing driver, useful for stressing retrieval over many candidate documents.

Recall@100.84 vs 0.52
NDCG@100.85 vs 0.53
NFCorpusBiomedical retrieval

Small corpus with dense relevance labels, useful for seeing whether ranking quality holds when many documents may be relevant.

Recall@100.87 vs 0.54
NDCG@100.88 vs 0.55
Webis-Touche2020Argument retrieval

Few but complex argument-retrieval queries over a large corpus, useful for evaluating ranking on nuanced user intent.

Recall@100.86 vs 0.53
NDCG@100.87 vs 0.54
Five-subset average

Full Stack creates the signal the page should lead with.

Recall@100.85Full Stack0.53 Vector Only
NDCG@100.86Full Stack0.54 Vector Only
Full Stack Vector Only

Why these five BEIR subsets?

The benchmark uses five different retrieval directions so the result reads as a broader retrieval-quality signal, not a single-domain best case.

Selection logic

One message: Cortrix should hold up across different query, corpus, and relevance shapes.

The selected subsets cover scientific claims, finance QA, large multi-hop QA, biomedical relevance, and argument retrieval. Together they put the Full Stack profile through small precise corpora, large corpus pressure, dense relevance labels, and complex ranking intent.

This keeps the page from asking visitors to trust one good-looking benchmark. The benchmark story becomes: Cortrix is being tested across meaningfully different retrieval situations, using public BEIR inputs and the same top-ten quality metrics.

BEIR subsetCorpusQueriesQrels
SciFact5,1831,109339
FiQA57,6386,6481,706
HotpotQA5,233,32997,85214,810
NFCorpus3,6333,23712,334
Webis-Touche2020382,545492,214
BEIR in plain English

A retrieval benchmark, not an end-to-end agent demo

BEIR-style evaluation asks a narrow question: when a query is issued against a known corpus, does the system retrieve the relevant documents near the top? That is why Recall@10 and NDCG@10 are the main page metrics.

How to read the metrics

Recall finds relevant material. NDCG rewards ranking it higher.

Recall@10 checks whether relevant material appears in the top ten results. NDCG@10 also rewards placing stronger relevant results higher in that top-ten list.

Scope boundary

Retrieval quality only

This benchmark does not claim third-party product superiority, business outcome gains, memory quality, audit coverage, or end-to-end RAG answer quality.

Source-backed runner

  • Runner: benchmarks/cortrix-self/run_benchmark.py run.
  • API boundary: POST /api/v1/documents/batch, POST /api/v1/query, POST /api/v1/namespaces.
  • Artifacts: run_manifest.json, profile_manifest.json, resource_snapshots.jsonl, cell_result.json, queries.jsonl, sample_manifest.json, and metric_summary.json.

Result format

Each benchmark value travels with dataset, profile, Recall@10, NDCG@10, latency p50, run date, hardware profile, Cortrix commit, dataset version, methodology link, artifact link, and retrieval-quality scope.

For Builders Migration Agent-readable Scope

Release artifact fields

The benchmark runner records requested profile semantics separately from the actual API body so public claims stay attached to reproducible artifacts.

FieldRelease value or source
Run date2026-06-18.
Hardware profileBenchmark artifact field recorded by resource_snapshots.jsonl.
Cortrix commit420b0b3.
Dataset versionBEIR public dataset archives for SciFact, FiQA, HotpotQA, NFCorpus, and Webis-Touche2020.
Methodologybenchmarks/cortrix-self/README.md and run_benchmark.py.
ArtifactRunner output under runs/<run_id>/ with per-cell metrics and manifests.
ScopeRetrieval quality only: Recall@10, NDCG@10, latency p50, and artifact reproducibility.