Skip to content

Performance tuning

The settings that affect query latency, recall, and ingestion speed, and how to measure them on your own data.

A handful of settings affect Cortrix’s performance. Learn which stage each one acts on and how to measure its effect on your own data.

Important

Which way to turn a setting depends on your task and your data. The numbers on this page only illustrate direction. Measure on your own data before you change a default.

Measure first

Every query response includes the latency the server measured:

curl -fsS -H 'Content-Type: application/json' \
  -d '{"namespaces": ["demo"], "query": "semantic storage", "top_k": 5}' \
  http://127.0.0.1:8420/api/v1/query
{
  "meta": {
    "coverage_ratio": 1.0,
    "latency_ms": 242,
    "namespaces_succeeded": ["demo"]
  }
}

Change one variable at a time, repeat the same set of queries several times, and compare the median meta.latency_ms. Check result quality too: faster isn’t the same as better.

Add ?explain=true to see which retrieval paths a query used. See Query and rerank.

Settings at a glance

SettingWhereDefaultAffects
rerankQuery requesttrueQuery latency and result ordering
top_kQuery request10How many results come back, and the candidate pool size
retrieval.candidate_multiplierConfiguration file3Candidate pool size
retrieval.max_candidatesConfiguration file50The candidate pool ceiling, and so the recall ceiling
embedding.execution_providerConfiguration fileautoWhere embedding runs
reranker.execution_providerConfiguration fileautoWhere reranking runs
server.thread_countConfiguration file4The server’s worker threads
spc.worker_countConfiguration file2How many documents are processed in parallel
spc.embedding_batch_sizeConfiguration file1Blocks embedded per batch
spc.onnx_intra_threadsConfiguration file4Intra-op threads for ONNX inference
spc.onnx_inter_threadsConfiguration file1Parallel ONNX inferences

The configuration file defaults come from config.yaml.example in the repository.

Queries

Reranking

Reranking is the single switch with the biggest effect on query latency. When it’s on, Cortrix scores each retrieved candidate with the cross-encoder model bge-reranker-v2-m3.

As a rough illustration, here is one measurement taken while writing this page. It isn’t a published benchmark, and it isn’t in the repository. On the quickstart service, with 3 short documents and CPU inference in Docker on an Apple silicon Mac, the median over 8 queries per setting was:

SettingMedian latency_ms
rerank: true242
rerank: false52

The measurements published in the repository show that on the Quora corpus (522,931 documents, a 32-core CPU, no GPU, serial queries), mean query latency goes from 1.7 seconds without reranking to 19.0 seconds with it.

Whether reranking is worth that cost depends on the kind of task:

Kind of taskExamplesRecommendation
Questions and answersDocumentation search, finding solutions in support tickets, grounding a RAG answerLeave it on
Finding identical or duplicate itemsDuplicate-question detectionTurn it off

In the published measurements, the three question-answering datasets (SciFact, NFCorpus, FiQA) gained 0.013 to 0.059 nDCG@10 with reranking on, while duplicate-question detection (Quora) dropped from 0.5003 to 0.2031. The reranker optimizes for “relevant,” and that kind of task is looking for “the same.”

For neighboring cases such as duplicate tickets, similar products, or “find similar” features, the same problem is a hypothesis, not a measurement. Test it on your own data.

To turn reranking off, set "rerank": false on each query.

Warning

Only the rerank flag on the query request takes effect today. The namespace-level reranker_config is accepted, but the query path doesn’t read it yet. See cortrix/cortrix#75. For a workload that shouldn’t be reranked, every request needs to pass "rerank": false explicitly.

For the measurement conditions and full data, see When to rerank and Benchmark evidence.

Candidate pool size

Cortrix fetches more candidates than top_k, then ranks and truncates. The number of candidates is:

candidate_k = min(top_k × candidate_multiplier × oversample, max_candidates)
  • A larger top_k means more candidates and more content returned.
  • max_candidates is a hard ceiling. On a large corpus, relevant documents can rank below 50, so raising max_candidates is the only way to lift the recall ceiling.
config.yaml
retrieval:
  candidate_multiplier: 3
  max_candidates: 50

You can also override these with the CORTRIX_RETRIEVAL_CANDIDATE_MULTIPLIER and CORTRIX_RETRIEVAL_MAX_CANDIDATES environment variables.

Query scope

List only the namespaces you actually need in namespaces. ["*"] queries every namespace you have access to, up to 100.

Time limits

A query request has no time budget of its own on this release. A query over several namespaces has two default wait limits in the step that collects the namespaces’ results: that step waits up to 5 seconds for each namespace in turn, counted from when it reaches that namespace, and stops waiting altogether 30 seconds after it started. A namespace it stops waiting for is listed among the failed namespaces with CX_ERR_NS_TIMEOUT, and the results from the other namespaces still come back.

These limits bound waiting, not work. A namespace can take longer than 5 seconds and still succeed when the collection reaches it later. The 30 seconds are not an end-to-end limit for the request: waiting for a free worker, RAG-Fusion and reranking come on top. The work of a namespace the collection stopped waiting for is not cancelled. A query over a single namespace has no wait limit.

Execution providers

The embedding model and the reranker each have an execution_provider setting. The values are auto, cpu, coreml, and cuda:

config.yaml
embedding:
  execution_provider: "auto"

reranker:
  execution_provider: "auto"

The matching environment variables are CORTRIX_EMBEDDING_EXECUTION_PROVIDER and CORTRIX_RERANKER_EXECUTION_PROVIDER. The quickstart pins both to cpu.

The readiness check shows which execution provider is actually in use:

curl -fsS http://127.0.0.1:8420/api/v1/system/health/ready
{
  "embedding_execution_provider": {
    "active_ep": "cpu",
    "configured_ep": "cpu",
    "fallback": false,
    "status": "ok"
  }
}

active_ep is the provider in use. A fallback of true means the configured provider wasn’t available and Cortrix fell back.

Using an NVIDIA GPU on Linux needs a separate image. See NVIDIA CUDA. The CUDA document in the repository is a deployment guide and doesn’t include performance data.

Ingestion

After upload, a document is parsed, chunked, and embedded. These settings control how much of that pipeline runs in parallel:

config.yaml
spc:
  worker_count: 2
  chunk_size: 512
  chunk_overlap: 50
  embedding_batch_size: 1
  onnx_intra_threads: 4
  onnx_inter_threads: 1
SettingPurpose
worker_countThreads processing documents in parallel. 2 to 4 is recommended
embedding_batch_sizeBlocks sent to the embedding model per batch
onnx_intra_threadsThreads used inside a single ONNX inference
onnx_inter_threadsONNX inferences that run in parallel
chunk_sizeThe target number of tokens per text block. Smaller blocks mean more blocks per document

These are configuration-file keys. They have no environment-variable override.

The readiness check reports the current queue depth and worker count under spc_pipeline:

{ "spc_pipeline": { "queue_depth": 0, "status": "ok", "workers": 2 } }

Server

config.yaml
server:
  thread_count: 4
  • server.thread_count is the number of worker threads handling requests. The recommended value is your CPU core count. You can override it with CORTRIX_SERVER_THREADS.

What these numbers do and don’t tell you

The measurements published in the repository cover retrieval quality at top_k=10 and mean latency under serial queries. They say nothing about production latency or capacity under concurrency, or about end-to-end answer quality.

This page doesn’t recommend values for thread counts, batch size, or execution providers, because the repository has no published controlled measurements for them. Measure those on your own hardware and data, using the method above, before you settle on values.

Next steps

Table reader