Performance tuning
A handful of settings affect Cortrix’s performance. Learn which stage each one acts on and how to measure its effect on your own data.
Which way to turn a setting depends on your task and your data. The numbers on this page only illustrate direction. Measure on your own data before you change a default.
Measure first
Every query response includes the latency the server measured:
Change one variable at a time, repeat the same set of queries several times, and compare the median meta.latency_ms. Check result quality too: faster isn’t the same as better.
Add ?explain=true to see which retrieval paths a query used. See Query and rerank.
Settings at a glance
| Setting | Where | Default | Affects |
|---|---|---|---|
rerank | Query request | true | Query latency and result ordering |
top_k | Query request | 10 | How many results come back, and the candidate pool size |
retrieval.candidate_multiplier | Configuration file | 3 | Candidate pool size |
retrieval.max_candidates | Configuration file | 50 | The candidate pool ceiling, and so the recall ceiling |
embedding.execution_provider | Configuration file | auto | Where embedding runs |
reranker.execution_provider | Configuration file | auto | Where reranking runs |
server.thread_count | Configuration file | 4 | The server’s worker threads |
spc.worker_count | Configuration file | 2 | How many documents are processed in parallel |
spc.embedding_batch_size | Configuration file | 1 | Blocks embedded per batch |
spc.onnx_intra_threads | Configuration file | 4 | Intra-op threads for ONNX inference |
spc.onnx_inter_threads | Configuration file | 1 | Parallel ONNX inferences |
The configuration file defaults come from config.yaml.example in the repository.
Queries
Reranking
Reranking is the single switch with the biggest effect on query latency. When it’s on, Cortrix scores each retrieved candidate with the cross-encoder model bge-reranker-v2-m3.
As a rough illustration, here is one measurement taken while writing this page. It isn’t a published benchmark, and it isn’t in the repository. On the quickstart service, with 3 short documents and CPU inference in Docker on an Apple silicon Mac, the median over 8 queries per setting was:
| Setting | Median latency_ms |
|---|---|
rerank: true | 242 |
rerank: false | 52 |
The measurements published in the repository show that on the Quora corpus (522,931 documents, a 32-core CPU, no GPU, serial queries), mean query latency goes from 1.7 seconds without reranking to 19.0 seconds with it.
Whether reranking is worth that cost depends on the kind of task:
| Kind of task | Examples | Recommendation |
|---|---|---|
| Questions and answers | Documentation search, finding solutions in support tickets, grounding a RAG answer | Leave it on |
| Finding identical or duplicate items | Duplicate-question detection | Turn it off |
In the published measurements, the three question-answering datasets (SciFact, NFCorpus, FiQA) gained 0.013 to 0.059 nDCG@10 with reranking on, while duplicate-question detection (Quora) dropped from 0.5003 to 0.2031. The reranker optimizes for “relevant,” and that kind of task is looking for “the same.”
For neighboring cases such as duplicate tickets, similar products, or “find similar” features, the same problem is a hypothesis, not a measurement. Test it on your own data.
To turn reranking off, set "rerank": false on each query.
Only the rerank flag on the query request takes effect today. The namespace-level reranker_config is accepted, but the query path doesn’t read it yet. See cortrix/cortrix#75. For a workload that shouldn’t be reranked, every request needs to pass "rerank": false explicitly.
For the measurement conditions and full data, see When to rerank and Benchmark evidence.
Candidate pool size
Cortrix fetches more candidates than top_k, then ranks and truncates. The number of candidates is:
- A larger
top_kmeans more candidates and more content returned. max_candidatesis a hard ceiling. On a large corpus, relevant documents can rank below 50, so raisingmax_candidatesis the only way to lift the recall ceiling.
You can also override these with the CORTRIX_RETRIEVAL_CANDIDATE_MULTIPLIER and CORTRIX_RETRIEVAL_MAX_CANDIDATES environment variables.
Query scope
List only the namespaces you actually need in namespaces. ["*"] queries every namespace you have access to, up to 100.
Time limits
A query request has no time budget of its own on this release. A query over several namespaces has two default wait limits in the step that collects the namespaces’ results: that step waits up to 5 seconds for each namespace in turn, counted from when it reaches that namespace, and stops waiting altogether 30 seconds after it started. A namespace it stops waiting for is listed among the failed namespaces with CX_ERR_NS_TIMEOUT, and the results from the other namespaces still come back.
These limits bound waiting, not work. A namespace can take longer than 5 seconds and still succeed when the collection reaches it later. The 30 seconds are not an end-to-end limit for the request: waiting for a free worker, RAG-Fusion and reranking come on top. The work of a namespace the collection stopped waiting for is not cancelled. A query over a single namespace has no wait limit.
Execution providers
The embedding model and the reranker each have an execution_provider setting. The values are auto, cpu, coreml, and cuda:
The matching environment variables are CORTRIX_EMBEDDING_EXECUTION_PROVIDER and CORTRIX_RERANKER_EXECUTION_PROVIDER. The quickstart pins both to cpu.
The readiness check shows which execution provider is actually in use:
active_ep is the provider in use. A fallback of true means the configured provider wasn’t available and Cortrix fell back.
Using an NVIDIA GPU on Linux needs a separate image. See NVIDIA CUDA. The CUDA document in the repository is a deployment guide and doesn’t include performance data.
Ingestion
After upload, a document is parsed, chunked, and embedded. These settings control how much of that pipeline runs in parallel:
| Setting | Purpose |
|---|---|
worker_count | Threads processing documents in parallel. 2 to 4 is recommended |
embedding_batch_size | Blocks sent to the embedding model per batch |
onnx_intra_threads | Threads used inside a single ONNX inference |
onnx_inter_threads | ONNX inferences that run in parallel |
chunk_size | The target number of tokens per text block. Smaller blocks mean more blocks per document |
These are configuration-file keys. They have no environment-variable override.
The readiness check reports the current queue depth and worker count under spc_pipeline:
Server
server.thread_countis the number of worker threads handling requests. The recommended value is your CPU core count. You can override it withCORTRIX_SERVER_THREADS.
What these numbers do and don’t tell you
The measurements published in the repository cover retrieval quality at top_k=10 and mean latency under serial queries. They say nothing about production latency or capacity under concurrency, or about end-to-end answer quality.
This page doesn’t recommend values for thread counts, batch size, or execution providers, because the repository has no published controlled measurements for them. Measure those on your own hardware and data, using the method above, before you settle on values.