Skip to content

NVIDIA CUDA

Run embedding and reranking on an NVIDIA GPU on Linux.

Build and run the Cortrix CUDA image, then confirm that the models are really running on the GPU.

The default deploy/Dockerfile and deploy/docker-compose.yml are a CPU deployment. CUDA is turned on only when you build deploy/Dockerfile.cuda and layer deploy/docker-compose.cuda.yml on top.

Note

This page is adapted from the CUDA execution provider runbook and hasn’t been tested on a GPU machine. It’s a deployment guide and contains no performance data. Complete the verification steps on your target machine before you treat CUDA as verified.

Prerequisites

  • A Linux x86_64 host and container architecture.
  • An NVIDIA GPU and a compatible driver.
  • Docker, Docker Compose, and NVIDIA Container Toolkit configured for Docker.

The image runtime is CUDA 11.8 and cuDNN 8 on Ubuntu 22.04, with the repository-pinned ONNX Runtime 1.17.1 GPU build.

Host driver requirements:

Host NVIDIA driverNotes
>= 520.61.05The required baseline, and the recommended version for deployment
>= 450.80.02 and < 520.61.05NVIDIA documents compatibility in this range, with limitations. Cortrix doesn’t treat this range as validated
< 450.80.02Below the compatibility floor for CUDA 11.8. Don’t use it

Check the host before you build:

uname -m
nvidia-smi
nvidia-smi --query-gpu=name,uuid,driver_version --format=csv,noheader
docker version
docker compose version

uname -m must print x86_64, and nvidia-smi must list the GPU you intend to use.

Execution provider values

Embedding and reranking each choose an execution provider independently, from the same set of values:

ValueBehavior in the CUDA image
autoTry CUDA first. If it fails, record the reason and create a fresh CPU-only session
cpuUse the CPU without trying CUDA
cudaRequire CUDA. A failure aborts startup, with no CPU fallback
coremlFor builds on Apple platforms. Not available in the Linux CUDA image

Environment variables take precedence over the YAML configuration. Values aren’t case-sensitive, but lowercase is recommended.

Steps

Run every command from the root of the repository.

Prepare the environment file

cp deploy/.env.example deploy/.env

At a minimum, confirm these settings:

deploy/.env
CORTRIX_PROFILE=full
CORTRIX_EMBEDDING_EXECUTION_PROVIDER=auto
CORTRIX_RERANKER_EXECUTION_PROVIDER=auto
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_DRIVER_CAPABILITIES=compute,utility

The CUDA override file passes these values into the container. When the profile is full, the Quick Start demo data isn’t loaded.

To use only one GPU, set NVIDIA_VISIBLE_DEVICES to its index or UUID. Cortrix needs only the compute and utility capabilities.

Build the image

docker compose \
  -f deploy/docker-compose.yml \
  -f deploy/docker-compose.cuda.yml \
  build cortrix

The build is restricted to linux/amd64, and model files aren’t baked into the image.

Start the stack

docker compose \
  -f deploy/docker-compose.yml \
  -f deploy/docker-compose.cuda.yml \
  up -d

Follow the startup log until the service is healthy:

docker compose \
  -f deploy/docker-compose.yml \
  -f deploy/docker-compose.cuda.yml \
  logs -f cortrix

Verify

Don’t conclude that CUDA is in use from the image name or from GPU visibility alone. Check each of these.

Check that the container sees the GPU

docker compose \
  -f deploy/docker-compose.yml \
  -f deploy/docker-compose.cuda.yml \
  exec cortrix nvidia-smi -L

Only the GPUs selected by NVIDIA_VISIBLE_DEVICES should be listed.

Check the runtime libraries

docker compose \
  -f deploy/docker-compose.yml \
  -f deploy/docker-compose.cuda.yml \
  exec cortrix sh -c \
  'out=$(ldd /usr/local/lib/libonnxruntime_providers_cuda.so) && printf "%s\n" "$out" && ! printf "%s\n" "$out" | grep -q "not found"'

This command succeeds only when there are no missing dependencies.

Check the active execution provider

curl -fsS http://localhost:8420/api/v1/system/health/ready

Look at components.embedding_execution_provider and components.reranker_execution_provider:

SituationWhat you see
CUDA is workingactive_ep is cuda
auto fell back to the CPUThe service is still ready, but fallback is true, active_ep is cpu, and preferred_ep is cuda
The explicit setting and reality don’t matchThe service isn’t ready, and policy_mismatch is true
No model is configuredmodel_configured is false and active_ep is stub

Check the metrics

The metrics port, 9091, is open only inside the container, so query it from there:

docker compose \
  -f deploy/docker-compose.yml \
  -f deploy/docker-compose.cuda.yml \
  exec -T cortrix sh -c \
  'curl -fsS http://127.0.0.1:9091/metrics | grep -E "cortrix_onnx_build_info|embedding_(configured|active)_ep|reranker_(configured|active)_ep"'

A CUDA build includes this line:

cortrix_onnx_build_info{runtime_flavor="cuda"} 1

Require the GPU

A typical CUDA deployment keeps both components on auto. If you’d rather have startup fail than fall back to the CPU when GPU initialization fails, change that component to cuda and recreate the service:

docker compose \
  -f deploy/docker-compose.yml \
  -f deploy/docker-compose.cuda.yml \
  up -d --force-recreate cortrix

Roll back to CPU

To switch back to the default CPU image and keep the data volume:

docker compose \
  -f deploy/docker-compose.yml \
  -f deploy/docker-compose.cuda.yml \
  down
docker compose -f deploy/docker-compose.yml up -d --build
Caution

Don’t add --volumes to down unless you really mean to delete your data.

To keep the CUDA image and just use the CPU for now, set both execution provider variables to cpu and recreate the service.

Troubleshooting

SymptomWhat to do
Docker reports that no GPU driver is availableConfirm nvidia-smi works on the host, and that NVIDIA Container Toolkit is installed and configured for Docker
The container is rejected before the entrypoint runsCheck the Docker daemon log for an NVIDIA_REQUIRE_CUDA or driver-version constraint error, and upgrade the host driver. Don’t set NVIDIA_DISABLE_REQUIRE=true to bypass the check
The pre-flight check can’t see a GPUCheck the value of NVIDIA_VISIBLE_DEVICES, and that NVIDIA_DRIVER_CAPABILITIES includes compute,utility
ldd reports CUDA or cuDNN libraries as not foundRebuild from deploy/Dockerfile.cuda and confirm the runtime stage is the CUDA 11.8 and cuDNN 8 image

Next steps