NVIDIA CUDA
Build and run the Cortrix CUDA image, then confirm that the models are really running on the GPU.
The default deploy/Dockerfile and deploy/docker-compose.yml are a CPU deployment. CUDA is turned on only when you build deploy/Dockerfile.cuda and layer deploy/docker-compose.cuda.yml on top.
This page is adapted from the CUDA execution provider runbook and hasn’t been tested on a GPU machine. It’s a deployment guide and contains no performance data. Complete the verification steps on your target machine before you treat CUDA as verified.
Prerequisites
- A Linux x86_64 host and container architecture.
- An NVIDIA GPU and a compatible driver.
- Docker, Docker Compose, and NVIDIA Container Toolkit configured for Docker.
The image runtime is CUDA 11.8 and cuDNN 8 on Ubuntu 22.04, with the repository-pinned ONNX Runtime 1.17.1 GPU build.
Host driver requirements:
| Host NVIDIA driver | Notes |
|---|---|
>= 520.61.05 | The required baseline, and the recommended version for deployment |
>= 450.80.02 and < 520.61.05 | NVIDIA documents compatibility in this range, with limitations. Cortrix doesn’t treat this range as validated |
< 450.80.02 | Below the compatibility floor for CUDA 11.8. Don’t use it |
Check the host before you build:
uname -m must print x86_64, and nvidia-smi must list the GPU you intend to use.
Execution provider values
Embedding and reranking each choose an execution provider independently, from the same set of values:
| Value | Behavior in the CUDA image |
|---|---|
auto | Try CUDA first. If it fails, record the reason and create a fresh CPU-only session |
cpu | Use the CPU without trying CUDA |
cuda | Require CUDA. A failure aborts startup, with no CPU fallback |
coreml | For builds on Apple platforms. Not available in the Linux CUDA image |
Environment variables take precedence over the YAML configuration. Values aren’t case-sensitive, but lowercase is recommended.
Steps
Run every command from the root of the repository.
Prepare the environment file
At a minimum, confirm these settings:
The CUDA override file passes these values into the container. When the profile is full, the Quick Start demo data isn’t loaded.
To use only one GPU, set NVIDIA_VISIBLE_DEVICES to its index or UUID. Cortrix needs only the compute and utility capabilities.
Build the image
The build is restricted to linux/amd64, and model files aren’t baked into the image.
Start the stack
Follow the startup log until the service is healthy:
Verify
Don’t conclude that CUDA is in use from the image name or from GPU visibility alone. Check each of these.
Check that the container sees the GPU
Only the GPUs selected by NVIDIA_VISIBLE_DEVICES should be listed.
Check the runtime libraries
This command succeeds only when there are no missing dependencies.
Check the active execution provider
Look at components.embedding_execution_provider and components.reranker_execution_provider:
| Situation | What you see |
|---|---|
| CUDA is working | active_ep is cuda |
auto fell back to the CPU | The service is still ready, but fallback is true, active_ep is cpu, and preferred_ep is cuda |
| The explicit setting and reality don’t match | The service isn’t ready, and policy_mismatch is true |
| No model is configured | model_configured is false and active_ep is stub |
Check the metrics
The metrics port, 9091, is open only inside the container, so query it from there:
A CUDA build includes this line:
Require the GPU
A typical CUDA deployment keeps both components on auto. If you’d rather have startup fail than fall back to the CPU when GPU initialization fails, change that component to cuda and recreate the service:
Roll back to CPU
To switch back to the default CPU image and keep the data volume:
Don’t add --volumes to down unless you really mean to delete your data.
To keep the CUDA image and just use the CPU for now, set both execution provider variables to cpu and recreate the service.
Troubleshooting
| Symptom | What to do |
|---|---|
| Docker reports that no GPU driver is available | Confirm nvidia-smi works on the host, and that NVIDIA Container Toolkit is installed and configured for Docker |
| The container is rejected before the entrypoint runs | Check the Docker daemon log for an NVIDIA_REQUIRE_CUDA or driver-version constraint error, and upgrade the host driver. Don’t set NVIDIA_DISABLE_REQUIRE=true to bypass the check |
| The pre-flight check can’t see a GPU | Check the value of NVIDIA_VISIBLE_DEVICES, and that NVIDIA_DRIVER_CAPABILITIES includes compute,utility |
ldd reports CUDA or cuDNN libraries as not found | Rebuild from deploy/Dockerfile.cuda and confirm the runtime stage is the CUDA 11.8 and cuDNN 8 image |