# AMD Container Runtime Toolkit > AMD Container Toolkit documentation > Note: ROCm documentation is split across multiple projects. In addition to this file, each project publishes its own `llms.txt` and `llms-full.txt` under `https:///projects//en/latest/`. ## Docs - [Overview](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/overview.html) - [Requirements](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/requirements.html) - [Quick Start Guide](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/quick-start-guide.html) - [Container Device Interface](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/cdi-guide.html) - [Running Workloads](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/running-workloads.html) - [Framework Integration](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/framework-integration.html) - [Troubleshooting](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/troubleshooting.html) - [Migration Guide](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/migration-guide.html) - [Developer Guide](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/developer-guide.html) - [Docker Compose Usage](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/docker-compose.html) - [Enroot Pyxis Installation](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/enroot-pyxis-installation.html) - [Support for Docker Swarm](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/docker-swarm.html) - [GPU Tracker](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/gpu-tracker.html) - [Release Notes](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/release-notes.html) --- Source: https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/index.html # AMD Container Toolkit Documentation This documentation site provides information about the AMD Container Toolkit, which is a set of tools and libraries for running GPU-accelerated applications in containers. The ROCm platform is designed to enable high-performance computing (HPC) and machine learning workloads on AMD GPUs. - [Overview](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/overview.html) - [Requirements](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/requirements.html) - [Quick Start Guide](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/quick-start-guide.html) - [Container Device Interface](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/cdi-guide.html) - [Running Workloads](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/running-workloads.html) - [Framework Integration](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/framework-integration.html) - [Troubleshooting](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/troubleshooting.html) - [Migration Guide](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/migration-guide.html) - [Developer Guide](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/developer-guide.html) - [Docker Compose](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/docker-compose.html) - [Enroot Pyxis Installation](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/enroot-pyxis-installation.html) - [Support for Docker Swarm](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/docker-swarm.html) - [GPU Tracker](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/gpu-tracker.html) - [Release Notes](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/release-notes.html) --- Source: https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/overview.html # Overview - The AMD Container Toolkit provides a robust and flexible framework to streamline the use of AMD Instinct GPUs with containerized applications. - It simplifies GPU access within Docker environments, enhances device discovery, and enables better integration with modern container technologies. - The toolkit is designed to work seamlessly with the ROCm software stack, allowing developers to leverage the full power of AMD GPUs for high-performance computing, machine learning, and other GPU-accelerated workloads. - The AMD Container Toolkit architecture integrates directly with the Docker daemon to manage GPU resources seamlessly. The toolkit consists of two primary components: - **amd-container-runtime**: A custom container runtime (wrapper around `runc`) for injecting AMD GPUs into container specifications. - **amd-ctk (Container Toolkit CLI)**: A command-line utility for managing GPU configurations, runtime settings, and container orchestration integrations. ## Key Benefits: - Seamless GPU access in containers with minimal configuration. - Simplified device management and discovery through CDI and environment variables. - Smooth integration with popular containerized machine learning, HPC, and data science frameworks. - Enables efficient image building and development workflows for AMD GPUs. ## Use Cases: - Building machine learning applications with GPU acceleration. - Running ROCm-compatible containerized workloads. - Rapid experimentation in development environments. # Core Concepts ## Architecture Overview The AMD Container Toolkit sits between Docker and the Linux container runtime, enabling GPU access by modifying the OCI specification before handing off to `runc`. Visual Flow: ```text +------------+ +---------------+ +-----------------------+ | | | | | | | Docker CLI | AMD_VISIBLE_DEVICES | Docker Daemon | OCI SPEC | AMD Container Runtime | | | --> | | --> | | +------------+ +---------------+ +-----------------------+ | UPDATED OCI SPEC INJECTED GPU DEVICES | v +----------------------+ | RUNC + +----------------------+ | v +----------------------+ | CONTAINER PROCESS + +----------------------+ | v +----------------------+ | GPU DRIVER + +----------------------+ ``` ## Docker integration with amd-container-runtime For installation, configuration, and running GPU workloads with Docker using the amd-container-runtime, see the [Quick Start Guide](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/quick-start-guide.html). ## Using CDI for GPU injection The toolkit also supports the Container Device Interface (CDI) for GPU injection. To set up and use CDI to run workloads, see the [CDI guide](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/cdi-guide.html) and [Running Workloads](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/running-workloads.html). --- Source: https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/requirements.html # System Requirements Before installing or using the AMD Container Toolkit, ensure your environment meets the following prerequisites: ## Supported Operating Systems - Ubuntu 22.04 LTS (Jammy Jellyfish) - Ubuntu 24.04 LTS (Noble Numbat) - Ubuntu 26.04 LTS (Resolute Raccoon) - RHEL 9.5 #### NOTE - RHEL 9.5 support is new in v1.1.0 and requires the RPM-based installation flow. - Follow the installation instructions in the Quick Start Guide document for these platforms. ## Docker Compatibility - Docker 25.0+ is required for all features. - Docker 28.3.0+ is required to use the standardized `--gpus` flag for AMD GPU selection. #### NOTE **Docker Desktop on Linux:** Docker Desktop on Linux is not supported for GPU workloads. See the [Troubleshooting](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/troubleshooting.html) guide for details. #### IMPORTANT **Rootless mode is not supported.** The AMD Container Runtime must be run as root. Rootless container runtimes are not supported and will result in a runtime error. ## ROCm and Driver Compatibility - ROCm 6.4.1 or newer is required to view and verify partitioned GPUs inside containers. ## Note A mismatch between ROCm and driver versions may lead to runtime failures. ## System Prerequisites The following packages and configurations are required on the host system: - **Kernel Headers** and **Extra Kernel Modules** for your running kernel. - **Docker** (preferably installed via the docker.io package or Docker’s official repositories). - **User Permissions**: - The user running containers must belong to the render and video groups. - Example: > ```bash > sudo usermod -aG render,video $USER > newgrp render && newgrp video > ``` - For RHEL 9.5, ensure the AMD Container Toolkit YUM repository is configured as described in the installation guide. ## GPU Partitioning Requirements - To use GPU partitioning, ensure your ROCm version is 6.4.1 or newer. - After any partitioning change, you must regenerate the CDI spec to reflect the new GPU topology. - The amd-smi tool can be used to inspect partition status and details from within containers. ## Important Notes - **Rootless mode is not supported.** The AMD Container Runtime requires root privileges. Running the runtime as a non-root user will fail immediately with an error. - ROCm must be installed on the host system and must match the expected version compatibility with your container images. - Using mismatched amdgpu driver and runtime versions may result in runtime errors or undefined behavior. - Ensure CDI specs are kept up to date in environments where GPU topology can change frequently (e.g., partitioned systems or multi-GPU deployments). - Always refer to the latest documentation for platform-specific installation and configuration steps. Failure to meet these requirements may result in incomplete functionality or runtime errors. For troubleshooting and advanced configuration, consult the relevant sections of the documentation. --- Source: https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/quick-start-guide.html # Quick Start Guide This guide walks you through installing the AMD Container Toolkit, configuring Docker for AMD GPUs, and running GPU workloads with the AMD container runtime. The steps are tailored for ease of use, production-readiness, and compatibility across AMD Instinct GPU-enabled systems. # Installing the Container Toolkit ## Prerequisites Before installing the AMD Container Toolkit, ensure the following dependencies are installed. - **Docker or a CDI-compatible runtime:** : - The toolkit is designed to work with Docker or any CDI-compatible container runtime. Ensure you have Docker (or your chosen runtime) installed on your system. - Docker version 25.0 or newer is required for all features. #### NOTE Docker Desktop on Linux is not supported for GPU workloads; see [Troubleshooting](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/troubleshooting.html) to know more. #### NOTE > - The Container Device Interface (CDI) format, used by modern container runtimes to abstract and expose GPUs, is not supported in older Docker versions. > - Without Docker 25+, CDI functionality such as dynamic device enumeration and CDI-style run commands will not work as intended. - Docker version 28.3.0 or newer is required to use the standardized `--gpus` flag for AMD GPU selection. ```bash sudo apt-get install docker.io ``` You can verify your Docker version using: ```bash docker --version ``` If you are on an earlier Docker version, please upgrade to at least Docker 25 before proceeding with toolkit configuration and GPU-based workloads. - **ROCm:** - ROCm 6.4.1 or newer is required to view and verify partitioned GPUs inside containers. - **Other container runtimes:** If you plan to use a container runtime other than Docker, ensure your runtime version supports device injection through CDI. **jq** - Required during uninstallation to parse configuration settings cleanly. ```bash sudo apt-get install jq ``` ## Step 1: Update System and Group Settings - Update your system: ```bash sudo apt update ``` - Add your user to the required groups for GPU device access: ```bash sudo usermod -a -G render,video $LOGNAME ``` ## Step 2: Install the AMDGPU Driver - Refer to the latest ROCm documentation for driver installation here, [ROCm Install Quick Start](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/quick-start.html). - Download the AMDGPU driver installer package from the [Radeon Repository](https://repo.radeon.com/amdgpu-install). - Install the downloaded package. - Load the driver. ```bash #Example (for Ubuntu 22.04, ROCm 6.3.4) wget https://repo.radeon.com/amdgpu-install/6.3.4/ubuntu/jammy/amdgpu-install_6.3.60304-1_all.deb sudo apt install ./amdgpu-install_6.3.60304-1_all.deb sudo apt update amdgpu-install --usecase=dkms sudo modprobe amdgpu ``` ## Step 3: Configure Repositories - Install required dependencies: ```bash sudo apt update sudo apt install vim wget gpg ``` - Create keyrings directory: ```bash sudo mkdir --parents --mode=0755 /etc/apt/keyrings ``` - Install GPG keys and repository links: ```bash wget https://repo.radeon.com/rocm/rocm.gpg.key -O - | gpg --dearmor | sudo tee /etc/apt/keyrings/rocm.gpg > /dev/null ``` - Add the AMD Container Toolkit repository. ```bash echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/amd-container-toolkit/apt/ $(. /etc/os-release && echo $VERSION_CODENAME) main" | sudo tee /etc/apt/sources.list.d/amd-container-toolkit.list ``` - Update package index and install the toolkit: ```bash sudo apt update ``` RHEL 9.5: ```bash tee --append /etc/yum.repos.d/amd-container-toolkit.repo <` to request a specific number of GPUs (e.g., `--gpus 2`) - Use `--gpus device=` to request exact GPU indices (e.g., `device=1,2,3`) ## GPU Partitioning: Enabling Fine-Grained Resource Allocation GPU partitioning empowers users to divide a single physical GPU into multiple logical units, each of which can be independently accessed and managed within containerized workloads. This capability is essential for fine-grained control over GPU resources, enabling scenarios such as workload isolation, resource sharing, and maximizing GPU utilization within containerized environments. Starting with version **1.1.0**, the AMD Container Toolkit introduces full support for **GPU partitioning** #### NOTE Partitioned GPUs behave identically to physical GPUs within containers. Applications and monitoring tools like rocm-smi or amd-smi will detect and report them as separate devices. ### Partitioning Schemes and Access With the AMD Container Toolkit, you can apply various partitioning schemes to your GPUs. Once partitioned, each logical GPU appears to the container runtime as a distinct device, indistinguishable from a standard, unpartitioned GPU. This allows you to allocate specific GPU partitions to different containers, optimizing performance and isolation. This functionality is particularly useful in multi-tenant or resource-constrained environments where full GPU allocation is not necessary. ### Regenerating and Validating CDI Specifications Whenever you modify GPU partitioning on your system, regenerate and validate the CDI specification so the container runtime sees the current GPU topology. See the **Generating CDI Specifications** and **Validating CDI Specifications** sections in the [CDI guide](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/cdi-guide.html). ### Inspecting GPU Partition Status You can use the amd-smi tool inside your container to inspect the status of each GPU, determine whether it is partitioned or unpartitioned, and view details about the partitioning scheme in use. ```bash docker run --rm --runtime=amd -e AMD_VISIBLE_DEVICES=all rocm/dev-ubuntu-24.04 amd-smi ``` ### Selecting GPUs and Partitions Partitioning can result in a large number of logical GPUs on your system. To simplify device selection, the AMD Container Toolkit supports specifying a range or set of GPUs using the AMD_VISIBLE_DEVICES environment variable. For example: ```bash docker run --rm --runtime=amd -e AMD_VISIBLE_DEVICES=0-3,8,17-20,52-54 rocm/dev-ubuntu-24.04 amd-smi ``` This command grants the container access to GPUs 0 through 3, 8, 17 through 20, and 52 through 54. The range specifier is especially useful for efficiently targeting all partitions within specific physical GPUs, as partitions are typically numbered contiguously. ![Example output of rocm-smi showing partitioned GPUs inside a container](container-runtime/images/rocm-smi-partitioned.png) #### NOTE To view and verify partitioned GPUs inside containers, ensure you are using ROCm version 6.4.1 or newer. By leveraging GPU partitioning, you can achieve fine-grained resource allocation, improved workload isolation, and greater flexibility in deploying GPU-accelerated containers across your infrastructure. ## Uninstallation Guide To remove the amd-container-toolkit, you must have jq installed. The uninstallation script relies on it to parse configuration files. ```bash sudo apt-get install jq ``` Then proceed with the removal: ```bash sudo apt-get remove --purge amd-container-toolkit ``` If you encounter issues, inspect the logs: ```bash sudo journalctl -u apt sudo tail -f /var/log/amd-container-runtime.log ``` If you continue to face errors, you may need to force the removal: ```bash sudo dpkg --remove --force-all amd-container-toolkit sudo apt-get autoremove ``` --- Source: https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/cdi-guide.html # Container Device Interface ## Overview The [Container Device Interface](https://github.com/cncf-tags/container-device-interface) (CDI) is a standardized specification for exposing specialized hardware devices, such as AMD GPUs, to containers in a runtime-agnostic manner. This works consistently across different container runtimes. CDI eliminates the need for runtime-specific hooks or shims, like `amd-container-runtime`, by allowing container runtimes to natively understand and inject device resources into containers. The `amd-ctk` tool provides commands to generate and manage CDI specifications for AMD GPU devices on your system. ## Prerequisites Before using CDI with AMD GPUs, ensure: * AMD GPU drivers are properly installed on the host system * The `amd-ctk` tool is installed * Your container runtime supports CDI ## Generating CDI Specifications To generate a CDI specification for AMD GPUs on your system, run: ```bash sudo amd-ctk cdi generate ``` This command: * Scans the system for available AMD GPU devices * Creates a CDI specification file at `/etc/cdi/amd.json` * Defines device nodes, mount points, and environment variables needed for each GPU **Custom Output Location** To generate the specification in a different location, use the `--output` flag: ```bash amd-ctk cdi generate --output /path/to/custom/amd.json ``` ## Validating CDI Specifications To verify that your CDI specification matches the actual GPU hardware on the system, run: ```bash sudo amd-ctk cdi validate ``` This command: * Reads the CDI specification from `/etc/cdi/amd.json` * Scans the system for available AMD GPU devices * Verifies that the devices defined in the specification accurately reflect the hardware present on the host **Custom Specification Path** To validate a specification at a different location, use the `--path` flag: ```bash amd-ctk cdi validate --path /path/to/custom/amd.json ``` #### NOTE The `amd-ctk` tool requires appropriate permissions to read and write CDI specification files. When operating on the default location (`/etc/cdi`), it requires elevated privileges, hence `sudo` is typically needed. If you want to operate on a different user-owned location (using the `--output` or `--path` flags for generation or validation respectively), `sudo` can be omitted, provided the user has necessary read/write permissions for that location. When using a custom output location, ensure your container runtime is configured to read CDI specifications from that directory. Most runtimes default to `/etc/cdi` and `/var/run/cdi`. #### IMPORTANT Regenerate the CDI specification whenever you: * Add or remove GPU devices * Modify GPU partitioning or configuration ## Running Workloads with CDI Once your CDI specification is generated and validated, you can run containers with GPU access using the `--device amd.com/gpu=` pattern. For examples, see the [Running Workloads](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/running-workloads.html) guide. ## Troubleshooting ### Containers Cannot Access GPUs If containers do not see the expected GPU devices: 1. **Validate the specification:** ```bash sudo amd-ctk cdi validate ``` If the validation fails, it indicates a mismatch between the CDI specification and the actual hardware. You may need to regenerate the specification in such cases. 2. **Verify runtime configuration:** Ensure your container runtime is configured to read CDI specifications from the directory containing `amd.json`. Check the runtime’s CDI configuration settings. 3. **Check file permissions:** ```bash ls -l /etc/cdi/amd.json ``` The file should be readable by the container runtime process. If you’re using a custom location, ensure the permissions allow the runtime to access it. 4. **Regenerate if hardware changed:** If you’ve added, removed, or reconfigured GPUs, regenerate the specification: ```bash sudo amd-ctk cdi generate ``` 5. **Verify device names:** Ensure you’re using the correct CDI device names (e.g., `amd.com/gpu=0`) while requesting devices. ### Validation Errors If `amd-ctk cdi validate` reports errors: * Check that GPU devices are properly detected by the system (verify with `rocm-smi`, `amd-smi` or similar tools) * Ensure GPU drivers are correctly installed * Regenerate the specification to reflect the current system state * Use the `--debug` flag for verbose output to help diagnose the issue: ```bash amd-ctk --debug cdi validate ``` --- Source: https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/running-workloads.html # Running Workloads This page describes how to use the AMD Container Toolkit to run GPU-accelerated workloads through different container runtimes and CLIs. You can inject AMD GPUs into containers in two ways: via **CDI specs** (recommended) or via the **amd-container-runtime**. ## Through CDI The [Container Device Interface (CDI)](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/cdi-guide.html) lets you expose AMD GPUs to containers in a runtime-agnostic way. Once a CDI specification is generated and available (e.g. at `/etc/cdi/amd.json`), any CDI-aware runtime and CLI can inject GPUs using the `--device amd.com/gpu=` pattern. ### Prerequisites Ensure CDI specs are set up on your system. Refer to the [CDI guide](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/cdi-guide.html) for details. #### Docker Use `--device amd.com/gpu=`. You do **not** need `--runtime=amd` when using CDI. ```bash docker run --rm --device amd.com/gpu=all rocm/rocm-terminal rocm-smi ``` #### NOTE Starting with Docker **29.3.0**, you can also use `--gpus ` for AMD GPUs with CDI, without `--runtime=amd`. `` can be `all`, a count (e.g. `2`), or specific devices (e.g. `'"device=0,1,2"'`). ```bash docker run --rm --gpus all rocm/rocm-terminal rocm-smi ``` #### Podman ```bash podman run --rm --device amd.com/gpu=all rocm/rocm-terminal rocm-smi ``` #### NOTE Starting with Podman **6.0.0**, you can also use `--gpus ` for AMD GPUs with CDI. `` can be `all` to request all GPUs. ```bash podman run --rm --gpus all rocm/rocm-terminal rocm-smi ``` #### NOTE To access the GPU inside the container, the process must run under the video and render groups. When running in rootless mode, ensure the user starting the Podman container is a member of these groups on the host, and use the `--group-add keep-groups` flag to pass these supplementary groups to the container process. #### nerdctl nerdctl works with containerd and supports CDI via `--device`. ```bash nerdctl run --rm --device amd.com/gpu=all rocm/rocm-terminal rocm-smi ``` #### NOTE Starting with nerdctl **2.3.0**, you can also use `--gpus ` for AMD GPUs with CDI. `` can be `all` or a count (e.g. `2`). ```bash nerdctl run --rm --gpus all rocm/rocm-terminal rocm-smi ``` #### ctr ctr is containerd’s native CLI and supports CDI via `--device`. ```bash ctr run --rm --device amd.com/gpu=all docker.io/rocm/rocm-terminal:latest mycontainer rocm-smi ``` #### NOTE Starting with containerd **2.3.0**, `ctr` also supports the `--gpus` flag for AMD GPUs with CDI. GPUs are requested by index (e.g. `--gpus 0`). ```bash ctr run --rm --gpus 0 docker.io/rocm/rocm-terminal:latest mycontainer rocm-smi ``` #### Requesting specific GPUs To request specific GPUs instead of all, use `--device amd.com/gpu=` with the available entry for the corresponding GPU(s) on your machine. List valid entries with: ```bash amd-ctk cdi list ``` Example output: ```text Found 2 AMD GPU devices amd.com/gpu=all amd.com/gpu=0 /dev/dri/renderD128 amd.com/gpu=1 /dev/dri/renderD129 ``` Use the listed device names (e.g. `all`, `0`, `1`) as `` in the CLI commands above. #### NOTE nerdctl and ctr use the containerd backend; Docker and Podman use their own runtimes. All of the above rely on the same CDI spec (e.g. `/etc/cdi/amd.json`) and `amd-ctk cdi list` for `` values. ## Through amd-container-runtime The **amd-container-runtime** is a custom OCI runtime that injects AMD GPUs into containers. At this time it is supported only with **Docker**. For setup (registering the runtime and restarting Docker), see the [Quick Start Guide](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/quick-start-guide.html). **All GPUs:** ```bash docker run --rm --runtime=amd -e AMD_VISIBLE_DEVICES=all rocm/rocm-terminal rocm-smi ``` For particular GPUs, use exact GPU indices, ranges, or UUIDs with `AMD_VISIBLE_DEVICES`: ```bash docker run --rm --runtime=amd -e AMD_VISIBLE_DEVICES=0,1 rocm/rocm-terminal rocm-smi docker run --rm --runtime=amd -e AMD_VISIBLE_DEVICES=0-3,5 rocm/rocm-terminal rocm-smi docker run --rm --runtime=amd -e AMD_VISIBLE_DEVICES=0xEF2C1799A1F3E2ED rocm/rocm-terminal rocm-smi ``` Use `amd-ctk gpu list` to discover available GPUs and their UUIDs: ```bash amd-ctk gpu list ``` Example output: ```text Found 2 AMD GPU devices --------------------------------------------------------------------------- GPU Id UUID DRM Devices --------------------------------------------------------------------------- 0 0xEF2C1799A1F3E2ED /dev/dri/renderD128 1 0x1234567890ABCDEF /dev/dri/renderD129 ``` #### NOTE Docker 28.3.0+ supports the standardized `--gpus` flag (e.g. `--gpus all` or `--gpus device=0,1`) as an alternative to `-e AMD_VISIBLE_DEVICES=all`. For setup and installation, see the [Quick Start Guide](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/quick-start-guide.html). For troubleshooting, see the [Troubleshooting](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/troubleshooting.html) guide. --- Source: https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/framework-integration.html # Framework Integration The AMD Container Toolkit is framework-agnostic but works seamlessly with popular machine learning, HPC, and AI frameworks that require GPU access, including: - TensorFlow (ROCm builds) - PyTorch (ROCm builds) - ONNX Runtime - OpenMPI + ROCm - Custom AI/ML workflows The examples below use [CDI](https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/cdi-guide.html) device notation (`--device amd.com/gpu=`). Ensure a CDI specification has been generated before running these commands. ## TensorFlow Run ROCm-enabled TensorFlow with a single GPU: ```bash docker run --rm --device amd.com/gpu=0 tensorflow/tensorflow:rocm-latest ``` Or with all available GPUs: ```bash docker run --rm --device amd.com/gpu=all tensorflow/tensorflow:rocm-latest ``` ## PyTorch Use ROCm-enabled PyTorch containers: ```bash docker run --rm --device amd.com/gpu=all rocm/pytorch:latest ``` ## Triton Inference Server Serving models with Triton using AMD GPUs is supported by adapting container images for ROCm: ```bash docker run --rm --device amd.com/gpu=all ``` ## Best Practices - Always use container images tested against the matching ROCm version. - Prefer CDI device notation (`--device amd.com/gpu=`) for portability across container runtimes. - Use `amd-ctk cdi list` to discover available device entries for multi-GPU setups. --- Source: https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/troubleshooting.html # Troubleshooting The AMD Container Toolkit is designed to integrate smoothly into Docker-based environments. However, issues may arise due to system configurations, driver installations, or runtime settings. This guide aims to provide detailed, step-by-step troubleshooting methods to identify and resolve common issues effectively. ## Common Issues: ## 1. **Driver Not Loaded** If the AMD GPU driver is not detected, verify that the amdgpu module is loaded: ```bash lsmod | grep amdgpu ``` If the module is not present, attempt to load it manually: ```bash sudo modprobe amdgpu ``` If you encounter errors, check the kernel logs for driver loading issues: ```bash dmesg | grep amdgpu ``` This will provide information about any problems during the driver initialization. ## 2. **Permission Denied Errors** If GPU devices are not visible inside containers: - Verify GPU accessibility using rocm-smi outside the container. - Ensure the user belongs to the following groups: - render - video Verify your group membership: ```bash groups $USER ``` If you are not a member, add yourself to the necessary groups: ```bash sudo usermod -a -G render,video $USER ``` **Note:** Log out and back in for the changes to take effect. ## 3. **Docker Daemon Restart Failure** If Docker fails to restart after configuring the AMD runtime, inspect the Docker logs: ```bash sudo journalctl -u docker ``` Look for errors related to: - Container runtime conflicts - GPU device issues - Improper /etc/docker/daemon.json configuration Verify that the runtime path is correctly set for AMD: ```bash cat /etc/docker/daemon.json ``` ## 4. **Runtime Configuration Issues** If Docker does not recognize the AMD runtime, validate the Docker configuration: ```bash cat /etc/docker/daemon.json ``` Ensure the runtime is set correctly: ```json { "runtimes": { "amd": { "path": "/usr/bin/amd-container-runtime", "runtimeArgs": [] } } } ``` If the configuration is missing or incorrect, regenerate it and restart Docker: ```bash sudo amd-ctk configure runtime sudo systemctl restart docker ``` ## 5. **CDI Specification Not Applied** If Docker does not recognize the GPU under CDI specifications, regenerate the CDI configuration: ```bash sudo amd-ctk cdi generate --output=/etc/cdi/amd.json ``` Check the integrity of the generated specification: ```bash cat /etc/cdi/amd.json ``` If issues persist, restart Docker: ```bash sudo systemctl restart docker ``` ## 6. **Docker Desktop and /dev/kfd Access** Docker Desktop on Linux is not supported for GPU workloads. You may see: ```text docker: Error response from daemon: error gathering device information while adding custom device "/dev/kfd": no such file or directory ``` **Why:** Docker Desktop on Linux runs the Docker daemon inside a VM (or similar isolated context). That VM does not have the host’s `/dev/kfd` and `/dev/dri` devices mounted, so containers started by that daemon cannot access them. **Workaround:** Install Docker via the `docker.io` package or Docker’s official repository so the daemon runs on the host and can expose these devices to containers. Alternatively, quit Docker Desktop and use Docker installed on the host. This applies to any run that relies on host GPU devices (e.g. `docker run --device=/dev/kfd --device=/dev/dri ...` or `docker run --runtime=amd -e AMD_VISIBLE_DEVICES=...`). ## Log File Reference The AMD container runtime (`amd-container-runtime`) logs events and errors to the following location: > **/var/log/amd-container-runtime.log** You can view logs in real-time using: ```bash sudo tail -f /var/log/amd-container-runtime.log ``` This log captures detailed interactions between Docker and the AMD container runtime, including: - Runtime initialization - GPU device injection and allocation - OCI specification modifications - Exclusive GPU enforcement errors If a container fails to start with the AMD runtime, this log will contain the specific error (e.g. `GPUs [0] are exclusive and already in use`), even when Docker only shows a generic runtime failure message. #### NOTE The `amd-ctk` CLI tool prints errors directly to the terminal (not to a log file). For verbose debug output from `amd-ctk`, use the `--debug` (or `-d`) flag: ```bash amd-ctk --debug gpu-tracker status amd-ctk --debug cdi validate ``` This prints debug-level messages to stderr, which can help diagnose GPU enumeration, tracker state, or CDI specification issues. ## Diagnostic Commands - **List Available Devices:** > ```bash > amd-ctk cdi list > ``` - **Check Runtime Configuration:** > ```bash > cat /etc/docker/daemon.json > ``` - **Inspect Docker Logs:** > ```bash > sudo journalctl -u docker > ``` ## Next Steps If the above steps do not resolve your issue: - Validate your amdgpu driver installation with: ```bash rocminfo ``` - Verify GPU accessibility with: ```bash rocm-smi ``` - Consult the official AMD Container Toolkit documentation or reach out to the support community for advanced troubleshooting. --- Source: https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/migration-guide.html # Migration Guide: NVIDIA to AMD - Migrating from the NVIDIA Container Toolkit to the AMD Container Toolkit is a streamlined process that enables developers to leverage AMD Instinct GPUs in containerized environments with minimal modifications. This guide provides step-by-step instructions to make this transition smooth and efficient. - Migrating container workflows from NVIDIA GPUs (using nvidia-docker) to AMD Instinct GPUs (using AMD Container Toolkit) involves the following key changes: ## Step 1: Environment Variable Updates NVIDIA GPUs are typically managed through the environment variable `NVIDIA_VISIBLE_DEVICES`. In the AMD Container Toolkit, this is replaced with: ```bash export AMD_VISIBLE_DEVICES=all # To enable all GPUs export AMD_VISIBLE_DEVICES=0,1 # To enable specific GPUs ``` The variable syntax remains consistent, but the prefix changes to `AMD`. This environment variable is recognized by the AMD runtime to expose GPUs to your container workloads. ## Step 2: Update Runtime Configuration NVIDIA’s container runtime is identified as `nvidia` in Docker commands. For AMD, the runtime flag needs to be updated to: ```bash sudo docker run --rm --runtime=amd ``` To set AMD as the default runtime: ```bash sudo amd-ctk runtime configure --runtime=docker --set-as-default ``` If you previously used: ```bash sudo docker run --rm --runtime=nvidia ``` You would now use the equivalent: ```bash sudo docker run --rm --runtime=amd ``` ## Step 3: Command Line Utility Replacement NVIDIA provides `nvidia-ctk` for container configurations and runtime settings. In the AMD ecosystem, this is replaced with: ```bash amd-ctk ``` For example, listing all available GPUs using NVIDIA would look like: ```bash nvidia-ctk list ``` The AMD equivalent: ```bash amd-ctk list ``` You can also generate CDI specifications with: ```bash amd-ctk cdi generate --output=/etc/cdi/amd.json ``` ## Step 4: Container Images Containers built for NVIDIA GPUs often rely on CUDA-based images. AMD’s container toolkit is designed to work seamlessly with ROCm-enabled images: - TensorFlow: `tensorflow/tensorflow:rocm-latest` - PyTorch: `rocm/pytorch:latest` - Triton Inference Server: `rocm/tritonserver:latest` For example, running PyTorch with AMD GPUs: ```bash sudo docker run --rm --runtime=amd -e AMD_VISIBLE_DEVICES=all rocm/pytorch:latest ``` ## Step 5: Framework Adjustments To fully leverage AMD Instinct GPUs, frameworks like TensorFlow and PyTorch must use their ROCm-enabled versions. This ensures compatibility and optimized performance for machine learning workloads. ## Compatibility Notes - Ensure Docker is version 25 or above for CDI compatibility. - Some CUDA-specific applications may require minor modifications. - ROCm supports a majority of ML frameworks, but always validate with your application stack. ## Comparison: #### Feature Comparison: NVIDIA Docker vs AMD Container Toolkit | **Feature** | **NVIDIA Docker** | **AMD Container Toolkit** | |------------------------|---------------------------------------------------------------------------------------|---------------------------------------------------------------------------------| | GPU Enumeration | `nvidia-smi` - Lists available GPUs and their statuses. | `rocm-smi` - Lists AMD GPUs and exposes detailed hardware information. | | Container Runtime | `nvidia-container-runtime` - Manages container interactions with NVIDIA GPUs. | `amd-container-runtime` - Integrates AMD Instinct GPUs seamlessly with Docker. | | Environment Variable | `NVIDIA_VISIBLE_DEVICES` - Specifies which NVIDIA GPUs are visible inside containers. | `AMD_VISIBLE_DEVICES` - Specifies which AMD GPUs are visible inside containers. | | Framework Images | NVIDIA-specific images optimized for CUDA. | ROCm-optimized images designed for AMD GPUs. | | TensorFlow Support | CUDA TensorFlow - Supports TensorFlow operations on NVIDIA GPUs. | ROCm TensorFlow - Optimized TensorFlow builds for AMD GPUs. | | PyTorch Support | CUDA PyTorch - Optimized for NVIDIA architectures. | ROCm PyTorch - Optimized for AMD Instinct architectures. | | Configuration Toolkit | `nvidia-ctk` - NVIDIA’s CLI for runtime configuration. | `amd-ctk` - AMD’s CLI for Docker runtime integration and device management. | | Default Docker Runtime | `nvidia runtime` - Configures Docker to use NVIDIA GPUs by default. | `amd runtime` - Configures Docker to use AMD GPUs by default. | ## Testing and Validation After migration, it is crucial to validate workloads with: ```bash sudo docker run --rm --runtime=amd -e AMD_VISIBLE_DEVICES=all rocm/rocm-terminal rocm-smi ``` Ensure that all intended GPUs are detected and functioning as expected. ## Next Steps Once migration is complete: - Update your CI/CD pipelines to reflect runtime changes. - Adjust Dockerfiles if specific runtime flags were set for NVIDIA. - Monitor GPU usage using tools like `rocm-smi` and `amd-ctk list`. This completes the NVIDIA to AMD migration process, enabling you to leverage the full power of AMD Instinct GPUs in containerized workflows. By following this migration guide, users can rapidly transition their GPU workloads to AMD Instinct platforms. --- Source: https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/developer-guide.html # Developer Guide The AMD Container Toolkit can be built and customized from the source code. This guide provides step-by-step instructions to set up your environment, install necessary dependencies, and build the toolkit packages for deployment. ## System Preparation To successfully build the AMD Container Toolkit from source, you need the following build dependencies installed: 1. **Build Essentials** ```bash sudo apt update sudo apt install build-essential cmake pkg-config ``` 1. **Kernel Headers and Modules** (Required for DKMS module compilation) ```bash sudo apt install "linux-headers-$(uname -r)" "linux-modules-extra-$(uname -r)" **Note:** These are necessary for integrating the ROCm GPU drivers with your Linux kernel. Ensure the version matches your current kernel. ``` 1. **Development Libraries** ```bash sudo apt install libssl-dev libelf-dev libudev-dev ``` 1. **Docker CLI and Daemon** (Required for container runtime testing) ```bash sudo apt install docker.io # Start Docker and enable it on boot sudo systemctl enable docker sudo systemctl start docker ``` 1. **ROCm Dependencies** Ensure that ROCm is properly installed and configured: ```bash sudo apt update sudo apt install rocm-dev rocm-utils ``` ## Building the Toolkit To build Debian packages: ```bash make pkg-deb ``` The generated .deb files will be available under the bin/ directory. These can be installed using: ```bash sudo dpkg -i bin/amd-container-toolkit-.deb ``` To build RPM packages: ```bash make pkg-rpm ``` The RPM packages will also be located in the bin/ directory. For installation: ```bash sudo rpm -i bin/amd-container-toolkit-.rpm ``` ## Contribution Guidelines Contributions to the AMD Container Toolkit are welcomed and encouraged. Follow these guidelines to ensure smooth collaboration: 1. **Coding Standards**: - Adhere to the coding conventions outlined in the developer README. - Maintain clear, concise, and well-structured code. 2. **Testing Requirements**: - All changes must be tested with Docker and ROCm environments. - Use amd-ctk list and Docker integration tests to validate GPU access. 3. **Pull Request Requirements**: - Include detailed descriptions of changes. - Reference any related issues or bug fixes. - Attach testing logs or screenshots if applicable. ## Advanced Configuration For developers looking to extend the runtime or integrate custom modules, make sure you: - Rebuild the kernel modules if kernel headers are updated. ```bash sudo dkms install -m amdgpu -v ``` - Restart Docker to load new configurations: ```bash sudo systemctl restart docker ``` ## Next Steps - Deploy the built packages in a development environment for further testing. - Validate compatibility with your ROCm-based applications. - Document any discrepancies or runtime anomalies. By following these steps, you will have a robust, production-ready build of the AMD Container Toolkit, optimized for high-performance containerized workloads. --- Source: https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/docker-compose.html # Docker Compose Usage The AMD Container Toolkit can be used with Docker Compose, enabling GPU access in multi-container applications. ## Prerequisites Before using this guide, ensure you have: 1. Completed the AMD Container Toolkit installation as described in the Quick Start Guide. 2. Docker Compose installed (v2 or higher recommended). 3. Docker engine properly configured with the AMD runtime. ### Example Docker Compose Configuration Below is a basic example of a Docker Compose file configured to use AMD GPUs: ```yaml version: '3' services: pytorch: image: rocm/pytorch runtime: amd environment: - AMD_VISIBLE_DEVICES=all command: python -c "import torch; print('GPU available:', torch.cuda.is_available()); print('Number of GPUs:', torch.cuda.device_count())" ``` The key elements in this configuration are: 1. **runtime: amd** - Specifies that the AMD Container Runtime should be used for this service. 2. **AMD_VISIBLE_DEVICES** - Controls which GPUs are visible to the container. ## GPU Visibility Control Control GPU visibility through environment variables. The AMD_VISIBLE_DEVICES variable can be set to: - **AMD_VISIBLE_DEVICES=all** - Makes all GPUs visible to the container - **AMD_VISIBLE_DEVICES=0,1** - Makes only GPU indices 0 and 1 visible - **AMD_VISIBLE_DEVICES=none** - Disables GPU visibility When using Docker Compose to orchestrate multiple containers, you can specify the GPU resources for each service independently. For example, if you have a training service and an inference service, you can assign different GPUs to each: ```yaml version: '3' services: training: image: rocm/tensorflow runtime: amd environment: - AMD_VISIBLE_DEVICES=0 inference: image: rocm/pytorch runtime: amd environment: - AMD_VISIBLE_DEVICES=1 ``` ## Converting Existing Docker Compose Files If you have existing Docker Compose files using NVIDIA GPUs or other GPU setups, use these guidelines to migrate to the AMD Container Toolkit: 1. Replace runtime specifications: ```diff - runtime: nvidia + runtime: amd ``` 2. Update environment variables: ```diff - NVIDIA_VISIBLE_DEVICES: all + AMD_VISIBLE_DEVICES: all - HIP_VISIBLE_DEVICES: 0,1 + AMD_VISIBLE_DEVICES: 0,1 ``` 3. Remove explicit device mappings if present (not needed with AMD Container Toolkit): ```diff services: myservice: # Remove these lines - devices: - - /dev/kfd - - /dev/dri ``` --- Source: https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/enroot-pyxis-installation.html # Enroot and Pyxis [Enroot](https://github.com/NVIDIA/enroot) and [Pyxis](https://github.com/NVIDIA/pyxis) are tools created to run containerized AI/HPC workloads on SLURM. These tools can now be used on a SLURM cluster with AMD GPUs to make them run efficiently and achieve isolation for these GPUs. Traditional runtimes like Docker/Podman bring additional overhead such as daemons, root privileges and extra storage layers. With Enroot, users can convert Docker images into a simple unpacked filesystem tree and run containers as a regular Linux process. Also, with Enroot and Pyxis, each job is granted exclusive GPU device files which prevents jobs from accidentally accessing the same GPU device. This guide provides the steps to install enroot/pyxis on a SLURM cluster as well as examples to run containerized images isolating specific AMD GPUs on Ubuntu. ## Installation Pre-requisites: Make sure SLURM is already installed and the cluster is up and running.\\\\ Since GPUs are used with enroot and pyxis, the /etc/slurm/gres.conf should be configured with the correct renderD IDs. Sample gres.conf file : ```bash Name=gpu Type=rocm File=/dev/dri/renderD128 Name=gpu Type=rocm File=/dev/dri/renderD136 Name=gpu Type=rocm File=/dev/dri/renderD144 Name=gpu Type=rocm File=/dev/dri/renderD152 Name=gpu Type=rocm File=/dev/dri/renderD160 Name=gpu Type=rocm File=/dev/dri/renderD168 Name=gpu Type=rocm File=/dev/dri/renderD176 Name=gpu Type=rocm File=/dev/dri/renderD184 ``` Enroot Installation: ```bash #Check requirements curl -fSsL -O https://github.com/NVIDIA/enroot/releases/download/v3.5.0/enroot-check_3.5.0_$(uname -m).run chmod +x enroot-check_*.run ./enroot-check_*.run --verify ./enroot-check_*.run ``` Install enroot through a package ```bash arch=$(dpkg --print-architecture) curl -fSsL -O https://github.com/NVIDIA/enroot/releases/download/v3.5.0/enroot_3.5.0-1_${arch}.deb curl -fSsL -O https://github.com/NVIDIA/enroot/releases/download/v3.5.0/enroot+caps_3.5.0-1_${arch}.deb sudo apt install -y ./*.deb ``` Validate Enroot installation ```bash enroot import docker://rocm/pytorch:latest enroot create rocm+pytorch+latest.sqsh enroot start rocm+pytorch+latest rocm-smi ENROOT_RESTRICT_DEV=y enroot start rocm+pytorch+latest rocm-smi ``` Reference: https://github.com/NVIDIA/enroot/blob/master/doc/installation.md ## Steps to build and install Pyxis Install the following packages ```bash sudo apt update sudo apt install -y devscripts sudo apt install -y debhelper ``` Create a deb package ```bash git clone https://github.com/NVIDIA/pyxis cd pyxis git checkout v0.20.0 make orig CPPFLAGS="-I/usr/local/slurm-24.05.5.1/include" LDFLAGS="-L/usr/local/slurm-24.05.5.1/lib" make deb ``` After this step, nvslurm-plugin-pyxis_0.20.0-1_amd64.deb will be created in the same directory. ## Steps to install pyxis deb on all the compute nodes and also the slurm head-node Install the same pyxis deb package on the headnode and all the compute nodes. While installing pyxis on the headnode/controller node, it will throw error that enroot is not installed but we can ignore this error since we need not have enroot on the head-bode. ```bash sudo dpkg -i ./nvslurm-plugin-pyxis_0.20.0-1_amd64.deb sudo mkdir /etc/slurm/plugstack.conf.d sudo ln -s /usr/share/pyxis/pyxis.conf /etc/slurm/plugstack.conf.d/pyxis.conf sudo touch /etc/slurm/plugstack.conf echo "include /etc/slurm/plugstack.conf.d/*" | sudo tee -a /etc/slurm/plugstack.conf ``` Restart slurmd on the compute ```bash sudo systemctl restart slurmd ``` Restart slurmd and slurmctld on all nodes ```bash sudo systemctl restart slurmd sudo systemctl restart slurmctld ``` ### Test with SLURM Following shows 4 isolated AMD GPUs running a containerized image rocm/pytorch ```bash ubuntu@node-4:~$ srun --gres=gpu:4 --container-image=docker://rocm/pytorch:latest rocm-smi pyxis: importing docker image: docker://rocm/pytorch:latest pyxis: imported docker image: docker://rocm/pytorch:latest ============================================ ROCm System Management Interface ============================================ ====================================================== Concise Info ====================================================== Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU% (DID, GUID) (Junction) (Socket) (Mem, Compute, ID) ========================================================================================================================== 0 2 0x74a1, 28851 45.0°C 137.0W NPS1, SPX, 0 123Mhz 900Mhz 0% auto 750.0W 0% 0% 1 3 0x74a1, 43178 41.0°C 133.0W NPS1, SPX, 0 124Mhz 900Mhz 0% auto 750.0W 0% 0% 2 4 0x74a1, 32898 44.0°C 133.0W NPS1, SPX, 0 124Mhz 900Mhz 0% auto 750.0W 0% 0% 3 5 0x74a1, 22683 40.0°C 136.0W NPS1, SPX, 0 124Mhz 900Mhz 0% auto 750.0W 0% 0% ========================================================================================================================== ================================================== End of ROCm SMI Log =================================================== ``` Following shows isolating 2 AMD GPUs running a containerized image rocm/pytorch ```bash ubuntu@node-4:~$ srun --gres=gpu:2 --container-image=docker://rocm/pytorch:latest rocm-smi pyxis: importing docker image: docker://rocm/pytorch:latest pyxis: imported docker image: docker://rocm/pytorch:latest ============================================ ROCm System Management Interface ============================================ ====================================================== Concise Info ====================================================== Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU% (DID, GUID) (Junction) (Socket) (Mem, Compute, ID) ========================================================================================================================== 0 2 0x74a1, 28851 45.0°C 137.0W NPS1, SPX, 0 123Mhz 900Mhz 0% auto 750.0W 0% 0% 1 3 0x74a1, 43178 41.0°C 133.0W NPS1, SPX, 0 124Mhz 900Mhz 0% auto 750.0W 0% 0% ========================================================================================================================== ================================================== End of ROCm SMI Log =================================================== ``` Following command runs a test.py script to different torch.cuda variables ```bash ubuntu@node-4:~$ srun --gres=gpu:2 --container-image=docker://rocm/pytorch:latest --container-mounts="$HOME:/home/$MY_USER" python3 /home/$MY_USER/test.py pyxis: importing docker image: docker://rocm/pytorch:latest pyxis: imported docker image: docker://rocm/pytorch:latest --- PyTorch CUDA Status --- torch.cuda.is_available(): True torch.cuda.device_count(): 2 Visible Devices (from CUDA_VISIBLE_DEVICES env var): CUDA_VISIBLE_DEVICES=0,1 Detected Devices: Device 0: AMD Instinct MI300X Capability: (9, 4) Memory (GB): 191.98 Device 1: AMD Instinct MI300X Capability: (9, 4) Memory (GB): 191.98 --- End PyTorch CUDA Status --- The following command can be used to save the image locally to use for subsequent runs ubuntu@node-4:~$ srun --gres=gpu:8 --container-image=docker://rocm/pytorch:latest --container-save=/var/lib/ubuntu/enroot/rocm+pytorch+latest.sqsh rocm-smi ubuntu@node-4:~$ srun --gres=gpu:8 --container-image=/var/lib/ubuntu/enroot/rocm+pytorch+latest.sqsh rocm-smi Following command shows the usage of --exclusive directive with 2 GPUS requested. root@head-node:~# srun --exclusive --gres=gpu:2 --container-image=./rocm+ubuntu.sqsh --pty bash root@valid-prawn:/# rocm-smi ============================================ ROCm System Management Interface ============================================ ====================================================== Concise Info ====================================================== Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU% (DID, GUID) (Junction) (Socket) (Mem, Compute, ID) ========================================================================================================================== 0 2 0x74a1, 28851 35.0°C 142.0W NPS1, SPX, 0 132Mhz 900Mhz 0% auto 750.0W 0% 0% 1 3 0x74a1, 44463 34.0°C 136.0W NPS1, SPX, 0 134Mhz 900Mhz 0% auto 750.0W 0% 0% ========================================================================================================================== ================================================== End of ROCm SMI Log =================================================== ``` ## Enroot and Pyxis with GPU partition We can use partitioned GPUs just like any other unpartitioned GPU when we use enroot and pyxis. But for this, slurm first needs to identify partitioned GPUS as the generic resources. Some config changes: 1. Add the below line to /etc/slurm/gres.conf file, so that whenever GPUS are partitioned, the slurm automatically detects the number of gres resources. AutoDetect=rsmi Example gres.conf file : ```bash AutoDetect=rsmi Name=gpu File=/dev/dri/renderD128 Name=gpu File=/dev/dri/renderD136 Name=gpu File=/dev/dri/renderD144 Name=gpu File=/dev/dri/renderD152 Name=gpu File=/dev/dri/renderD160 Name=gpu File=/dev/dri/renderD168 Name=gpu File=/dev/dri/renderD176 Name=gpu File=/dev/dri/renderD184 ``` 1. If gres is specified in the node info in /etc/slurm/slurm.conf file, make sure it specifies the correct number of GPUs for that node Eg : ```bash NodeName=localhost CPUs=160 Boards=1 SocketsPerBoard=2 CoresPerSocket=80 ThreadsPerCore=1 RealMemory=1285717 Gres=gpu:8 ``` Gres=gpu:8 , can be omitted as well if the partitions keep changing. 3. Restart slurm on both worker node and head node. Head node : ```bash sudo service slurmctld restart && sudo service slurmd restart ``` Worker node : ```bash sudo service slurmd restart ``` Now, pyxis would be able to use all the partitioned GPUS as the resources and allocate them as requested. ```bash root@node2:~# srun --gres=gpu:62 --container-image=./rocm+pytorch+latest.sqsh --pty bash root@node2:/var/lib/jenkins# python3 Python 3.12.10 | packaged by conda-forge | (main, Apr 10 2025, 22:21:13) [GCC 13.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import torch >>> torch.cuda.device_count() 62 >>> exit() ``` --- Source: https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/docker-swarm.html # Docker Swarm Integration ## Purpose Docker Swarm integration allows orchestrated GPU workloads to be deployed across multiple nodes by leveraging **GPU UUIDs** and Docker’s resource framework. ## Docker Daemon Configuration for Swarm Configure each swarm node’s Docker daemon with GPU resources in `/etc/docker/daemon.json`: ```json { "default-runtime": "amd", "runtimes": { "amd": { "path": "amd-container-runtime", "runtimeArgs": [] } }, "node-generic-resources": [ "AMD_GPU=0x378041e1ada6015", "AMD_GPU=0xef39dad16afb86ad", "GPU_COMPUTE=0x583de6f2d99dc333" ] } ``` After updating the configuration, restart the Docker daemon: ```bash sudo systemctl restart docker ``` ## Deploy GPU Enabled Services Deploy services with specific GPU requirements using docker-compose: **Using generic resources:** ```yaml # docker-compose.yml for Swarm deployment version: '3.8' services: rocm-service: image: rocm/dev-ubuntu-24.04 command: rocm-smi deploy: replicas: 1 resources: reservations: generic_resources: - discrete_resource_spec: kind: 'AMD_GPU' # Matches daemon.json key value: 1 ``` **Using environment variables:** ```yaml # docker-compose.yml for Swarm deployment with environment variable version: '3.8' services: rocm-service: image: rocm/dev-ubuntu-24.04 command: rocm-smi environment: - AMD_VISIBLE_DEVICES=all deploy: replicas: 1 ``` Deploy the service: ```bash docker stack deploy -c docker-compose.yml rocm-stack ``` --- Source: https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/gpu-tracker.html # GPU Tracker Currently, barebones Docker provides no way to track access of GPUs in containers. Additionally, by default, multiple containers in Docker can be granted access to the same GPU simultaneously. GPU Tracker is an extremely lightweight feature of AMD Container Toolkit that solves these issues. GPU Tracker state is initialized during AMD Container Toolkit installation and is by default disabled. Users can enable or disable the GPU Tracker feature by using the `enable` or `disable` CLIs. When enabled, the GPU Tracker automatically maintains the state of GPUs and the containers that they are made accessible to, only if the containers are launched and granted access to the GPUs using the `AMD_VISIBLE_DEVICES` environment variable. When the container process completes execution or is stopped, the GPU Tracker state is automatically updated to reflect GPUs released by the specific container. **NOTE:** GPU Tracker feature is currently supported only if containers are started using the `docker run` command and GPUs are made accessible in containers using the `AMD_VISIBLE_DEVICES` environment variable. If containers are started and granted access to GPUs in any other manner, GPU Tracker feature is not supported. GPU Tracker provides CLIs that can be used to control the accessibility of GPUs in containers. The accessibility of GPUs can be set to either `shared` or `exclusive`. - The `shared` accessibility indicates that the GPU can be made accessible to multiple containers simultaneously. By default, all GPUs are granted the `shared` accessibility to reflect the default Docker behavior. - The `exclusive` accessibility indicates that the GPU can be made accessible to at most one container at any point of time. GPU Tracker status can be queried at any point of time using the `status` command and reset using the `reset` CLIs. ```text > sudo amd-ctk gpu-tracker -h NAME: AMD Container Toolkit CLI gpu-tracker - GPU Tracker related commands USAGE: amd-ctk gpu-tracker [gpu-ids] [accessibility] Arguments: gpu-ids Comma-separated list of GPU IDs (comma separated list, range operator, all) accessibility Must be either 'exclusive' or 'shared' Examples: amd-ctk gpu-tracker 0,1,2 exclusive amd-ctk gpu-tracker 0,1-2 shared amd-ctk gpu-tracker all shared OR amd-ctk gpu-tracker [command] [options] COMMANDS: disable Disable the GPU Tracker enable Enable the GPU Tracker reset Reset the GPU Tracker status Show Status of GPUs help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` ## Using GPU Tracker Let us assume that the node has 4 GPUs as indicated below ```text > rocm-smi ========================================= ROCm System Management Interface ========================================= =================================================== Concise Info =================================================== Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU% (DID, GUID) (Edge) (Avg) (Mem, Compute, ID) ==================================================================================================================== 0 4 0x740f, 12261 33.0°C 42.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 300.0W 0% 0% 1 5 0x740f, 13566 38.0°C 40.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 300.0W 0% 0% 2 3 0x740f, 57300 34.0°C 42.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 300.0W 0% 0% 3 2 0x740f, 1997 38.0°C 41.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 300.0W 0% 0% ==================================================================================================================== =============================================== End of ROCm SMI Log ================================================ ``` 1. Show GPU Tracker Status: Once AMD Container Toolkit, is installed, the GPU Tracker is initialized and the status can be queried using the `status` CLI. If GPU Tracker is enabled, by default it can be seen that GPUs are granted the `shared` accessibility. ```text > amd-ctk gpu-tracker status ------------------------------------------------------------------------------------------------------------------------ GPU Id UUID Accessibility Container Ids ------------------------------------------------------------------------------------------------------------------------ 0 0xEA35F57CC80DEB35 Shared - 1 0x89CAA15875FF5A43 Shared - 2 0x6E32F10EFC982B4C Shared - 3 0x12FE4F7FDAF06B9 Shared - ``` If GPU Tracked feature is not enabled, then a message indicating this is printed. ```text > amd-ctk gpu-tracker status GPU Tracker is disabled ``` 2. Enabling GPU Tracker: GPU Tracker can be enabled using the `enable` CLI. When GPU Tracker is newly enabled, it starts tracking usage of GPUs in containers with no prior knowledge of GPUs state. If GPU Tracker is already currently enabled, then nothing happens and a message indicating this is printed. ```text > amd-ctk gpu-tracker status GPU Tracker is disabled > amd-ctk gpu-tracker enable GPU Tracker has been enabled > amd-ctk gpu-tracker enable GPU Tracker is already enabled ``` 3. Disabling GPU Tracker: GPU Tracker can be disabled using the `disable` CLI. If GPU Tracker is again enabled in the future, all the GPUs state related information will be lost. ```text > amd-ctk gpu-tracker disable GPU Tracker has been disabled > amd-ctk gpu-tracker status GPU Tracker is disabled ``` 4. Granting access to GPUs in Docker containers: If GPU Tracker is enabled before launching container, it automatically tracks the usage of GPUs in containers as indicated below. ```text > docker run --runtime=amd -itd -e AMD_VISIBLE_DEVICES=0-2 rocm/rocm-terminal bash 36b012bb34c96149a6ef5b28623e6e75cf9f71eb2b824b2c8f44e0449c7a1aa8 > docker run --runtime=amd -itd -e AMD_VISIBLE_DEVICES=1,3 rocm/rocm-terminal bash 90cb29e11e83aa3ae497c68c90e1f0894b85262188c1ef9c7284457a9bc35ffd > amd-ctk gpu-tracker status ------------------------------------------------------------------------------------------------------------------------ GPU Id UUID Accessibility Container Ids ------------------------------------------------------------------------------------------------------------------------ 0 0xEA35F57CC80DEB35 Shared 36b012bb34c96149a6ef5b28623e6e75cf9f71eb2b824b2c8f44e0449c7a1aa8 1 0x89CAA15875FF5A43 Shared 36b012bb34c96149a6ef5b28623e6e75cf9f71eb2b824b2c8f44e0449c7a1aa8 90cb29e11e83aa3ae497c68c90e1f0894b85262188c1ef9c7284457a9bc35ffd 2 0x6E32F10EFC982B4C Shared 36b012bb34c96149a6ef5b28623e6e75cf9f71eb2b824b2c8f44e0449c7a1aa8 3 0x12FE4F7FDAF06B9 Shared 90cb29e11e83aa3ae497c68c90e1f0894b85262188c1ef9c7284457a9bc35ffd > docker rm -f 36b012bb34c96149a6ef5b28623e6e75cf9f71eb2b824b2c8f44e0449c7a1aa8 36b012bb34c96149a6ef5b28623e6e75cf9f71eb2b824b2c8f44e0449c7a1aa8 > amd-ctk gpu-tracker status ------------------------------------------------------------------------------------------------------------------------ GPU Id UUID Accessibility Container Ids ------------------------------------------------------------------------------------------------------------------------ 0 0xEA35F57CC80DEB35 Shared - 1 0x89CAA15875FF5A43 Shared 90cb29e11e83aa3ae497c68c90e1f0894b85262188c1ef9c7284457a9bc35ffd 2 0x6E32F10EFC982B4C Shared - 3 0x12FE4F7FDAF06B9 Shared 90cb29e11e83aa3ae497c68c90e1f0894b85262188c1ef9c7284457a9bc35ffd ``` 5. Setting GPUs to have `exclusive` accessibility: If GPU Tracker is enabled, GPUs can be set to have exclusive access in containers. If the user tries to make GPUs exclusive when GPU Tracker is disabled, nothing happens and a message indicating that GPU Tracker is disabled is printed. ```text > amd-ctk gpu-tracker 1-3 exclusive GPUs [1 2 3] have been made exclusive > amd-ctk gpu-tracker status ------------------------------------------------------------------------------------------------------------------------ GPU Id UUID Accessibility Container Ids ------------------------------------------------------------------------------------------------------------------------ 0 0xEA35F57CC80DEB35 Shared - 1 0x89CAA15875FF5A43 Exclusive 90cb29e11e83aa3ae497c68c90e1f0894b85262188c1ef9c7284457a9bc35ffd 2 0x6E32F10EFC982B4C Exclusive - 3 0x12FE4F7FDAF06B9 Exclusive 90cb29e11e83aa3ae497c68c90e1f0894b85262188c1ef9c7284457a9bc35ffd > docker run --runtime=amd -itd -e AMD_VISIBLE_DEVICES=0-2 rocm/rocm-terminal bash d23ff3dce1839cbf8ce7ad362641ab85e80b315c319edf73b269c460e348053a docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: unable to retrieve OCI runtime error: amd-container-runtime did not terminate successfully: exit status 1 The runtime log at /var/log/amd-container-runtime.log will contain details about the failure: > grep -E "allocated|exclusive" /var/log/amd-container-runtime.log time=... level=INFO msg="amd-container-runtime GPUs allocated" gpus=[0 2] time=... level=ERROR msg="amd-container-runtime Failed to run container runtime" error="update OCI spec (add GPU devices): GPUs [1] are exclusive and already in use" > amd-ctk gpu-tracker status ------------------------------------------------------------------------------------------------------------------------ GPU Id UUID Accessibility Container Ids ------------------------------------------------------------------------------------------------------------------------ 0 0xEA35F57CC80DEB35 Shared - 1 0x89CAA15875FF5A43 Exclusive 90cb29e11e83aa3ae497c68c90e1f0894b85262188c1ef9c7284457a9bc35ffd 2 0x6E32F10EFC982B4C Exclusive - 3 0x12FE4F7FDAF06B9 Exclusive 90cb29e11e83aa3ae497c68c90e1f0894b85262188c1ef9c7284457a9bc35ffd ``` In the above example, GPUs 1,2 and 3 have been granted `exclusive` access. When a new container `d23ff3dce1839cbf8ce7ad362641ab85e80b315c319edf73b269c460e348053a` that requests access to GPUs 0,1 and 2 is launched, the following happens: - The new container is created. - The new container is granted access to GPU 0 as no container is currently using GPU 0. - GPUs 1 is already being used by container `90cb29e11e83aa3ae497c68c90e1f0894b85262188c1ef9c7284457a9bc35ffd`. Hence, the new container is not granted access to it as GPU 1 has `exclusive` accessibility. - The new container is granted access to GPU 2 as no container is currently using GPU 2 though GPU 2 has `exclusive` accessibility. - The container is not started since it has not been granted access to the required GPU resources. - The resources that have been granted to the new container are released. **NOTE:** - Even though the new container `d23ff3dce1839cbf8ce7ad362641ab85e80b315c319edf73b269c460e348053a` is not successfully started, it is still visible when we run `docker ps -a` command. ```text > docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d23ff3dce183 rocm/rocm-terminal "bash" 11 seconds ago Created funny_gagarin 90cb29e11e83 rocm/rocm-terminal "bash" 45 seconds ago Up 44 seconds practical_williams ``` This is because Docker has already created the container when the runtime errors out due to non-availability of resources. This behavior is similar to behavior exhibited by Docker when a container fails to start in any stage after the container is created in Docker. In such cases also, the container is visible in the `docker ps -a` command output with status as `Created` as depicted below. ```text > docker run -itd ubuntu incorrect_command 94f11c132e8cd0a35d05bcc8bcaf77264563998d07f6ad5c73798cf9ddd94726 docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "incorrect_command": executable file not found in $PATH: unknown. > docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 94f11c132e8c ubuntu "incorrect_command" 17 seconds ago Created elastic_ardinghelli ``` - Only GPUs that are currently not being used by more than 1 container can be set to have `exclusive` accessibility. ```text > amd-ctk gpu-tracker status ------------------------------------------------------------------------------------------------------------------------ GPU Id UUID Accessibility Container Ids ------------------------------------------------------------------------------------------------------------------------ 0 0xEA35F57CC80DEB35 Shared 8463b475b55b104b30edec8ddf6249b6214b27127106aa0ff4a8a514b856810e 1 0x89CAA15875FF5A43 Shared 90cb29e11e83aa3ae497c68c90e1f0894b85262188c1ef9c7284457a9bc35ffd 8463b475b55b104b30edec8ddf6249b6214b27127106aa0ff4a8a514b856810e 2 0x6E32F10EFC982B4C Exclusive 8463b475b55b104b30edec8ddf6249b6214b27127106aa0ff4a8a514b856810e 3 0x12FE4F7FDAF06B9 Exclusive 90cb29e11e83aa3ae497c68c90e1f0894b85262188c1ef9c7284457a9bc35ffd > amd-ctk gpu-tracker 1 exclusive GPUs [1] have not been made exclusive because more than one container is currently using it ``` 6. Setting GPUs to have `shared` accessibility: If GPU Tracker is enabled, GPUs can be set to have shared access in containers. If the user tries to make GPUs shared when GPU Tracker is disabled, nothing happens and a message indicating that GPU Tracker is disabled is printed. By default when GPU Tracker is disabled, GPUs have shared accessibility. ```text > amd-ctk gpu-tracker status ------------------------------------------------------------------------------------------------------------------------ GPU Id UUID Accessibility Container Ids ------------------------------------------------------------------------------------------------------------------------ 0 0xEA35F57CC80DEB35 Shared - 1 0x89CAA15875FF5A43 Exclusive 90cb29e11e83aa3ae497c68c90e1f0894b85262188c1ef9c7284457a9bc35ffd 2 0x6E32F10EFC982B4C Exclusive - 3 0x12FE4F7FDAF06B9 Exclusive 90cb29e11e83aa3ae497c68c90e1f0894b85262188c1ef9c7284457a9bc35ffd > amd-ctk gpu-tracker 1 shared GPUs [1] have been made shared > docker run --runtime=amd -itd -e AMD_VISIBLE_DEVICES=0-2 rocm/rocm-terminal bash a8ce87c99727107ab467508bd431a170b148001fe8a866fcf96d5cc6af9a7f5e > amd-ctk gpu-tracker status ------------------------------------------------------------------------------------------------------------------------ GPU Id UUID Accessibility Container Ids ------------------------------------------------------------------------------------------------------------------------ 0 0xEA35F57CC80DEB35 Shared a8ce87c99727107ab467508bd431a170b148001fe8a866fcf96d5cc6af9a7f5e 1 0x89CAA15875FF5A43 Shared 90cb29e11e83aa3ae497c68c90e1f0894b85262188c1ef9c7284457a9bc35ffd a8ce87c99727107ab467508bd431a170b148001fe8a866fcf96d5cc6af9a7f5e 2 0x6E32F10EFC982B4C Exclusive a8ce87c99727107ab467508bd431a170b148001fe8a866fcf96d5cc6af9a7f5e 3 0x12FE4F7FDAF06B9 Exclusive 90cb29e11e83aa3ae497c68c90e1f0894b85262188c1ef9c7284457a9bc35ffd ``` In the above example, GPU 1 has been set to `shared` access from the previous `exclusive` access. When a new container `a8ce87c99727107ab467508bd431a170b148001fe8a866fcf96d5cc6af9a7f5e` that requests access to GPUs 0,1 and 2 is launched, the following happens: - The new container is created. - The new container is granted access to GPU 0 as no container is currently using GPU 0. - GPUs 1 is already being used by container `90cb29e11e83aa3ae497c68c90e1f0894b85262188c1ef9c7284457a9bc35ffd`. However, the new container is granted access to GPU 1 as GPU 1 has `shared` accessibility. - The new container is granted access to GPU 2 as no container is currently using GPU 2 though GPU 2 has `exclusive` accessibility. - The container is successfully started since it has been granted access to the required GPU resources. 7. Resetting GPU Tracker Status: Resetting GPU Tracker clears the GPU Tracker state, i.e. the accessibility of all GPUs is set to `shared` and all information about which GPUs have been made accessible in containers is cleared. If GPU Tracker is enabled, then after the reset operation also the GPU Tracker is enabled. Conversely, if GPU Tracker is disabled, then after the reset operation also the GPU Tracker remains disabled. Resetting GPU Tracker is primarily useful in cases where GPU Tracker is enabled and the partitioning scheme of the GPUs has been altered. Changing the partitioning scheme of the GPUs invalidated the CDI Spec and GPU Tracker state. In these cases, it is required to: - Stop all running containers - Reset GPU Tracker - Regenerate CDI Spec - Restart containers If GPU Tracker is disabled when the partitioning scheme of the GPUs have been altered, then GPU Tracker need not be reset. However, it is recommended to still perform the other actions. It makes no difference if GPU Tracker is reset when it is disabled. ```text > amd-ctk gpu-tracker status showing GPU status: GPU info mismatch: please reset GPU Tracker > amd-ctk gpu-tracker reset GPU Tracker has been reset Since GPU Tracker was enabled, it is recommended to stop and restart running containers to get the most accurate GPU Tracker status > sudo amd-ctk cdi generate Generated CDI spec: /etc/cdi/amd.json > sudo docker run --runtime=amd -itd -e AMD_VISIBLE_DEVICES=0-2 rocm/rocm-terminal bash 988135dafcd94bf98fbd92ca97f4a07c9bcfff0521359ee9bc8a6973cc3e25ce > sudo amd-ctk gpu-tracker status ------------------------------------------------------------------------------------------------------------------------ GPU Id UUID Accessibility Container Ids ------------------------------------------------------------------------------------------------------------------------ 0 0xEA35F57CC80DEB35 Shared 988135dafcd94bf98fbd92ca97f4a07c9bcfff0521359ee9bc8a6973cc3e25ce 1 0x89CAA15875FF5A43 Shared 988135dafcd94bf98fbd92ca97f4a07c9bcfff0521359ee9bc8a6973cc3e25ce 2 0x6E32F10EFC982B4C Shared 988135dafcd94bf98fbd92ca97f4a07c9bcfff0521359ee9bc8a6973cc3e25ce 3 0x12FE4F7FDAF06B9 Shared - ``` ## Debugging For verbose debug output when troubleshooting GPU Tracker issues, use the `--debug` (or `-d`) flag: ```text > amd-ctk --debug gpu-tracker status ``` This prints debug-level log messages to stderr, which can help diagnose GPU enumeration or tracker state issues. For container runtime errors (e.g. exclusive GPU enforcement failures), check the runtime log: ```text > sudo tail -f /var/log/amd-container-runtime.log ``` --- Source: https://advanced-micro-devices-dcgpu-documentation--142.com.readthedocs.build/projects/container-toolkit/en/142/container-runtime/release-notes.html # Release Notes This document provides an overview of the AMD Container Toolkit’s release history, including new features, improvements, and compatibility information. The toolkit enables seamless integration of AMD GPUs into containerized environments, enhancing GPU utilization and management. ## Compatibility Matrix #### Compatibility Matrix | AMD Container Toolkit | Docker Version | Supported OS | |-------------------------|----------------------------------|------------------------------------------| | 1.3.0 | 25.0+ | Ubuntu 22.04, Ubuntu 24.04, Ubuntu 26.04 | | 1.2.0 | 25.0+ | Ubuntu 22.04, Ubuntu 24.04 | | 1.1.0 | 25.0+ (`--gpus` flag in 28.3.0+) | Ubuntu 22.04, Ubuntu 24.04, RHEL 9.5 | | 1.0.0 | 25.0+ | Ubuntu 22.04, Ubuntu 24.04 | ## Versioning Information #### Toolkit Versions | Version | Release Date | Highlights | |-----------|----------------|-----------------------------------------------------------------------------------| | v1.3.0 | May 2026 | `amd-ctk gpu list`, GPU partition grouping, CDI and runtime improvements | | v1.2.0 | November 2025 | GPU Tracker feature support, Docker Swarm Support | | v1.1.0 | July 2025 | GPU Partitioning Support , Docker –gpus Support, RHEL 9.5 Support | | v1.0.0 | June 2025 | Initial Release with amd-ctk, CDI, Docker Integration, Ubuntu 22.04/24.04 Support | ### v1.3.0 (May 2026) ## Overview This release extends the command-line tool with GPU discovery and tightens CDI and runtime behavior. - **\`\`amd-ctk gpu list\`\`:** A dedicated command to display GPU information from the host. - **CDI hardening and usability:** Human-readable CDI spec formatting, support for custom CDI spec file names, and the ability for non-root users to generate and validate CDI specifications where appropriate. - **GPU partition grouping:** Partitions of the same physical GPU are now grouped by PCI device topology (`location_id` and `domain`) instead of `unique_id`, improving reliability on multi-GPU and partitioned systems. - **Runtime and GPU Tracker:** Rootful-Docker checks and more consistent behavior and messaging where GPU Tracker and the CLI interact. - **\`\`amd-ctk gpu list\`\`** - Lists GPU details for AMD devices on the system, complementing existing `amd-ctk cdi` workflows. - **CDI and runtime** - CDI spec output is formatted for easier reading and review. - Non-root users can run CDI spec generation and validation in supported configurations. - Custom CDI spec file names (not limited to `amd.json`) are supported when loading specs. ## Improvements - **GPU partition grouping:** GPU partitions are now grouped by parent PCI device address derived from topology `location_id` and `domain` fields, rather than by `unique_id`. This produces correct grouping on systems where partitions of the same GPU have distinct `unique_id` values. - **Container runtime in rootless / non-rootful Docker:** The runtime exits early when the Docker setup is not rootful, matching supported deployment models. - **Docker and CDI:** Using `--gpus` together with CDI-backed AMD devices requires Docker **29.3.0** or newer. ## Upgrade Notes - Regenerate or validate CDI specifications after upgrading; manage specs explicitly with `amd-ctk cdi`. - If you want to use `--gpus` with CDI-backed AMD GPUs, upgrade Docker to **29.3.0** or newer. - Rootless or non-rootful Docker configurations are not supported for the same runtime paths; ensure a rootful Docker engine where the toolkit expects it. ### v1.2.0 (November 2025) ## Overview Version 1.2.0 of the AMD Container Toolkit introduces two major enhancements aimed at improving GPU visibility and orchestration flexibility in containerized environments: - **GPU Tracker:** A new monitoring utility for real-time tracking of GPU usage across containers. - **Docker Swarm Support:** Native integration with Docker Swarm for orchestrating AMD GPU workloads at scale. - **GPU Tracker** - GPU Tracker is an extremely lightweight feature of AMD Container Toolkit that allows you to track access of GPUs in containers. - GPU Tracker provides CLIs that can be used to control the accessibility of GPUs in containers. The accessibility of GPUs can be set to either shared or exclusive. - **Docker Swarm Support** - Allows users to deploy and manage GPU-accelerated containers across a cluster instead of being limited to a single host. - Uses GPU UUIDs for accurate resource mapping and scheduling, ensuring workloads run on specific GPUs. ## Upgrade Notes - GPU Tracker feature is currently supported only if containers are started using the docker run command and GPUs are made accessible in containers using the AMD_VISIBLE_DEVICES environment variable. If containers are started and granted access to GPUs in any other manner, GPU Tracker feature is not supported. ### v1.1.0 (July 2025) ## Overview Version 1.1.0 of the AMD Container Toolkit delivers a significant leap in flexibility, usability, and platform reach for GPU-accelerated container workloads. This release introduces three impactful features: **GPU Partitioning**, **support for RHEL 9.5**, and **integration with Docker’s standardized \`\`–gpus\`\` flag**. These enhancements empower users to maximize GPU utilization, streamline deployment across diverse environments, and adopt industry-standard container interfaces for specifying GPU resources. - **GPU Partitioning Support** - ROCm-based GPUs can now be partitioned into multiple logical devices that are independently accessible from within containers. - Fine-grained control via AMD_VISIBLE_DEVICES environment variable. - Support for range-based device specification (e.g., 0-3,8,17-20). - Compatibility with container runtimes using updated CDI specs. - **Full Support for RHEL 9.5** - This release introduces native RPM packaging and support for RHEL 9.5 systems. - **Support for \`–gpus\` Flag in Docker 28.x+** - Starting from Docker **28.3.0**, containerized GPU workloads can now utilize the standardized –gpus flag to request AMD GPUs directly in docker run commands. - Declarative selection of GPU resources without manual device path specification. - Improved integration with Docker’s native GPU management features. ## Improvements - **Range of GPU Device Selection:** The AMD_VISIBLE_DEVICES environment variable allows users to specify range of GPUs, making it easier to select multiple GPUs or partitions in a concise manner. - **Documentation Updates:** All documentation related to GPU partitioning, RHEL installation, and Docker `--gpus` flag usage has been updated to reflect these new capabilities. - None reported for this release. ## Upgrade Notes - After any GPU partitioning changes, always regenerate and validate the CDI spec to ensure containers have access to the correct devices. - For partitioned GPU visibility inside containers, ensure you are using ROCm version 6.4.1 or newer. - For RHEL 9.5, follow the new installation instructions in the documentation. - To use the `--gpus` flag, upgrade Docker to version 28.3.0 or newer. ## Next Steps 1. Review the updated requirements and quick start guide in the documentation. 2. For GPU partitioning, users can provide range of GPUs and remember to regenerate CDI specs after changes. 3. For RHEL 9.5, follow the new RPM-based installation workflow. 4. To use the `--gpus` flag, ensure you are running Docker 28.3.0 or newer. 5. Deploy your first container using: ```bash docker run --rm --runtime=amd -e AMD_VISIBLE_DEVICES=all rocm/rocm-terminal rocm-smi ``` For more information, refer to the full documentation. ### v1.0.0 (June 2025) ## Overview The initial release of the AMD Container Toolkit provided seamless GPU acceleration for containerized applications using AMD Instinct GPUs. Version 1.0.0 introduced robust Docker integration, CDI support, and developer tooling, making it easier than ever to deploy GPU-accelerated workloads in a containerized environment. ## New Features - **CDI (Container Device Interface) Support:** - Full support for CDI-based GPU enumeration and allocation in Docker containers. - Simplified integration with Kubernetes CDI configurations. - **Enhanced Docker Runtime Integration:** - The AMD Container Runtime (amd-container-runtime) is now fully aligned with Docker 25.0+, ensuring smooth device injection and runtime detection. - **Command Line Utility (\`amd-ctk\`):** Introduction of amd-ctk, a CLI tool for: - Listing GPU devices. - Generating CDI specs. - Configuring Docker runtime. - **Simplified Installation Flow:** - Reduced installation steps with clear dependency management for Ubuntu systems. ## Improvements - Optimized Docker Daemon Configuration - Improved detection of AMD GPUs in /etc/docker/daemon.json. - Easier integration with Docker Compose and multi-container setups. - Better Log Management - All runtime logs are now centralized in /var/log/amd-container-runtime.log for easy access and troubleshooting. - Enhanced GPU Discovery - Faster and more reliable device discovery through amd-ctk cdi list. - Partitioned GPUs were not supported. - RPM builds were considered experimental. ## Upgrade Notes - Docker must be upgraded to version **25.0 or higher**. - Ensure the amdgpu driver version matches the compatibility matrix listed in requirements.rst. - If migrating from NVIDIA, follow the steps outlined in migration-guide.rst. ## Next Steps To get started: 1. Follow the installation steps in the quick-start-guide.rst. 2. Configure Docker using amd-ctk configure runtime. 3. Deploy your first container: ```bash docker run --rm --runtime=amd -e AMD_VISIBLE_DEVICES=all rocm/rocm-terminal rocm-smi ``` Additional details on GPU partitioning, runtime configuration, and CDI validation can be found in the Developer Guide.