Strategies for Selecting GPUs in CUDA Jobs within Multi-GPU Environments

Introduction

In multi-GPU computing environments, efficiently allocating CUDA jobs to specific GPUs is crucial for optimizing resource utilization. Users often encounter issues where all jobs default to GPU 0, leaving other GPUs idle, typically due to improper configuration of the CUDA_VISIBLE_DEVICES environment variable. Drawing from practical cases, this article systematically explains how to manage GPU selection via the command line to ensure balanced job distribution.

Role of the CUDA_VISIBLE_DEVICES Environment Variable

CUDA_VISIBLE_DEVICES is an environment variable provided by the NVIDIA CUDA toolkit that controls the list of GPU devices visible to CUDA applications. When unset, CUDA defaults to all available GPUs, but applications may preferentially select device index 0. By setting this variable, users can restrict jobs to specific GPUs, enabling resource isolation and load balancing.

Methods for Setting CUDA_VISIBLE_DEVICES

There are two primary ways to set CUDA_VISIBLE_DEVICES: temporary and permanent. Temporary setting affects only the current command or session, while permanent setting persists throughout the shell's lifetime.

Temporary Setting

Set the variable directly before running a CUDA executable, e.g., CUDA_VISIBLE_DEVICES=1 ./nbody. This command ensures the nbody simulation runs exclusively on GPU 1. This approach is suitable for one-off jobs and does not interfere with other processes.

Permanent Setting

Use the export command to set the variable in the current shell environment, e.g., export CUDA_VISIBLE_DEVICES=1. After this, all subsequent CUDA commands will default to GPU 1. To revert to default behavior, unset the variable: unset CUDA_VISIBLE_DEVICES.

Specifying Multiple GPUs and Syntax

For jobs requiring multiple GPUs, CUDA_VISIBLE_DEVICES supports a comma-separated list of device indices. For example, export CUDA_VISIBLE_DEVICES=0,1 makes both GPU 0 and GPU 1 visible to applications. This is particularly useful in parallel processing tasks, such as hyperparameter searches, where each job can be bound to a different GPU.

Case Study Analysis

Referencing the Q&A data, the user initially did not set CUDA_VISIBLE_DEVICES, causing all nbody simulations to run on GPU 0. By correctly setting the variable to 1, the job successfully switched to GPU 1. Additionally, the user found that nbody -device=1 worked, but this relies on application-specific parameters, whereas the environment variable offers a more universal solution.

Connection to Reference Article

The auxiliary article discusses efficient methods for running independent jobs on multi-GPU machines, emphasizing the use of CUDA_VISIBLE_DEVICES to limit jobs to specific GPUs (option 2). This aligns with the core of this article but also addresses challenges in multi-process management, such as memory release and locking mechanisms, which complement post-selection optimization strategies.

Code Examples and Operational Steps

The following examples demonstrate how to set and verify CUDA_VISIBLE_DEVICES:

1. Check current setting: echo $CUDA_VISIBLE_DEVICES. If output is empty, the variable is unset.
2. Set variable to GPU 1: export CUDA_VISIBLE_DEVICES=1.
3. Run CUDA job: ./nbody. The job should now use GPU 1.
4. Monitor GPU usage: watch -n 1 nvidia-smi to confirm GPU 1 activity.

For multi-device scenarios, after setting export CUDA_VISIBLE_DEVICES=0,1, applications can utilize both GPUs based on internal allocation logic.

Common Issues and Solutions

Users may face issues where variable settings are ineffective, due to reasons such as incorrect shell environment loading, applications ignoring the environment variable, or driver problems. Recommendations include:

• Ensure a supported environment (e.g., bash) is used.
• Verify CUDA version compatibility (e.g., in CUDA 8.0, deviceQuery may not be in the standard path).
• Use nvidia-smi to confirm GPU status and indices.

Conclusion

By properly utilizing the CUDA_VISIBLE_DEVICES environment variable, users can efficiently manage multi-GPU resources, preventing single GPU overload. Based on real cases and supplementary materials, this article provides a comprehensive guide from basic setup to advanced multi-device management, aiding in the optimization of CUDA job performance.