Ray on Kubernetes & Dynamic Resource Allocation for GPUs

Overview

The job

Ray has become a common runtime for distributed training, batch inference and model serving, and KubeRay is how it lands on a cluster. The platform team owns the operator, the node pools and the failure modes: a head pod that takes every worker down with it, two autoscalers that disagree about capacity, a training job that starts with half its GPUs.

At the same time Kubernetes replaced the device plugin's "give me N GPUs" with Dynamic Resource Allocation (DRA). A request can now say which GPU, how much memory, which MIG profile, or "an H100, else two A100s", and the scheduler, not a node-label convention, enforces it.

A concrete case. An ML platform team at a mid-size software company shares 128 GPUs between three teams. Training runs as RayJobs admitted whole by Kueue; nightly batch inference is a RayJob whose GPU group scales with Ray Data; the support assistant serves from a RayService with GCS fault tolerance on an HA Redis. Evaluation pods that only need a MIG slice ask for it through a ResourceClaim on a DRA node pool, while the big training pools stay on the device plugin until the migration reaches them.

• Fine-tuning, 16 GPUs — RayJob, fixed worker replicas — nvidia.com/gpu: 8 per worker — Kueue, gang (all or nothing)
• Batch inference — RayJob, autoscaling GPU group — nvidia.com/gpu per worker — Kueue, borrows idle quota
• Online serving — RayService, GCS fault tolerance — whole GPUs — Reserved pool, never preempted
• Evaluation — Plain Pod — ResourceClaim for a MIG slice — DRA scheduler

The cluster the operator builds from one file:

Interview questions

• Design a Ray platform on Kubernetes for three teams: one trains models, one runs nightly batch inference, one serves models online. Which KubeRay resources, and how do they share 128 GPUs?
• The head pod of a production RayService fails. What happens, and how do you design for it?
• Ray jobs wait for GPUs while the cluster has idle GPU nodes, and at other times GPU nodes boot and shut down every few minutes. Explain both, and fix them.
• Walk a pod from a ResourceClaimTemplate to a running container with a GPU under DRA. Where can it fail?
• Compare MIG through the device plugin with MIG through DRA. Which do you choose for a mixed inference cluster today?
• Plan a migration of a 300-node GPU cluster from the NVIDIA device plugin to DRA without changing every manifest at once.

More MLOps & LLMOps topics

• Model Packaging and Formats
• Model Registry
• Model Serving
• Model Supply Chain: Signing, Provenance & Safe Formats
• Online and Continual Learning
• Operating Vector Databases: HA, Reindexing & Backups
• Real-World MLOps Case Studies
• Reproducible ML Pipelines
• Responsible AI and Explainability
• Terraform for GPU Clusters on EKS, AKS & GKE