Terraform for GPU Clusters on EKS, AKS & GKE

Overview

The job

A GPU cluster is the most expensive thing most platform teams run. One 8-GPU H100 node costs more per hour than a rack of ordinary servers, capacity is scarce, and the mistake that is harmless on a CPU pool (an oversized default, a forgotten taint, an apply that resets a node count) wastes thousands of dollars or kills a training run.

The platform or SRE engineer owns the Terraform that creates those pools: how it is laid out across clouds and environments, how capacity is bought at all, how an accidental 64-GPU pool is stopped before production, and what happens when the real cluster no longer matches the code.

A concrete case. An AI product company trains on a Capacity Block of p5.48xlarge nodes in one AWS Availability Zone, serves inference on autoscaled A100 pools in AKS near its European customers, and runs research jobs on GKE flex-start nodes that arrive when capacity frees up. One pull request edits gpu_pools.tfvars; CI plans each affected root, policy reads the plan JSON, Infracost comments the monthly delta, and apply runs dev, then staging, then production.

• GPU pool resource — aws_eks_node_group, or a Karpenter NodePool + EC2NodeClass — azurerm_kubernetes_cluster_node_pool — google_container_node_pool
• Pick the GPU — Instance type (p5.48xlarge, g6e.12xlarge) or Karpenter labels — vm_size (Standard_NC24ads_A100_v4, ND-series) — machine_type plus a guest_accelerator block
• Driver — GPU AMI (AL2023_x86_64_NVIDIA) or GPU Operator — AKS installs it unless gpu_driver = "None" — gpu_driver_installation_config (DEFAULT, LATEST, INSTALLATION_DISABLED)
• Keep CPU pods off — Taint you set (nvidia.com/gpu) — node_taints you set — GKE adds a GPU taint automatically
• Reserved capacity — Capacity Blocks for ML, On-Demand Capacity Reservations — Capacity reservation groups — Reservations, DWS flex-start and queued provisioning
• Fast interconnect — EFA + cluster placement group — InfiniBand on ND-series SKUs — GPUDirect on A3/A4, compact placement

The one interface every cloud module takes, so policy and reviewers read one shape:

Interview questions

• Design the Terraform layout for GPU clusters on EKS, AKS and GKE, used by both training and inference teams.
• GPUs are scarce. How do you secure capacity, and how do you model reservations in Terraform when they cannot be destroyed?
• Write cost guardrails for GPU node pools as policy. What do you check, and where does it run?
• A terraform apply removed nodes from a running training job. Walk through the investigation and the prevention.
• Multi-node training needs EFA or InfiniBand. What changes in the infrastructure code, and how do you prove the fabric works?

More MLOps & LLMOps topics

• Online and Continual Learning
• Operating Vector Databases: HA, Reindexing & Backups
• Ray on Kubernetes & Dynamic Resource Allocation for GPUs
• Real-World MLOps Case Studies
• Reproducible ML Pipelines
• Responsible AI and Explainability