Hyperparameter Tuning and AutoML

Hyperparameter tuning systematically searches for optimal model configuration values that cannot be learned from data. AutoML extends this to automate feature engineering, model selection, and ensembling to minimize human effort in the ML development lifecycle.

Overview

What hyperparameters are

Hyperparameters are configuration choices made before training: learning rate, tree depth, regularization strength, batch size, number of layers. Unlike weights, gradient descent never updates them. Poor choices cause underfitting or overfitting; good ones can improve accuracy by 5-30% over defaults.

Tuning looks for the configuration that minimizes a validation metric (val_loss, or maximizes AUC) within a compute budget.

Search strategies

  • Grid search — Every combination in a discrete grid — Guaranteed best within the grid — Cost grows exponentially; only for 1-2 parameters with small ranges
  • Random search — Uniform random samples from the space — Often beats grid for the same budget (Bergstra and Bengio, 2012) because few parameters matter — No learning between trials
  • Bayesian optimization — A surrogate (Gaussian Process or TPE) plus an acquisition function (Expected Improvement, UCB) picks the next config — Typically 3-5x fewer trials than random search — Surrogate fitting overhead

> [!NOTE] The acquisition function balances exploration (uncertain regions) against exploitation (near the current best). Too much exploitation converges early; too much exploration wastes budget.

Tree-structured Parzen Estimator (TPE)

Optuna and Hyperopt use TPE. Instead of modeling p(y | x) directly, it models two densities: l(x) for configurations with good results and g(x) for bad ones. The next candidate maximizes l(x) / g(x).

Optuna

Optuna is built on a study/trial abstraction. A Study is the optimization campaign; each Trial is one configuration. The objective function receives a trial and defines the search space lazily with trial.suggest_float, trial.suggest_int and trial.suggest_categorical.

Pruning stops unpromising trials early based on intermediate values, cutting total compute by 50-80% on deep learning workloads.

  • MedianPruner: stops a trial whose intermediate metric is below the median of completed trials at the same step.
  • HyperbandPruner: runs the Hyperband algorithm inside Optuna.

Ray Tune and distributed tuning

Ray Tune runs trials as Ray actors across a cluster, with pluggable search algorithms (Optuna, Hyperopt, Ax/BoTorch) and trial schedulers.

  • ASHA — Asynchronously promotes the top fraction of trials at each resource rung, so workers never sit idle; the recommended default
  • Population-Based Training (PBT) — Copies weights from top agents into bottom ones and perturbs their hyperparameters, searching configuration and training trajectory together

Hyperband

Hyperband handles the explore-exploit tradeoff in budget allocation. With a maximum budget R and halving rate eta (commonly 3), it runs several successive-halving brackets with different starting sizes.

> [!TIP] Hyperband is asymptotically optimal in the bandit-with-budgets formulation and typically beats random search by 5-10x for neural networks.

Azure ML AutoML

AutoML automates the full pipeline, not just the hyperparameters.

Hyperparameter importance

After a study, optuna.importance.get_param_importances uses fANOVA (functional ANOVA) to estimate how much of the objective's variance each parameter explains. Drop low-importance parameters from future searches.

Tuning versus more data

More data almost always beats better hyperparameters when the model is data-limited.

  • Train accuracy much higher than validation — Overfitting (high variance) — More data or stronger regularization
  • Both low — Underfitting — Architecture changes or hyperparameter search

Interview questions

  • Explain the Hyperband algorithm. Why does it outperform pure random search for neural network tuning, and what are its failure modes?
  • Compare Grid Search, Random Search, and Bayesian Optimization. When would you choose each in a production MLOps pipeline?
  • How does Optuna's pruning mechanism work, and what pruner would you choose for a deep learning training run on a GPU cluster?
  • What is Population-Based Training and when does it provide advantages over standard Bayesian hyperparameter search?

Sign up free to read the full topic

More MLOps & LLMOps topics

Start free on Cariara · Coding problems · System design studio