ML Experiment Tracking

Overview

What experiment tracking is

Experiment tracking logs every meaningful detail of a training run so it can be reproduced, compared and audited later. At minimum that means hyperparameters, per-step metrics, artifact files, the code commit and the software environment.

Without it, teams rerun experiments that were already done, cannot explain why a deployed model differs from the last champion, and have no audit trail for compliance or debugging.

Core concepts

A tracking system has two levels. An experiment is a named collection of related runs, such as "bert-fine-tune-v2". A run is one execution of training code, with a unique ID, start and end times, a status (running, finished, failed) and logged values.

• Parameters — Fixed inputs set before training starts — Learning rate, batch size, number of layers, regularization
• Metrics — Numbers that change over time, logged with a step or epoch index — Training loss, validation AUC, GPU utilization
• Artifacts — Files produced or consumed by the run — Checkpoints, tokenizer configs, confusion matrix PNGs, dataset shards
• Tags — Free-form key-value strings for search and filtering — Team, model family, data version label

MLflow: the open-source standard

MLflow is the most widely used open-source tracker. The tracking server stores records in a relational database (SQLite or PostgreSQL) and artifacts in a storage backend (local disk, S3, Azure Blob, GCS).

> [!TIP] Call mlflow.autolog() before training to capture parameters, per-step metrics and the final model automatically for scikit-learn, XGBoost, LightGBM, PyTorch Lightning, Keras, Spark MLlib and others.

The four pillars of reproducibility

A run is reproducible when the same inputs recreate its output.

Hyperparameter sweeps

Tracking is the backbone of hyperparameter optimization. Optuna, Ray Tune and W&B Sweeps run many trials, and each trial becomes a child run under a parent sweep run.

Azure ML experiments and jobs

Azure ML wraps MLflow tracking in its Jobs API. Every azure.ai.ml job tracks automatically to the workspace's built-in MLflow-compatible server, and the same mlflow.* calls work inside the training script. Azure ML Studio adds comparison views, metric charts and one-click registration of the best run's model.

Alternatives

• Weights and Biases — Real-time system dashboards (GPU memory, utilization), interactive sweeps, collaborative run annotations
• Neptune.ai — Large-scale metadata management with a query language over thousands of runs
• Comet ML — Code diffing between two runs

All three are drop-in replacements for MLflow tracking in most workflows.

Model logging and the registry

mlflow.sklearn.log_model() or mlflow.pytorch.log_model() stores the model in a standard directory (an MLmodel manifest plus serialized weights). mlflow.register_model() then gives it a version number in the Model Registry.

Interview questions

• Walk me through how you would design an experiment tracking setup for a team of 10 ML engineers working on multiple models simultaneously.
• Explain MLflow's model flavors and why they matter for deployment.
• How do you implement reproducibility for large-scale experiments where the training data changes frequently?
• Compare MLflow, Weights and Biases, and Neptune for a production MLOps pipeline. When would you choose each?

More MLOps & LLMOps topics

• Data Versioning and Dataset Management
• DevOps to MLOps Transition
• Feature Stores
• Hyperparameter Tuning and AutoML
• LLM Serving
• LLMOps
• ML Governance and Compliance
• ML Monitoring and Alerting
• ML Pipeline Orchestration
• ML Security and Adversarial Robustness
• ML Testing Infrastructure
• MLOps CI/CD Automation