ML Governance and Compliance

Overview

What ML governance is

ML governance is the policies, processes and tools that keep ML systems developed and operated responsibly. It is not a checkbox after the fact: it is built in from the moment raw data is collected through every retraining cycle in production.

Data lineage

Data lineage tracks the provenance chain from raw source tables, through feature pipelines, to the exact training dataset a model used. Without it, reproducing a model for audit or debugging is impossible.

A complete record answers:

• Sources: which tables were joined.
• Code: which transformation ran, and which version of it.
• Timing: when each step executed.
• Selection: what sampling or filtering was applied.

Apache Atlas, OpenLineage and Azure Purview capture this automatically by instrumenting Spark jobs and SQL queries.

Model lineage

Model lineage extends data lineage to everything that turned the dataset into a model. MLflow, Azure ML and SageMaker Experiments record it per run.

Bias detection and fairness metrics

Bias enters through historical data, label noise and proxy features.

• Disparate impact — Ratio of positive outcome rates between groups — Below 0.8 is a common adverse-impact threshold under US employment law
• Demographic parity — Difference in positive prediction rates across groups — Ignores actual outcome rates
• Equalized odds — Equal true positive and false positive rates across groups — Stricter than accuracy alone
• Accuracy parity — Equal accuracy across groups — Often misleadingly easy to satisfy when base rates differ
• Calibration — Predicted probabilities match observed frequencies within each group — Checked per group, not overall
• Individual fairness — Similar individuals get similar predictions — Needs a notion of similarity

Fairlearn, IBM AIF360 and the Microsoft Responsible AI Toolbox compute these and show them in interactive dashboards.

Mitigation strategies

• Pre-processing — Reweighting, resampling, disparate impact remover — Changes examples or features before training
• In-processing — Adversarial debiasing — A second adversary network penalizes the main model for encoding protected attributes
• In-processing — Exponentiated gradient reduction — Optimizes directly under fairness constraints
• Post-processing — Threshold shifting, Platt scaling — Adjusts decision boundaries per group after training so equalized odds hold

Regulatory landscape

• EU AI Act (phased in 2024-2026) — Four risk tiers; high-risk systems need conformity assessment, human oversight, technical documentation and registration before deployment
• GDPR — Article 22 restricts solely automated decisions with legal or significant effects; purpose limitation; data minimization
• FDA guidance for Software as a Medical Device — A predetermined change control plan (PCCP) lets retraining within documented bounds proceed without full re-approval

Model cards

Model cards, introduced by Mitchell et al. (2019) at Google, are standard documentation that ships with every deployed model.

• Use: intended use cases and out-of-scope uses.
• Performance: metrics broken down by subgroup and operating condition.
• Data: training data description and evaluation datasets.
• Limits: known limitations, ethical considerations and caveats.

Governance tools

• Great Expectations — Data quality assertions that run as pipeline gates
• Evidently AI — Open-source data drift, model performance and bias monitoring
• Azure ML Data Monitoring — Scheduled jobs computing feature drift and data quality against a baseline
• IBM OpenScale — Enterprise bias monitoring, explainability and compliance reporting
• Fiddler AI — Performance management with SHAP explainability, drift detection and natural-language alert rules
• Microsoft Responsible AI Standard — Internal Microsoft policy: impact assessments, fairness reviews, transparency docs, incident response plans

> [!TIP] In an interview, frame governance as automated gates plus human sign-off. Regulators want to see both the checks and who approved.

Interview questions

• How would you design a complete data and model lineage system for a regulated ML pipeline, and what metadata must it capture?
• Walk through how you would detect and mitigate bias in a binary classification model used for loan approvals, covering metric selection, tooling, and mitigation strategy.
• What does the EU AI Act require of high-risk AI systems, and how should an ML team operationalize compliance?
• Describe how you would build an audit trail for a production ML system that satisfies both internal risk management and external regulatory review.

More MLOps & LLMOps topics

• DevOps to MLOps Transition
• Feature Stores
• Hyperparameter Tuning and AutoML
• LLM Serving
• LLMOps
• ML Experiment Tracking
• ML Monitoring and Alerting
• ML Pipeline Orchestration
• ML Security and Adversarial Robustness
• ML Testing Infrastructure
• MLOps CI/CD Automation
• MLOps Lifecycle