Responsible AI and Explainability

Overview

What responsible AI means

Responsible AI is a framework for building ML systems that are trustworthy, equitable and accountable. Microsoft states it as six principles: fairness, reliability and safety, privacy and security, inclusiveness, transparency and accountability.

Why explainability matters

Gradient-boosted trees and deep networks are effectively black boxes, and regulators increasingly want a reason for each decision. Explainability is also a debugging tool: it exposes data leakage, spurious correlations and cohort-specific failures.

• EU AI Act — High-risk models need human oversight and transparency documentation
• ECOA (US lending) — Denied applicants get adverse action reasons
• FDA (healthcare) — Model performance characterized across subgroups

Explanation methods

• TreeSHAP — Tree ensembles (XGBoost, LightGBM, Random Forest) — Exact Shapley values in polynomial time — Tree models only
• KernelSHAP — Any model — Weighted linear regression over perturbed inputs — Much slower, approximate
• LIME — Any model; popular for text and images — Local weighted linear surrogate — Unstable across seeds
• Integrated Gradients — Differentiable models — Integrates gradients from a baseline to the input — Depends on the baseline
• Counterfactuals (DiCE) — Any model — Smallest input change that flips the decision — Must stay actionable

SHAP

SHAP gives each feature a contribution score from Shapley values in cooperative game theory. The key identity: the SHAP values sum to the prediction minus the base rate (the expected prediction over the dataset).

• Local: a waterfall plot shows how each feature pushed one prediction above or below the baseline.
• Global: mean absolute SHAP across samples gives feature importance; a beeswarm shows direction and magnitude per feature.

LIME

LIME fits a locally faithful linear surrogate around one prediction. It is fast for a single case but unstable: a different random seed or perturbation strategy can change the explanation.

> [!TIP] SHAP is generally preferred for tabular data; LIME stays popular for text and images.

Integrated Gradients

For neural networks, Integrated Gradients integrates the output's gradient along a straight path from a baseline (a black image, a zero embedding) to the real input.

• Completeness: attributions sum to the output difference between the input and the baseline.
• Implementation invariance: two networks computing the same function get the same attributions.

Counterfactual explanations

A counterfactual answers: what is the smallest change that flips the decision? DiCE generates several that are diverse, actionable and proximate.

> [!NOTE] "You would have been approved if your debt-to-income ratio were 2% lower" helps a loan applicant more than a SHAP value.

Fairness assessment

• Demographic parity — Positive prediction rate
• Equalized odds — True positive and false positive rates
• Predictive parity — Precision

> [!WARNING] These metrics are mutually exclusive in most real settings (Chouldechova's impossibility theorem), so pick the one that fits the deployment context.

Error analysis

• Cohort error analysis: splits error across demographic or feature-based subgroups to find where the model fails disproportionately.
• Error trees: a decision tree trained on the wrongly predicted samples surfaces the feature interactions that drive errors.

Explainability in production

Exact SHAP at serving time adds latency. The usual workarounds:

EU AI Act

The EU AI Act (in force 2024, enforcement phased through 2026) sets risk tiers. High-risk systems (credit scoring, hiring, biometric identification, medical devices) must keep technical documentation, pass a conformity assessment, provide human oversight and log decisions.

> [!IMPORTANT] The Act does not prescribe an explanation method. Regulators care that decisions can be reviewed and challenged.

Interview questions

• Explain SHAP values end to end: the theoretical foundation, how TreeSHAP differs from KernelSHAP, and when you would use each in a production ML system.
• A credit scoring model is flagging loan applications from one demographic group at twice the rate of another. Walk through how you would assess fairness, choose a mitigation strategy, and document your decision for regulatory review.
• Describe when you would use LIME versus SHAP versus Integrated Gradients. What are the failure modes of each method?
• What are the EU AI Act's explainability and transparency requirements for high-risk AI systems, and how would you build an MLOps pipeline that ensures ongoing compliance?

More MLOps & LLMOps topics

• Model Packaging and Formats
• Model Registry
• Model Serving
• Online and Continual Learning
• Real-World MLOps Case Studies
• Reproducible ML Pipelines