ML Security and Adversarial Robustness

Adversarial attacks, data poisoning, model stealing, supply chain risks, and defenses including adversarial training and differential privacy.

Overview

The ML Security Surface

Machine learning adds attack surfaces that ordinary software does not have. The model is only as trustworthy as its data, its training run, the artifact it ships as and the API that serves it, and each of those can be attacked separately. Treat it like application security: model the threats, simulate the attacks, put defenses in place and keep monitoring.

Adversarial examples — Inference — Wrong prediction on a crafted input — Inputs far from the training distribution — Adversarial training
Data poisoning — Data and training — Hidden behavior or lower accuracy — Label-quality drift, odd clusters — Label review, data provenance
Model stealing — Inference — A working copy of your model — Heavy, systematic query patterns — Rate limits, query monitoring
Membership inference — Inference — Learn who was in the training data — Probing with near-duplicate records — Differential privacy
Supply chain — Artifact — Code execution or a trojaned model — Unverified downloads, pickle files — Hash checks, safetensors, signing

Adversarial Examples

An adversarial example is an input changed just enough to flip the model's prediction while still looking normal to a person. Both classic attacks follow the gradient of the loss with respect to the input, the direction that increases the error fastest.

Steps — One — Many (each FGSM-sized, then projected)
Strength — Weaker, easier to detect — Strongest standard attack in the ε budget
Cost — One gradient — k gradients
Used for — Quick robustness checks — Adversarial training and serious evaluation

Beyond images

Physical patches: a printed sticker in a camera's view fools an object detector.
Text: character swaps, synonym replacement and look-alike (homoglyph) characters flip a classifier with changes a reader barely notices.
Speech: psychoacoustic hiding places commands humans cannot hear but a voice model obeys.

Data Poisoning

Poisoning attacks the training pipeline instead of the live model. The most dangerous form is a backdoor: the model behaves perfectly until it sees the attacker's trigger.

Backdoor — Trigger pattern (a pixel patch, a token) paired with a target label — One hidden behavior; clean accuracy stays high
Clean-label — Inputs changed imperceptibly, labels left correct — The decision boundary moves where the attacker wants
Label flipping — A fraction of labels changed to wrong ones — Overall accuracy, or one targeted class
Federated poisoning — A malicious participant sends poisoned gradient updates — The global model, after aggregation

> [!WARNING] Backdoors pass normal evaluation because clean test accuracy is unchanged. Only trigger scanning (STRIP, Neural Cleanse) or provenance checks on the data find them.

Model Stealing and Membership Inference

Model stealing copies a model through its own prediction API. The stolen copy is also a white-box stand-in the attacker can use to craft adversarial inputs against the real model.

Membership inference asks whether one specific record was in the training set. Overfit models are more confident on examples they trained on, and that gap leaks the answer.

> [!NOTE] Anything that narrows the train-versus-unseen confidence gap weakens membership inference: regularization, early stopping and, with a formal guarantee, differential privacy.

Supply Chain Attacks

Pre-trained weights from public hubs can carry backdoors that survive fine-tuning. Worse, pickle-based formats run code while they load.

> [!WARNING] torch.load and pickle.load execute code embedded in the file. Load untrusted weights with torch.load(..., weights_only=True) or convert them to safetensors first.

Defenses

Adversarial training (PGD) — Adversarial examples — Trains on PGD-perturbed inputs alongside clean ones — Several times the training compute; some clean accuracy
Differential privacy (DP-SGD) — Membership inference, memorization — Clips each example's gradient and adds Gaussian noise — Lower accuracy; about 2-4x slower training
Randomized smoothing — Norm-bounded adversarial inputs, with a certificate — Majority vote over many noisy copies of the input — Many forward passes per prediction
Input preprocessing — Weak perturbations — JPEG compression, feature squeezing before inference — Adaptive attackers get around it
Query monitoring and rate limits — Model stealing, probing — Flags unusual query volume and patterns — Needs tuning to avoid blocking real users
Hashes, safetensors, signing — Supply chain — Verifies what you load and where it came from — Pipeline work, little runtime cost

DP-SGD, step by step

Stronger privacy guarantee — Weaker guarantee
More noise, lower accuracy — Less noise, higher accuracy

Randomized smoothing, the only defense here with a provable guarantee: the prediction cannot change for any perturbation inside the certified radius.

Threat Modeling

The NIST AI Risk Management Framework starts every ML security program with a threat model, repeated as the system changes.

> [!TIP] The NIST AI RMF groups this work into four functions: Govern, Map, Measure and Manage. In an interview, naming the assets and actors before the defenses shows you think like a security engineer.

Interview questions

Describe the threat landscape for a production ML model serving real-time fraud predictions, and prioritize the top three mitigations.
Explain differential privacy in ML training: what guarantee does it provide, how is it implemented with DP-SGD, and what are the practical tradeoffs?
How would you secure the ML model supply chain from pre-training artifact to production deployment?

More MLOps & LLMOps topics

Start free on Cariara · Coding problems · System design studio