Model Packaging and Formats

Overview

Why model packaging matters

A model that only runs on the training machine is not a product. Packaging captures the weights and architecture plus the runtime environment, the input/output schema and the preprocessing logic, so any downstream system can load and serve the model without manual steps.

Poor packaging is one of the top causes of training-serving skew. A different scikit-learn version, tokenizer or feature normalization order in serving makes predictions drift from what was validated offline.

• MLflow model — Unified loading, environment and schema on top of any framework — A wrapper, not a runtime of its own
• ONNX — Cross-framework, cross-hardware serving — Not every operator exports
• TorchScript — PyTorch without Python (C++, mobile, edge) — Restricted Python subset
• TF SavedModel — TensorFlow teams, TF Serving, cloud ML platforms — Tied to TensorFlow

MLflow model format

MLflow wraps a model in a directory centered on an MLmodel YAML file. It declares one or more flavors, each a different way to load the model.

• pyfunc flavor: the universal interface. Any model with it can be served by mlflow models serve, whatever the framework. Custom models subclass mlflow.pyfunc.PythonModel and override predict().
• Dependency pinning: conda.yaml (full Conda environment) and requirements.txt (pip only).
• ModelSignature: at serving time MLflow checks input shape and dtype against the signature, catching mismatches before they corrupt predictions.

ONNX: Open Neural Network Exchange

ONNX is a vendor-neutral graph format, originally from Facebook and Microsoft. It represents a model as a graph of standard operators, so models trained in PyTorch or TensorFlow run in any ONNX-compatible runtime.

> [!WARNING] Not every PyTorch operator exports: custom CUDA kernels and dynamic control flow can fail. Match opset_version in torch.onnx.export to your onnxruntime version.

TorchScript

TorchScript serializes a PyTorch model into a static graph that runs without a Python interpreter, and ports to C++ through LibTorch for mobile and edge.

• How — Records operations from one forward pass with example inputs — Parses Python source into TorchScript IR
• Control flow — Not handled — Handled
• Cost — Easy — Type annotations, restricted Python subset

TensorFlow SavedModel

TensorFlow's canonical format since TF 2.0. Signatures define the callable interface, and TF Serving, Vertex AI and SageMaker all consume it natively.

• saved_model.pb — Computation graph and metadata
• variables/ — Checkpoint files for weights
• assets/ — Arbitrary files such as vocabularies

Model compression formats

• Quantization — PTQ needs only a calibration dataset; QAT bakes quantization into training — QAT gives higher accuracy at the cost of a training run
• Pruning — Structured removes whole channels or heads; unstructured removes single weights — Structured maps to standard GEMM kernels; unstructured needs sparse matrix support to speed up
• Knowledge distillation — Student matches the teacher's soft labels — A new, separate model to train and package

> [!TIP] In an interview, tie format choice to the serving target: same-Python batch jobs, a C++ service, multi-vendor hardware or a TF-native cloud platform each point to a different format.

Interview questions

• Explain the MLflow model packaging system end to end. How does a model go from a training script to a deployable artifact, and what guarantees does the format provide?
• What are the trade-offs between ONNX, TorchScript, TensorFlow SavedModel, and native pickle for serving? When would you choose each?
• What is training-serving skew, and how does model packaging specifically address it?
• Describe the model compression pipeline. How do quantization, pruning, and distillation complement each other, and what format considerations apply to compressed models?

More MLOps & LLMOps topics

• ML Testing Infrastructure
• MLOps CI/CD Automation
• MLOps Lifecycle
• Model Deployment Strategies
• Model Drift Detection
• Model Evaluation — Offline and Online
• Model Registry
• Model Serving
• Online and Continual Learning
• Real-World MLOps Case Studies
• Reproducible ML Pipelines
• Responsible AI and Explainability