LLMOps

Overview

What changes compared to classical MLOps

LLMOps is what MLOps becomes when the artifact you ship is not a set of learned weights but a system built on foundation models: prompts, retrieval pipelines, tool definitions and orchestration logic. You orchestrate a model instead of training one, so quality comes from prompt design and retrieval quality, not weight optimization.

• Primary artifact — Trained model weights — Prompts, retrieval config, tools
• How you get it — Training — Orchestrating a foundation model
• What sets quality — Weight optimization — Prompt design and retrieval quality
• Evaluation — Holdout set, deterministic metrics — Open-ended text, judged at scale
• Cost — Mostly training compute — Meaningful per call, per tenant

The request path below is the reference LLMOps pipeline. Each stage is something you version, trace or guard.

The three pillars

• Prompt versioning: prompts as code, versioned and tested in CI.
• Evaluation: subjective quality measured at scale.
• Observability: full-chain tracing from query through retrieval to generation.

Prompts as code

A prompt is a versioned artifact with tests. It lives in the repository, is reviewed in PRs and is tested before merging.

• Prompt registry with lineage to eval results — LangSmith Prompts, MLflow 3 Prompts, PromptLayer, Humanloop, W&B Weave
• Eval suites in CI — Promptfoo (open-source CLI)

Evaluation methodology

LLM outputs are open-ended text, so a standard holdout set with deterministic metrics does not work. Teams combine several approaches.

• Reference-based — Narrow tasks (extraction, classification) — Needs known-correct answers
• LLM-as-judge — Open-ended generation at scale — Verbosity, position and model-family bias
• Human evaluation — Ground-truth calibration — Expensive, slow
• Pairwise preference — Reducing absolute-score bias — Relative ranking only; the technique behind ChatBot Arena
• Behavioral / red-team — Jailbreaks, hallucinations, PII leakage — Needs adversarial test design

RAG observability

In a RAG system the retrieval layer directly determines answer quality, so you measure both halves. Ragas provides four metrics.

> [!NOTE] Groundedness means the answer is supported by the retrieved context; it is the opposite of hallucination.

Cost tracking

Per-call cost is meaningful, and in multi-tenant SaaS so is cost per tenant. Track tokens in and out, cost per request, per tenant and per feature, and cache hit rate.

• Proxy-level tracking: LiteLLM and Helicone record cost across providers.
• Prompt caching: Anthropic and OpenAI cut cost by 80-90% on repeated prompt prefixes.

Guardrails

LLM outputs can leak PII, generate unsafe content or produce malformed structured output. Guardrails check both sides of the model call.

• Guardrails AI — Pythonic schema-and-rule validation
• NVIDIA NeMo Guardrails — Programmable dialog flow control
• LlamaGuard (Meta) — Fine-tuned safety classifier
• Lakera Guard — Prompt injection detection
• Bedrock Guardrails, Azure Content Safety — Cloud-native managed filtering

> [!TIP] First investments for a new LLM app: tracing on day one, an eval suite in week one, prompt versioning in week two.

Interview questions

• How would you set up a CI/CD pipeline for an LLM application with prompt versioning and eval gates?
• Explain evaluation methodology for LLMs. What are the tradeoffs of each approach?
• How do you monitor a RAG pipeline in production? What metrics matter and what tools do you use?
• What are guardrails for LLMs and how would you implement an input/output validation layer?
• How do you manage cost and token spend for a multi-tenant LLM application at scale?
• If we want to update the system prompt for our core agent, how do you mathematically prove to me that the new prompt is better than the old one before we merge the PR?

More MLOps & LLMOps topics

• Azure Machine Learning Platform
• Data Versioning and Dataset Management
• DevOps to MLOps Transition
• Feature Stores
• Hyperparameter Tuning and AutoML
• LLM Serving
• ML Experiment Tracking
• ML Governance and Compliance
• ML Monitoring and Alerting
• ML Pipeline Orchestration
• ML Security and Adversarial Robustness
• ML Testing Infrastructure