LLM Cost & Latency Engineering

Overview

Why this topic

An LLM feature that works in a demo can still fail in production for two plain reasons: it costs more per user than the user pays, or it answers too slowly for the screen it lives on. Both are engineering problems with numbers, and both are usually solved without changing the model's quality at all.

The levers are few and well understood. You pay per token, separately for input and output. You wait for two different things: the first token, and then every token after it. Almost every optimization moves one of those four quantities.

> [!IMPORTANT] Every price in this topic is illustrative: round numbers chosen to make the math easy to follow. Real prices change often and differ by provider and model. Plug your provider's current price sheet into the same formulas.

Interview questions

• Build the monthly cost model for an LLM feature and show where the money goes.
• Your p95 latency is 9 seconds and product wants under 3. How do you break it down and fix it?
• Design a model cascade. How do you decide when to escalate, and how do you prove it saves money?
• Prompt caching is enabled but the bill did not drop. Debug it.
• How do you control output length and format without hurting answer quality?
• Which work belongs on a batch API, and how do you run a large batch safely?

More Agentic AI topics

• Distillation & Small Models
• Embeddings & Vector Search
• Evaluating Agents: Task Suites, Trajectories & Regression Gates
• Evaluating RAG & Catching Hallucinations
• Fine-tuning LLMs: SFT, LoRA/QLoRA & Preference Tuning
• Guardrails & Prompt Injection Defense
• LLM Observability: Traces, Spans & OpenTelemetry GenAI
• Model Context Protocol (MCP) Servers
• Multi-Agent Systems with LangGraph
• RAG in Production: Ingestion to Answer
• Real-time Voice Agents: STT → LLM → TTS