LLMs in Production

Frontier Model Comparisonproductionrouting between models LLMOps and Productionproductionevaluation, deployment, monitoring vLLM and Quantizationproductionself-hosted serving LLM EvaluationproductionHow to evaluate LLM applications: benchmarks, task-specific eval sets, LLM-as-judge, human evaluation,…

Getting an LLM application from prototype to production raises questions a notebook does not: which model to call for which request, whether to use an API or host the model yourself, how to make serving affordable, and how to evaluate and monitor quality once real users arrive. The three pages in this section each take one of those questions.

Running LLM systems: LLMOps, self-hosted serving and quantization, and choosing between frontier models.

Frontier Model Comparison (2026)

Picking a model in 2026 is no longer a single decision — it is a routing decision. Most production systems use two or three models: a small fast one for…

LLMOps and Production

LLMOps is the operational discipline of running LLM-backed systems the same way you run any other production software: versioned, tested, observable…

vLLM and Quantization

Self-hosting open-weights LLMs in 2026 is a different problem than it was in 2023. Llama 3.x and Qwen 2.x match or beat hosted GPT-3.5-class models on most…

1. The production stack

Production LLM stack Application prompts, RAG, agents Router / gateway model choice, fallbacks, rate limits Frontier API models hosted by provider Self-hosted models vLLM, quantized weights LLMOps: evaluation, tracing, cost and latency monitoring

Requests flow from the application through a gateway that chooses a model per request: a small, cheap model for simple tasks and a frontier model for hard ones, with fallbacks when a provider is down. Some traffic may go to self-hosted open-weight models served by vLLM, often quantized to fit on fewer GPUs. Across all of it, LLMOps practices record prompts, responses, latency and cost, and run evaluations so that prompt or model changes are tested before release.

2. Which page answers which question

QuestionPageKey trade-off
Which model should handle this request?Frontier Model ComparisonQuality versus cost and latency
How do we test, deploy and monitor?LLMOps and ProductionRelease speed versus regression risk
Should we host models ourselves?vLLM and QuantizationControl and data locality versus operational burden

Hosted APIs are the default starting point because they need no infrastructure. Self-hosting becomes attractive with steady high volume, strict data residency, or a need for fine-tuned open models.

LLM EngineeringSectionprompts, agents and retrieval Azure AI Foundryazuremanaged evaluation and deployment