AI Systems Design
Agent Orchestration: 1,000 Concurrent Workflows, Checkpointing, HITL, Cost Limitssystem designThe brief: build a multi-agent orchestration platform that runs 1,000 concurrent agentic workflows with…
LLM Eval Platform: 50 Teams, Offline Suites + Online A/B + Regression Detectionsystem designThe brief: build an evaluation platform for LLM applications that supports offline eval suites (gated against…
Self-Hosted LLM Inference Service: Llama 3.3 70B at 200 Concurrent, p95 < 4ssystem designThe brief: build a self-hosted inference service that serves Llama 3.3 70B to 200 concurrent requests with…
Large-Scale RAG: 10M Documents, 100k Users, p95 < 2ssystem designThe brief is concrete: build a retrieval-augmented generation service that indexes 10 million documents ,…
Multi-Tenant LLM Platform: BYO-Data, BYO-Model, Per-Tenant Costsystem designThe brief: build a SaaS platform where each customer can bring their own data , choose their model provider…
Real-Time Embedding Pipeline: 1M Docs/Day, Sub-Minute Freshnesssystem designThe brief: build a pipeline that re-embeds 1 million documents per day with sub-minute freshness from…
Staff-level system design walk-throughs for AI/ML systems. Each page is an
end-to-end design exercise — the kind asked in a Staff or Principal
Engineer interview — covering functional and non-functional requirements,
back-of-envelope capacity math, component choices with explicit tradeoffs,
critical-path sequence walk-throughs, failure modes, and per-thousand-request
cost analysis.
The opinions are mine. Where I write "I would" or "we choose," I mean it: these
are the picks I would defend in a design review for a real production system,
not a survey of every option on the market.
Designs
-
Large-Scale RAG
— serving 10M documents to 100k users with p95 < 2s. Hybrid BM25 + dense
+ cross-encoder rerank, pgvector vs Pinecone vs Qdrant, embedding refresh,
tenant isolation, query-result caching.
-
Multi-Tenant LLM Platform
— per-customer data, model choice, cost attribution, and KMS keys. Control
plane vs data plane split; row-level security vs schema-per-tenant vs separate
indexes; per-tenant rate limits and provider routing.
-
Real-Time Embedding Pipeline
— re-embedding 1M documents/day with sub-minute freshness. CDC from
Postgres via Debezium → Kafka → vLLM-served bge-large workers →
idempotent upserts; backpressure, DLQs, cost-quality knobs.
-
Self-Hosted LLM Inference Service
— Llama 3.3 70B at 200 concurrent requests with p95 < 4s. vLLM with FP8
quantization, continuous batching, K/V cache math, GPU sizing (H100 vs L40S),
autoscaling on queue depth, OpenAI-compatible gateway.
How to read these
Every design page follows the same eleven-section skeleton: problem statement,
SLOs, capacity math, architecture, data model, critical paths, scaling
bottlenecks, failure modes, cost analysis, tradeoffs, and a final block of six
collapsible interview Q&A pairs. If you are using these to prep for an
interview, the Q&A block is the one to read out loud.