Skip to content
samit
Interviews/System Design

System Design

13 questions

13 questions
Framework
What framework do you use for ML system design problems?▶

State this before answering any design question:

  1. Requirements: throughput (QPS), latency (P50/P99), accuracy target, scale (users, data size), online vs batch, budget
  2. Data: collection, labeling strategy, preprocessing pipeline, train/val/test splits, handling distribution shift
  3. Model: problem framing (classification vs ranking vs generation), architecture selection, feature engineering, training infrastructure
  4. Evaluation: offline metrics (accuracy, AUC, NDCG), online metrics (CTR, conversion, engagement), A/B testing strategy
  5. Serving: inference latency, batching strategy, caching, hardware (GPU/CPU), scaling
  6. Monitoring: data drift, model drift, alert thresholds, retraining trigger, rollback strategy

Always clarify requirements before diving in. "Design a recommendation system" - is it for YouTube (100M users, cold start problem) or a small e-commerce (catalog of 10K items)?

LLM System Design
Design ChatGPT end-to-end: training to serving.frontier▶

Training pipeline:

  • Data: web crawl (CommonCrawl), books, code (GitHub), curated sources. ~15T tokens.
  • Tokenizer: BPE, vocab size ~100K
  • Pretraining: decoder-only Transformer, next-token prediction loss. 3D parallelism (DP×TP×PP). FSDP for optimizer state sharding.
  • SFT: curated instruction-following data, 10K-100K examples
  • RLHF/DPO: preference data, reward model, policy optimization

Serving pipeline:

  • Model deployment: vLLM or TensorRT-LLM on H100 clusters
  • KV cache: continuous batching, paged attention
  • Load balancer → gateway (rate limiting, auth, routing) → inference servers
  • Streaming: server-sent events (SSE) for token-by-token output
  • Safety: input/output filtering layer (classifier models)
  • Caching: semantic cache for repeated queries, prompt cache
  • Monitoring: latency (TTFT, TPS), GPU utilization, request queue depth, quality metrics
Design a RAG system for 1M internal documents.startup▶

Requirements clarification: what's the query rate? Latency SLA? Document types (PDF, email, markdown)?

Ingestion pipeline:

  • Document parsing: Unstructured / Apache Tika for PDFs, HTML, emails
  • Chunking: recursive character splitting (512 tokens, 50 overlap)
  • Embedding: text-embedding-3-large (OpenAI) or BGE-M3 (open source)
  • Vector DB: Pinecone / Weaviate / pgvector. HNSW index for ANN.
  • BM25 index: Elasticsearch/OpenSearch for sparse retrieval

Query pipeline:

  • Query embedding → dense retrieval (top-20)
  • BM25 → sparse retrieval (top-20)
  • Hybrid: RRF fusion → top-20 combined
  • Reranker (cross-encoder): BGE-reranker → top-5
  • Prompt construction + LLM generation
  • Semantic cache (Redis): skip retrieval + generation for repeated queries

Monitoring: retrieval recall@5 (via golden set), faithfulness score (LLM judge), P99 latency, cache hit rate

Design an LLM inference platform (vLLM-as-a-service).frontier▶

Requirements: multiple models (7B, 70B), multi-tenant, SLA: P50 < 1s TTFT, 50 tok/s TPS, 1000 RPS peak.

Architecture:

  • API Gateway: auth, rate limiting (per-user token budget), request validation, routing (select model)
  • Model router: route to appropriate model instance based on model_id, available capacity
  • Inference workers: vLLM instances, each serving one model. Continuous batching + paged attention. GPU: H100-80GB for 70B, A100 for 7B.
  • Autoscaler: scale workers based on request queue depth, GPU utilization. Kubernetes + KEDA.
  • Prompt cache: cache KV states for common system prompts (e.g., long system prompt shared across requests). vLLM prefix caching.
  • Model registry: versioned model weights, rollout/rollback
  • Observability: TTFT, TPS, GPU utilization, queue depth per model, cost per token
Classic ML System Design
Design a YouTube video recommendation system.▶

Two-stage architecture (YouTube DNN paper):

Stage 1: Candidate Generation (Recall)

  • Goal: retrieve ~100-500 relevant videos from 800M corpus
  • Two-tower model: user tower (watch history embeddings, demographics) + video tower (title, tags, view count)
  • Train with contrastive loss on (user, watched video) pairs
  • Offline: compute all video embeddings, build ANN index (ScaNN/FAISS)
  • Online: compute user embedding → ANN search → top-500 candidates

Stage 2: Ranking (Precision)

  • Goal: re-rank ~500 candidates for this specific user in this context
  • Wide & Deep model: deep features (cross-feature interactions) + wide features (memorization)
  • Features: user-video similarity, video CTR, watch time, freshness, diversity signals
  • Train on watch time (not clicks) to avoid clickbait

Post-ranking: diversity (no 5 videos from same channel), freshness, safety filtering.

Design a fraud detection system.▶

Requirements: real-time (<200ms), 99.9% precision (minimize false positives - blocking legitimate transactions), high recall for large fraud ($$).

Features:

  • Transaction: amount, merchant category, location, time
  • User history: typical transaction patterns (rolling stats), velocity features
  • Network: same device/IP used by flagged accounts, graph features

Model stack:

  • Rule engine (fast, first layer): catch obvious patterns (impossible travel, known fraud IPs)
  • Gradient Boosted Trees (XGBoost): tabular features, well-calibrated probabilities
  • Graph Neural Network: detect fraud rings (connected accounts)
  • Risk score → threshold: block / flag for review / allow

Challenges: extreme class imbalance (0.01% fraud), adversarial adaptation (fraudsters change behavior), label delay (fraud detected days later).

Monitoring: fraud rate, false positive rate (legitimate blocks), precision@k, AUC.

Profile-Specific ML System Design (CV + Startups)
Design a real-time traffic CV system : detect, track, and alert in <100ms.startup▶

Requirements: 25 FPS RTSP streams, edge GPU, alerts for wrong-way/phone/no-helmet, P99 latency <100ms per frame, high recall on safety violations.

  • Ingest: RTSP -> decode on GPU -> resize/normalize
  • Inference: YOLOv10 TensorRT engine + batched stream scheduling
  • Tracking: ByteTrack/DeepSORT for ID consistency and temporal smoothing
  • Rules: zone + direction logic, time-based debounce (avoid alert spam)
  • Serving: Triton + dynamic batching + model versioning
  • Monitoring: frame drop rate, per-camera latency, drift by location/time-of-day

Tradeoff: use slightly smaller model if needed to protect latency SLOs on edge hardware.

How would you design model rollback and canary release under drift for a CV production pipeline?startup▶
  1. Versioned registry: each model has metrics, calibration data, and hardware profile
  2. Canary: route 5-10% camera streams to candidate model; compare precision/recall proxy and latency against baseline
  3. Guardrails: if P99 latency or false-alert rate exceeds threshold -> auto rollback
  4. Shadow mode: run candidate model without user-visible alerts for 24-48h
  5. Rollback: atomic switch in Triton/vLLM router to previous stable model

Drift handling: track weather/nighttime/camera-angle slices; trigger retrain when slice-level quality drops persist.

Design an annotation + active learning loop for low-data CV startup environments.startup▶

Goal: maximize model gains per labeled sample.

  • Sampling: uncertainty sampling (low confidence, high entropy), plus diversity clustering to avoid near-duplicates
  • Auto-label first: teacher model prelabels easy frames
  • Human review: annotators only correct uncertain/hard cases
  • Quality control: inter-annotator agreement checks, gold set audits, guideline updates
  • Retrain cadence: fixed cycle (e.g., biweekly) with replay buffer to prevent forgetting

Metric: improvement per 1k labeled frames and reduction in production failure rate.

Given a fixed monthly budget, how do you allocate spend between training, inference, and data labeling?startup▶

Framework: optimize business KPI per dollar, not raw model score.

  • Estimate marginal gain of each bucket: +$1k labeling vs +$1k larger GPU vs +$1k retraining
  • Protect inference SLO budget first (production stability)
  • Allocate remainder to highest ROI loop (often labeling + hard-example mining early-stage)
  • Use quantization/TensorRT to reduce inference cost before scaling hardware

Interview answer: show simple numbers (cost per camera, alerts/day, false-alert cost, engineer time).

Design an end-to-end MLOps stack for a early-career ML engineer at an Indian AI startup.startup▶

Practical stack: PyTorch training + W&B tracking + DVC data versioning + Triton serving + Grafana alerts.

  • CI: schema checks, data leakage checks, training smoke test
  • CD: model packaging, canary deploy, rollback automation
  • Observability: latency, throughput, false-positive proxy, drift dashboards
  • Ops reality: keep stack simple; avoid over-engineering with 5 orchestration tools

Hiring signal: can you ship and maintain this with a small team.

Distributed Training
What are ZeRO stages 1/2/3? How does FSDP relate?▶

ZeRO (Zero Redundancy Optimizer) eliminates the memory redundancy of standard data parallelism, where every GPU holds a full copy of parameters, gradients, and optimizer state.

Stage 1: shard optimizer states across data-parallel workers (the biggest memory hog for Adam - 2 extra fp32 copies per parameter).

Stage 2: also shard gradients.

Stage 3: also shard the parameters themselves - each GPU only permanently holds 1/N of everything, and all-gathers the full parameters for a layer just before that layer's forward/backward pass, then releases them.

Memory drops roughly linearly with the number of GPUs (stage 3), at the cost of extra communication (frequent all-gathers).

FSDP (PyTorch's Fully Sharded Data Parallel) is essentially a native PyTorch implementation of the ZeRO-3 idea - it shards parameters/gradients/optimizer state and gathers/frees them per-module during the forward and backward passes, integrated directly into `torch.nn`.

What is gradient checkpointing? How does it trade compute for memory?▶

Standard backprop stores every intermediate activation from the forward pass so it can compute gradients during the backward pass - memory scales linearly with depth (number of layers).

Gradient checkpointing only stores activations at a subset of "checkpoint" layers and discards the rest. During backward, when an un-stored activation is needed, it's recomputed on the fly with a small local forward pass from the nearest checkpoint.

Trade-off: roughly +33% compute (one extra forward pass through the recomputed segments) in exchange for activation memory that scales like O(√num_layers) instead of O(num_layers) when checkpoints are placed optimally.

This is what makes it possible to fit very deep/large models on limited GPU memory, and is standard practice when training large transformers.

← Previous

Generative Models

Next →

MLOps & Production