Production AI Systems

How 5 real AI systems are architected at scale — stack, cost, performance and hard-won lessons.

Large-Scale Recommendation System

Personalized Product Recommendation · E-commerce (Unicorn)
50M
dau
180M
mau
1.2M
qps
12K+
features
2.5M
peak qps
45GB
model size
100TB
data volume
50M+
items catalog

⚡ Performance

85ms
p50 latency
180ms
p99 latency
99.95%
availability
ctr lift: +18%revenue impact: +$8M annuallyconversion lift: +12%

💰 Cost breakdown

compute55%
GPU clusters for training, CPU for serving
storage20%
Feature store + model artifacts + logs
network15%
Cross-region replication + CDN
monitoring ops10%
Observability stack + on-call team

🧰 Tech stack

languagesPythonGoJava
ml frameworkTensorFlow 2.xPyTorch (research)
model formatSavedModel + ONNX
data pipelineAirflow + Custom schedulers
model registryMLflow
experiment platformA/B testing framework (in-house)
feature engineeringFeast + Custom pipelines

💡 Lessons learned

  • ›Start with simple models and iterate based on business impact, not just accuracy metrics
  • ›Invest heavily in feature infrastructure early - it becomes the bottleneck
  • ›Real-time features are expensive but often provide the highest ROI for engagement
  • ›Build comprehensive A/B testing early; intuition fails at scale
  • ›Operational excellence (monitoring, alerting, rollback) is as important as model quality

architecture

Deployment

Regions: Multi-region active-active
Strategy: Blue-green with shadow traffic
Containers: 1200+ pods across 8 clusters
Auto Scaling: HPA + Custom metrics (QPS, latency, CPU)

Recall Layer

Type: Multi-channel recall

Channels

  • ›collaborative_filtering
  • ›content_based
  • ›deep_learning
  • ›graph_based
Recall Size: 500-1000 candidates
Latency Budget: 50ms

Ranking Layer

Model: Deep Interest Network (DIN) + Multi-task Learning
Output: Top 50 ranked items

Features

  • ›user_profile
  • ›item_features
  • ›context
  • ›cross_features
Latency Budget: 150ms

Infrastructure

Storage: HDFS + Ceph
Monitoring: Prometheus + Grafana + Custom Dashboards
Feature Store: Redis Cluster + Aerospike
Message Queue: Kafka
Model Serving: TensorFlow Serving on Kubernetes
Batch Processing: Spark on YARN
Real Time Features: Flink
Serving Pattern: Two-stage recall + ranking

challenges

Impact: Reduced cold start conversion gap from 45% to 22%
Solution: Hybrid approach: content-based + popularity + exploration strategies
Challenge: Cold start for new users and items
Impact: Feature freshness improved from 30min to <5min
Solution: Real-time feature computation with Flink, TTL-based cache invalidation
Challenge: Feature drift and staleness
Impact: P99 latency reduced from 350ms to 180ms
Solution: Model compression (quantization), batch inference, request coalescing
Challenge: Model serving latency under load
Impact: Production model performance matched offline by 95%
Solution: Shared feature transformation library, integration tests, shadow mode validation
Challenge: Training-serving skew

team structure

Sre: 4
Ml Engineers: 12
Data Engineers: 6
Product Analysts: 3
Platform Engineers: 8