Production AI Systems

How 5 real AI systems are architected at scale — stack, cost, performance and hard-won lessons.

Production AI systems look nothing like tutorial projects, and this page shows the difference. Five real systems are dissected — a large-scale recommendation system, a neural search engine, a real-time computer vision pipeline, a conversational AI platform and a real-time fraud detection system — each with its AI system architecture, scale metrics and company context.

Every system breaks down performance metrics, infrastructure cost by percentage, the full tech stack by layer, team structure, challenges and hard-won lessons. It is the clearest picture of what senior AI and MLOps work actually involves — useful both for system-design interview prep and for deciding whether this work appeals to you.

Explore: What is MLOps? · How to learn MLOps · Job market analysis

Large-Scale Recommendation System

Personalized Product Recommendation · E-commerce (Unicorn)
50M
dau
180M
mau
1.2M
qps
12K+
features
2.5M
peak qps
45GB
model size
100TB
data volume
50M+
items catalog

⚡ Performance

85ms
p50 latency
180ms
p99 latency
99.95%
availability
ctr lift: +18%revenue impact: +$8M annuallyconversion lift: +12%

💰 Cost breakdown

compute55%
GPU clusters for training, CPU for serving
storage20%
Feature store + model artifacts + logs
network15%
Cross-region replication + CDN
monitoring ops10%
Observability stack + on-call team

🧰 Tech stack

languagesPythonGoJava
ml frameworkTensorFlow 2.xPyTorch (research)
model formatSavedModel + ONNX
data pipelineAirflow + Custom schedulers
model registryMLflow
experiment platformA/B testing framework (in-house)
feature engineeringFeast + Custom pipelines

💡 Lessons learned

  • ›Start with simple models and iterate based on business impact, not just accuracy metrics
  • ›Invest heavily in feature infrastructure early - it becomes the bottleneck
  • ›Real-time features are expensive but often provide the highest ROI for engagement
  • ›Build comprehensive A/B testing early; intuition fails at scale
  • ›Operational excellence (monitoring, alerting, rollback) is as important as model quality

architecture

Deployment

Regions: Multi-region active-active
Strategy: Blue-green with shadow traffic
Containers: 1200+ pods across 8 clusters
Auto Scaling: HPA + Custom metrics (QPS, latency, CPU)

Recall Layer

Type: Multi-channel recall

Channels

  • ›collaborative_filtering
  • ›content_based
  • ›deep_learning
  • ›graph_based
Recall Size: 500-1000 candidates
Latency Budget: 50ms

Ranking Layer

Model: Deep Interest Network (DIN) + Multi-task Learning
Output: Top 50 ranked items

Features

  • ›user_profile
  • ›item_features
  • ›context
  • ›cross_features
Latency Budget: 150ms

Infrastructure

Storage: HDFS + Ceph
Monitoring: Prometheus + Grafana + Custom Dashboards
Feature Store: Redis Cluster + Aerospike
Message Queue: Kafka
Model Serving: TensorFlow Serving on Kubernetes
Batch Processing: Spark on YARN
Real Time Features: Flink
Serving Pattern: Two-stage recall + ranking

challenges

Impact: Reduced cold start conversion gap from 45% to 22%
Solution: Hybrid approach: content-based + popularity + exploration strategies
Challenge: Cold start for new users and items
Impact: Feature freshness improved from 30min to <5min
Solution: Real-time feature computation with Flink, TTL-based cache invalidation
Challenge: Feature drift and staleness
Impact: P99 latency reduced from 350ms to 180ms
Solution: Model compression (quantization), batch inference, request coalescing
Challenge: Model serving latency under load
Impact: Production model performance matched offline by 95%
Solution: Shared feature transformation library, integration tests, shadow mode validation
Challenge: Training-serving skew

team structure

Sre: 4
Ml Engineers: 12
Data Engineers: 6
Product Analysts: 3
Platform Engineers: 8