Blog

Daily posts on AI infrastructure, LLMs, GenAI, agents, recommendation systems, and ML engineering — with diagrams and runnable notebooks.

2026-11-15

System Design, Plainly — Part 8: Replication

System design (replication) — Part 8

2026-11-14

System Design, Plainly — Part 7: Consistent Hashing

System design (consistent hashing) — Part 7

2026-11-13

Prefill vs Decode: Why Splitting Them Wins

GDE #3 / vLLM serving (prefill/decode disaggregation — TTFT vs TPOT, NIXL, Mooncake, EPD)

2026-11-12

System Design, Plainly — Part 6: Rate Limiting

System design (rate limiting) — Part 6

2026-11-11

System Design, Plainly — Part 5: CAP and PACELC

System design (CAP/PACELC) — Part 5

2026-11-10

PagedAttention and the KV Cache, Plainly: Why Memory, Not FLOPs, Caps Your Concurrency

GDE #2 / vLLM serving (PagedAttention + KV cache; block-size 16→64, PR #63231)

2026-11-09

vLLM 0.33, Plainly: CUDA 13, Model Runner V2, and What Breaks on Upgrade

GDE #1 / vLLM serving (vLLM 0.33.0 — CUDA 13, Model Runner V2, upgrade breakages)

2026-11-08

System Design, Plainly — Part 4: Message Queues

System design (message queues) — Part 4

2026-11-07

System Design, Plainly — Part 3: Sharding

System design (sharding) — Part 3

2026-11-06

System Design, Plainly — Part 2: Caching

System design (caching) — Part 2

2026-11-05

System Design, Plainly — Part 1: Load Balancing

System design (load balancing) — Part 1

2026-11-04

109 Million Beats 72 Billion: LeCun's Theorem Says Every Model Sees the Same World

ML architectures / representation learning (JEPA Platonic theorem, arXiv 2610.39302)

2026-11-03

10 Hours to 3: How KAIST's CluSTER Picks the 31% of Data That Actually Teaches

AI training infra / data selection (KAIST CluSTER, EMNLP 2026)

2026-11-02

12% of the Economy: The Benchmark That Grades AI Agents Like Employees

Agents / benchmarks (SemiAnalysis GDPx)

2026-11-01

Stop Typing One Token at a Time: The Coarse-to-Fine Model That's 31× Faster

ML architectures / generative modeling (Next-Concept Modeling, arXiv 2610.14162)

2026-10-31

14 Million Tokens, Zero Humans: The Day Two AIs Killed a Quarter-Century-Old Conjecture

AI for science / complexity theory (ETH refutation — OpenAI + Anthropic, Oct 26-27)

2026-10-30

The Mean Lied: How a Second Statistic Fixed Long-Context Memory

ML architectures / linear attention (Princeton MERLIN, arXiv 2610.37645)

2026-10-29

2 Minutes of World, Then It Forgets: How Genie 4 Builds a Place You Can Walk Through

GenAI / world models (Google DeepMind Genie 4)

2026-10-28

Max Effort Costs 42× More and Solves Less: The Reasoning-Effort Trap

Agents / AI infrastructure (reasoning-effort economics — arXiv 2610.34491)

2026-10-27

90.5% to 81.0%: The Open Model That Ships Weaker on Purpose

Tech news / LLMs / AI safety (Kimi K3.1 Pro open-weight release + cyber cut)

2026-10-26

43% Fewer Tokens, 98.9% of the Quality: How VisionWeave Spends Visual Tokens Like Money

ML architectures / multimodal AI (VisionWeave elastic visual representations)

2026-10-25

The 5.5 Cliff: The Bug in Every Softmax That Silently Zeroes Your Model's Best Gradient

ML architectures / numerical methods (DeepMind softmax fix, arXiv 2610.31642)

2026-10-24

The Lookup Table That Beat 17 Experts: DeepSeek's Engram and the Second Axis of Sparsity

ML architectures / LLM sparsity (DeepSeek-linked Engram conditional memory)

2026-10-23

The Model That Trained Itself: 56% of Real GitHub Issues at 1/50th the Price

LLMs / agentic training (MiniMax M2.7 self-evolution)

2026-10-22

40 GB to 6.7 GB: The 3-Bit Trick That Shrinks Your KV Cache 6x

AI infrastructure / LLM inference (KV cache quantization — Google TurboQuant, arXiv 2504.19874, ICLR 2026)

2026-10-21

The 48.7% Upset: Your AI Agent Just Out-Tuned 30 Years of Optimization Science

Agents / optimization (AgenticBBO-Bench — LLM agents vs classical optimizers)

2026-10-20

11 Hours or 270? The Benchmark That Grades AI Agents in Workdays — and the Cheat That Broke It

Agents / benchmarks (METR time-horizon doubling + GPT-5.6 Sol scoring-rule episode)

2026-10-19

75% More Requests, Same GPUs: Why LLM Serving Runs on Two Machines Now

AI infrastructure / LLM inference serving (prefill/decode disaggregation — Splitwise/Mooncake/HMA-Serve/SPAD/DQ/KITE + Prime Inference)

2026-10-18

The 60% Ceiling: Why Your AI Agent Is a Bad Judge of Its Own Next Move

Agents / benchmarks (Taste-Bench — Microsoft + CityU HK)

2026-10-17

The 1M-Token Training Run: How Alibaba Trains Coding Agents on 8-Hour Bug Hunts

Agents / AI training infra (elastic RL for xLong-horizon agents — Alibaba QwenGyre)

2026-10-16

91% Passed, 25% Safe: Why Your Coding Agent Ships Code That Works and Gets You Hacked

Agents / AI safety (secure coding — LSCBench)

2026-10-15

The 21× Lie: Why AI Research Agents Can't Be Trusted With Their Own Results

Agents / AI eval & safety (InnovationEval + Princeton CRUX)

2026-10-14

The Model That Doesn't Refuse: Mistral's 1.05T Bet on Open Security

Tech news / ML architectures (open-weight MoE — Mistral Large 4, Oct 6)

2026-10-13

150 Milliseconds to First Sound: How ElevenLabs' v4 Turbo Talks Before the Sentence Ends

GenAI (voice synthesis / TTS — ElevenLabs Eleven v4 + v4 Turbo)

2026-10-12

18B Awake, $0.045 a Task: The Cheap Model That Beat Claude on the Agentic Loop

Tech news / ML architectures / agents (GLM-5.3-Flash cost-collapse)

2026-10-11

Passed Once, Failed 19 Times: The Reliability Gap Microsoft's ThinkingBox Found in Every Agent

Agents / benchmarks (ThinkingBox agent reliability — Microsoft + Hugging Face)

2026-10-10

35x Faster, Zero Tokens: The Week Decision Models Went Mainstream

Tech news / ML architectures (decision models wave — OpenAI Decisions API + Microsoft Decision-1)

2026-10-09

The 128-Vector Trick: Why Perplexity Stopped Squeezing Documents Into One Vector

Recommendation systems / AI infrastructure (late-interaction retrieval — Perplexity pplx-embed-v2-late)

2026-10-08

Your LLM Types One Token at a Time. It Doesn't Have To.

LLM systems / inference (speculative decoding)

2026-10-07

The Model Wasn't Stupid. It Was Blind: How a Four-Sentence Prompt Took Claude to a Perfect 100 on ARC-AGI-3

Agents / benchmarks (VISTA visual harness — MIT, ARC-AGI-3)

2026-10-06

501B Stored, 23B Awake: Reflection's Beam and the Efficiency Turn in the Open-Weight Race

Tech news / ML architectures (Reflection AI Beam open-weight MoE)

2026-10-05

78 Billion Parameters, 3.5 Awake: Germany's Sovereign AI Just Shipped

Tech news / LLMs / ML architectures (Aleph Alpha Kolibri sovereign AI)

2026-10-04

95% Harmful, Zero Red Flags: The Agent Handoff Problem Nobody Tests

Agents / AI safety (RogueHandoff-20 — multi-agent handoff injection)

2026-10-03

100,000 Videos in Your Pocket: How Douyin's Recommender Reads Your Whole Life in O(1)

Recommendation systems (ultra-long sequence modeling)

2026-10-01

The Frontier You Can't Touch Yet: Google's Gemini 4 Argon and the New Playbook for Model Launches

Tech news / LLMs (Gemini 4 Argon launch)

2026-09-30

The Model That Failed Its Own Safety Test: Why OpenAI Shelved GPT-6.1 Astra

Tech news / AI safety (GPT-6.1 Astra shelved; agent alignment evals)

2026-09-29

Tokens Are a Crutch: Why Byte Models Win the Long Game

NLP updates / ML architectures (byte-level models — "Breaking the Token Ceiling", arXiv 2609.12303)

2026-09-28

5.7x, 512 GPUs, One Endpoint Across the Pacific: AI's Report Card Just Grew Up

AI infrastructure / benchmarks (MLPerf Inference v6.1)

2026-09-27

Thinking Twice Can Make You Dumber: The Test-Time Compute Playbook Behind September's Smartest Models

AI infrastructure / ML architectures (test-time compute scaling)

2026-09-26

Only 3% of the Brain Wakes Up: The Mixture-of-Experts Playbook Behind September's Biggest Models

ML architectures (mixture-of-experts)

2026-09-25

The Paper That Answers Back: Stanford Turned 100 Research Papers into Agents

Agents / AI for science (Paper2Agent — papers as MCP agents)

2026-09-24

The Model That Doesn't Talk

Tech news / ML architectures (TypeSafe Jev — decision-output transformer)

2026-09-23

The Tuesday Intelligence Went on Sale

Tech news / LLM economics (frontier price war)

2026-09-22

One Update, One Quarter-Turn: The Attention Layer That Learned to Rotate

ML architectures (linear attention / Kimi Delta Attention)

2026-09-21

1,107 Tokens Per Second: The LLM That Doesn't Type

GenAI + ML architectures (diffusion LLMs)

2026-09-20

The Two-Phase Machine: Your LLM Request Is Two Jobs in a Trench Coat

AI infrastructure (LLM inference serving)

2026-09-19

One Model, Three Jobs, Zero Retraining: The Spreadsheet Just Got Its Foundation Model

Tech news / ML architectures (tabular foundation models)

2026-09-18

The Same Model, Three Agents: Why the Harness — Not the Model — Decides What Your AI Can Do

Agents (agent harnesses)

2026-09-17

The 10-Millisecond Trick: How Recommenders Search Millions of Items Before You Blink

Recommendation systems (two-tower retrieval)

2026-09-16

The 16x Context Trick: How Latent Context Models Finally Made Compression Work

LLM inference (context compression)