The Ideas Behind Every System You'll Ever Design

Dynamo, Raft, Attention Is All You Need. Engineers cite these papers constantly and read them rarely. We turn each landmark paper into a walkthrough you can actually finish, with real diagrams and the math spelled out instead of skipped.

Free to read
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
AI & ML

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

The paper where a model learned to reason from a plain reward for right answers, including its famous "wait" moment. Score a group of answers the way GRPO does.

DeepSeek-AI · arXiv 2025, then Nature 2025

beginner 5 minFree
Subliminal Learning: Language models transmit behavioral traits via hidden signals in data
AI & ML

Subliminal Learning: Language models transmit behavioral traits via hidden signals in data

An owl-loving model writes plain numbers, and a student trained on them loves owls too. A study published in Nature shows hidden traits can travel through distillation.

Alex Cloud, Minh Le et al. · arXiv 2025, then Nature 2026

beginner 4 minFree
The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models
AI & ML

The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models

Apple researchers tested reasoning models on puzzles with a difficulty dial. They found three regimes, a collapse, and a debate. Play the Tower of Hanoi yourself.

Parshin Shojaee, Iman Mirzadeh et al. · arXiv (Apple) 2025

beginner 4 minFree
Large Language Models Pass the Turing Test
AI & ML

Large Language Models Pass the Turing Test

A pre-registered three-party Turing test found GPT-4.5 with a persona was picked as the human 73 percent of the time. Explore the results by prompt and by crowd.

Cameron R. Jones, Benjamin K. Bergen · arXiv (UC San Diego) 2025

beginner 4 minFree
Generative Agents: Interactive Simulacra of Human Behavior
AI & ML

Generative Agents: Interactive Simulacra of Human Behavior

Twenty-five AI characters live in a tiny town and throw a party nobody scripted. See how a memory stream makes it work, and try the retrieval yourself.

Joon Sung Park, Joseph C. O'Brien et al. · UIST, arXiv 2023

beginner 4 minFree
White Belt· Pro
MapReduce: Simplified Data Processing on Large Clusters
Systems & Distributed

MapReduce: Simplified Data Processing on Large Clusters

Google turned web-scale data processing into two functions: map and reduce. This paper explains how that simple model let a framework handle parallelization, failures, and network efficiency automatically, and became the blueprint for Hadoop and the entire big-data ecosystem.

Jeffrey Dean, Sanjay Ghemawat · OSDI 2004

beginner 5 minPro
Why Language Models Hallucinate
AI & ML

Why Language Models Hallucinate

OpenAI researchers argue chatbots bluff because tests reward guessing. Try the scoring simulator and see why an honest model can lose a leaderboard.

Adam Tauman Kalai, Ofir Nachum et al. · arXiv (OpenAI, Georgia Tech) 2025

beginner 4 minPro
LLMs Get Lost In Multi-Turn Conversation
AI & ML

LLMs Get Lost In Multi-Turn Conversation

An award-winning ICLR 2026 study: the same information, spoken in pieces, makes chatbots about a quarter worse. See why, and what to do about it.

Philippe Laban, Hiroaki Hayashi et al. · arXiv 2025, then ICLR Outstanding Paper 2026

beginner 4 minPro
Measuring AI Ability to Complete Long Software Tasks
AI & ML

Measuring AI Ability to Complete Long Software Tasks

How long a task can an AI finish? METR measured it and found the answer has doubled about every seven months. Play with the trend yourself.

Thomas Kwa, Ben West et al. · arXiv (METR) 2025

beginner 4 minPro
Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
AI & ML

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs

Teach a model one bad habit, insecure code, and it misbehaves on unrelated questions. See the controls that show why intent, and not code, is the trigger.

Jan Betley, Daniel Tan et al. · arXiv 2025

beginner 4 minPro
Artificial Hivemind: The Open-Ended Homogeneity of Language Models
AI & ML

Artificial Hivemind: The Open-Ended Homogeneity of Language Models

Ask many chatbots for a metaphor and most say the same one. A NeurIPS 2025 best paper measures it. Sort the real answers and see the hive.

Liwei Jiang, Yuanjun Chai et al. · NeurIPS Best Paper 2025

beginner 3 minPro
Constitutional AI: Harmlessness from AI Feedback
AI & ML

Constitutional AI: Harmlessness from AI Feedback

Teach an AI to be harmless with a short list of written principles and its own feedback, instead of human harm labels. Step through a real critique and rewrite.

Yuntao Bai, Saurav Kadavath et al. · arXiv (Anthropic) 2022

beginner 3 minPro
The Curse of Recursion: Training on Generated Data Makes Models Forget
AI & ML

The Curse of Recursion: Training on Generated Data Makes Models Forget

What happens when AI learns from AI? Each copy forgets the rare cases. Run the bell-curve experiment from the paper yourself.

Ilia Shumailov, Zakhar Shumaylov et al. · arXiv 2023, then Nature 2024

beginner 4 minPro
Are Emergent Abilities of Large Language Models a Mirage?
AI & ML

Are Emergent Abilities of Large Language Models a Mirage?

Do AI skills really switch on suddenly at scale? An award-winning paper says the sudden jump may come from the ruler. Change the metric and watch the jump vanish.

Rylan Schaeffer, Brando Miranda et al. · NeurIPS Outstanding Paper 2023

beginner 4 minPro
Lost in the Middle: How Language Models Use Long Contexts
AI & ML

Lost in the Middle: How Language Models Use Long Contexts

Hide the answer in a pile of documents and move it. Models read the start and the end well and lose the middle. See the U-shaped curve for yourself.

Nelson F. Liu, Kevin Lin et al. · arXiv (Stanford, Berkeley, Samaya AI) 2023

beginner 4 minPro
A Relational Model of Data for Large Shared Data Banks
Databases

A Relational Model of Data for Large Shared Data Banks

The 1970 paper that gave us tables, keys and joins. See why data independence and normalization still matter, and try moving an office in one table versus two.

E. F. Codd · Communications of the ACM 1970

beginner 3 minPro
Space/Time Trade-offs in Hash Coding with Allowable Errors
Algorithms

Space/Time Trade-offs in Hash Coding with Allowable Errors

The 1970 paper behind the Bloom filter: a few bits and hash functions that say 'definitely not' or 'probably yes'. Build one and hunt for a false positive.

Burton H. Bloom · Communications of the ACM 1970

beginner 3 minPro
A Method for the Construction of Minimum-Redundancy Codes
Algorithms

A Method for the Construction of Minimum-Redundancy Codes

Huffman's 1952 paper: give common symbols short codes and rare ones long codes. Build a code tree for your own text, one merge at a time.

David A. Huffman · Proceedings of the IRE 1952

beginner 4 minPro
A Note on Two Problems in Connexion with Graphs
Algorithms

A Note on Two Problems in Connexion with Graphs

Dijkstra's three page 1959 note: settle nodes in order of distance and a shortest path falls out. Step through the method on a small road map.

Edsger W. Dijkstra · Numerische Mathematik 1959

beginner 4 minPro
Time, Clocks, and the Ordering of Events in a Distributed System
Systems & Distributed

Time, Clocks, and the Ordering of Events in a Distributed System

How can computers with imperfect clocks agree on what happened first? Lamport's answer: define order by cause and effect, then use a two-rule counter to stamp every event.

Leslie Lamport · Communications of the ACM 1978

beginner 5 minPro
A Fast, Minimal Memory, Consistent Hash Algorithm
Algorithms

A Fast, Minimal Memory, Consistent Hash Algorithm

Google's five line consistent hash: no memory, balanced buckets, and only the keys that must move do move. Add buckets and watch which keys jump.

John Lamping, Eric Veach · arXiv (Google) 2014

beginner 4 minPro
Bitcoin: A Peer-to-Peer Electronic Cash System
Systems & Distributed

Bitcoin: A Peer-to-Peer Electronic Cash System

The 2008 whitepaper: a chain of hashed blocks, proof of work, and the longest-chain rule. Mine a block, tamper with the chain, and see an attacker's odds.

Satoshi Nakamoto · Whitepaper 2008

beginner 4 minPro
The Anatomy of a Large-Scale Hypertextual Web Search Engine
Algorithms

The Anatomy of a Large-Scale Hypertextual Web Search Engine

Google's original paper: PageRank as a random surfer, anchor text, and the crawl, index and search pipeline. Edit a tiny web and watch ranks flow.

Sergey Brin, Lawrence Page · WWW 1998

beginner 4 minPro
Dapper, a Large-Scale Distributed Systems Tracing Infrastructure
Systems & Distributed

Dapper, a Large-Scale Distributed Systems Tracing Infrastructure

How Google follows one request through hundreds of machines: trace ids, spans, out-of-band collection and sampling, with a waterfall you can slow down.

Benjamin H. Sigelman, Luiz André Barroso et al. · Google Technical Report 2010

beginner 3 minPro
End-to-End Arguments in System Design
Networking

End-to-End Arguments in System Design

Why reliability belongs at the ends of a system, not in the middle. A careful file transfer, a flaky gateway and the performance exception, with a fault-injection playground.

J. H. Saltzer, D. P. Reed et al. · ACM Transactions on Computer Systems 1984

beginner 4 minPro
Deep Residual Learning for Image Recognition
AI & ML

Deep Residual Learning for Image Recognition

The shortcut connection that made 152-layer networks trainable: the degradation problem, learning F(x) + x, and a toy that shows why deep stacks need it.

Kaiming He, Xiangyu Zhang et al. · CVPR 2016

beginner 3 minPro
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
AI & ML

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Why asking a model to show its steps makes it better at math and logic. See a standard and a chain-of-thought prompt, and hunt for the wrong step.

Jason Wei, Xuezhi Wang et al. · NeurIPS 2022

beginner 3 minPro
Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services (The CAP Theorem)
Systems & Distributed

Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services (The CAP Theorem)

A distributed system split by a network partition must choose: answer with data that might be stale, or refuse to answer at all. Gilbert and Lynch turned Brewer's conjecture into a two-page proof that still decides how every database you use behaves when the network breaks.

Seth Gilbert, Nancy Lynch · ACM SIGACT News 2002

beginner 5 minPro
Efficient Estimation of Word Representations in Vector Space
AI & ML

Efficient Estimation of Word Representations in Vector Space

Before word2vec, a word was just a one-hot index, equally distant from every other word. This paper strips a neural language model down to a bare linear projection so it can train on billions of words, and the resulting vectors turn out to capture meaning well enough that king minus man plus woman lands on queen.

Tomas Mikolov, Kai Chen et al. · ICLR Workshop 2013

beginner 5 minPro
The Google File System
Systems & Distributed

The Google File System

Google needed a filesystem built for thousands of cheap machines that fail constantly, storing huge files that are mostly appended to, not edited. GFS answers with a single metadata-only master, giant 64MB chunks, and a data/control split that later became the blueprint for Hadoop's HDFS.

Sanjay Ghemawat, Howard Gobioff et al. · SOSP 2003

beginner 6 minPro
Yellow Belt· Pro
Dynamo: Amazon's Highly Available Key-value Store
Systems & Distributed

Dynamo: Amazon's Highly Available Key-value Store

Amazon's shopping cart can never say no. This paper shows how Dynamo trades strict consistency for always-on availability using consistent hashing, sloppy quorums, and vector clocks, the same ideas behind Cassandra and Riak today.

Giuseppe DeCandia, Werner Vogels et al. · SOSP 2007

intermediate 7 minPro
Consistent Hashing and Random Trees
Algorithms

Consistent Hashing and Random Trees

The 1997 idea that lets servers join and leave while moving only a few keys. Play with a hash ring and compare it with hash mod N.

David Karger, Eric Lehman et al. · STOC 1997

intermediate 3 minPro
Organization and Maintenance of Large Ordered Indices
Databases

Organization and Maintenance of Large Ordered Indices

The 1970 paper that introduced the B-tree: wide pages, perfect balance, splits that push a key up. Insert keys and watch the pages split.

Rudolf Bayer, Edward M. McCreight · Acta Informatica 1972

intermediate 4 minPro
Skip Lists: A Probabilistic Alternative to Balanced Trees
Algorithms

Skip Lists: A Probabilistic Alternative to Balanced Trees

Sorted lists with coin-flip express lanes: search in about log n steps without any rebalancing. Search and insert in a live skip list.

William Pugh · Communications of the ACM 1990

intermediate 4 minPro
Random Sampling with a Reservoir
Algorithms

Random Sampling with a Reservoir

Vitter's 1985 paper on picking a fair random sample from a stream of unknown length, in one pass and constant space. Watch the reservoir change.

Jeffrey Scott Vitter · ACM Transactions on Mathematical Software 1985

intermediate 3 minPro
An Improved Data Stream Summary: The Count-Min Sketch and its Applications
Algorithms

An Improved Data Stream Summary: The Count-Min Sketch and its Applications

A fixed size grid of counters that estimates how often any item appeared in a huge stream, never too low and rarely much too high. Feed it a stream and ask.

Graham Cormode, S. Muthukrishnan · Journal of Algorithms 2005

intermediate 3 minPro
On the Resemblance and Containment of Documents
Algorithms

On the Resemblance and Containment of Documents

Broder's 1997 paper behind MinHash: turn documents into sets of word groups and compare tiny sketches. Try it on your own texts.

Andrei Z. Broder · Compression and Complexity of Sequences 1997

intermediate 4 minPro
Bigtable: A Distributed Storage System for Structured Data
Databases

Bigtable: A Distributed Storage System for Structured Data

Google's giant sorted map explained: rows, column families, tablets, and the commit log, memtable and SSTable recipe, with two playgrounds for row keys and compaction.

Fay Chang, Jeffrey Dean et al. · OSDI 2006

intermediate 4 minPro
Gorilla: A Fast, Scalable, In-Memory Time Series Database
Databases

Gorilla: A Fast, Scalable, In-Memory Time Series Database

Facebook's in-memory time series store and its two compression tricks: the change in the change for timestamps and XOR for values. Count the bits.

Tuomas Pelkonen, Scott Franklin et al. · VLDB 2015

intermediate 4 minPro
Kafka: a Distributed Messaging System for Log Processing
Systems & Distributed

Kafka: a Distributed Messaging System for Log Processing

Why Kafka is an append-only log, not a queue: partitions, offsets, consumer groups and replay, explained with a playground you can break.

Jay Kreps, Neha Narkhede et al. · NetDB 2011

intermediate 4 minPro
The Chubby Lock Service for Loosely-Coupled Distributed Systems
Systems & Distributed

The Chubby Lock Service for Loosely-Coupled Distributed Systems

Google's lock service in plain terms: sessions kept alive by KeepAlive calls, grace periods, and the fencing tokens that stop a frozen old leader from corrupting data.

Mike Burrows · OSDI 2006

intermediate 5 minPro
ZooKeeper: Wait-free Coordination for Internet-scale Systems
Systems & Distributed

ZooKeeper: Wait-free Coordination for Internet-scale Systems

Yahoo's coordination service: a tiny replicated tree with ephemeral nodes and watches. Build a fair lock queue and crash a client.

Patrick Hunt, Mahadev Konar et al. · USENIX ATC 2010

intermediate 4 minPro
The Tail at Scale
Systems & Distributed

The Tail at Scale

Why the rare slow response ruins big systems, and how hedged requests, tied requests and good-enough answers keep the tail short. Includes a fan-out calculator.

Jeffrey Dean, Luiz André Barroso · Communications of the ACM 2013

intermediate 4 minPro
Metastable Failures in Distributed Systems
Systems & Distributed

Metastable Failures in Distributed Systems

Why some outages outlast their cause: a feedback loop such as retries keeps the system down. Trigger an outage and compare retry policies.

Nathan Bronson, Abutalib Aghayev et al. · HotOS 2021

intermediate 4 minPro
Congestion Avoidance and Control
Networking

Congestion Avoidance and Control

How TCP stopped the Internet from collapsing: slow start, additive increase and multiplicative decrease, and the packet-conservation principle. Watch a window grow and drop.

Van Jacobson, Michael J. Karels · SIGCOMM 1988

intermediate 4 minPro
BBR: Congestion-Based Congestion Control
Networking

BBR: Congestion-Based Congestion Control

Google's way to send fast without filling buffers: measure the path's width and length instead of waiting for loss. Race it against loss-based control.

Neal Cardwell, Yuchung Cheng et al. · ACM Queue 2016

intermediate 4 minPro
The QUIC Transport Protocol: Design and Internet-Scale Deployment
Networking

The QUIC Transport Protocol: Design and Internet-Scale Deployment

The protocol behind HTTP/3: faster handshakes, streams that do not block each other, and connections that survive changing networks. Count round trips and lose a packet.

Adam Langley, Alistair Riddoch et al. · SIGCOMM 2017

intermediate 4 minPro
Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing
Systems & Distributed

Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing

The paper behind Spark: datasets that remember their recipe, so lost memory is recomputed, not copied. Kill a worker and watch the repair.

Matei Zaharia, Mosharaf Chowdhury et al. · NSDI 2012

intermediate 3 minPro
TAO: Facebook's Distributed Data Store for the Social Graph
Systems & Distributed

TAO: Facebook's Distributed Data Store for the Social Graph

How Facebook serves its social graph: typed objects and edges behind a leader and follower cache. Write in one tier and read a stale value from another.

Nathan Bronson, Zach Amsden et al. · USENIX ATC 2013

intermediate 4 minPro
Large-scale Cluster Management at Google with Borg
Systems & Distributed

Large-scale Cluster Management at Google with Borg

How Google runs everything on shared machines: jobs, tasks, priorities and preemption. Place tasks yourself and compare scheduling policies.

Abhishek Verma, Luis Pedrosa et al. · EuroSys 2015

intermediate 4 minPro
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
AI & ML

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

How hiding 15% of the words taught machines to read in both directions. Masked language modeling, the 80/10/10 rule and fine-tuning, with a mask generator you can edit.

Jacob Devlin, Ming-Wei Chang et al. · NAACL 2019

intermediate 4 minPro
Language Models are Few-Shot Learners
AI & ML

Language Models are Few-Shot Learners

The GPT-3 paper explained: in-context learning, zero, one and few-shot prompting, and why scale matters. Build a prompt and price a 175-billion-parameter model.

Tom B. Brown, Benjamin Mann et al. · NeurIPS 2020

intermediate 3 minPro
LoRA: Low-Rank Adaptation of Large Language Models
AI & ML

LoRA: Low-Rank Adaptation of Large Language Models

Fine-tune a giant model by training a tiny add-on. How LoRA's low-rank update shrinks trainable parameters by 10,000 times with no slowdown.

Edward J. Hu, Yelong Shen et al. · ICLR 2022

intermediate 4 minPro
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
AI & ML

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

The paper that named RAG: a retriever plus a generator, with knowledge you can update by swapping an index. Try a miniature retrieve-then-answer pipeline.

Patrick Lewis, Ethan Perez et al. · NeurIPS 2020

intermediate 4 minPro
ReAct: Synergizing Reasoning and Acting in Language Models
AI & ML

ReAct: Synergizing Reasoning and Acting in Language Models

The paper behind tool-using agents: think, act, observe, repeat. Compare an answer from memory with one grounded in a tiny encyclopedia.

Shunyu Yao, Jeffrey Zhao et al. · ICLR 2023

intermediate 3 minPro
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
AI & ML

Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

A NeurIPS 2025 runner-up asks if RL teaches reasoning models anything new. Slide the number of tries and see why the answer was surprising.

Yang Yue, Zhiqi Chen et al. · arXiv, NeurIPS Runner-Up 2025

intermediate 4 minPro
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
AI & ML

Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference

Where leaderboards come from: tens of thousands of people voting between two hidden chatbots. Simulate an arena and see how many votes a ranking needs.

Wei-Lin Chiang, Lianmin Zheng et al. · arXiv (UC Berkeley) 2024

intermediate 4 minPro
Cassandra: A Decentralized Structured Storage System
Systems & Distributed

Cassandra: A Decentralized Structured Storage System

Avinash Lakshman, Prashant Malik

intermediate 8 minPro
Adam: A Method for Stochastic Optimization
AI & ML

Adam: A Method for Stochastic Optimization

A single global learning rate is a poor fit for millions of parameters with wildly different gradient behavior. Adam gives every parameter its own adaptive learning rate by combining momentum with a bias-corrected running average of squared gradients, and became deep learning's default optimizer.

Diederik P. Kingma, Jimmy Ba · ICLR 2015

intermediate 4 minPro
In Search of an Understandable Consensus Algorithm (Raft)
Systems & Distributed

In Search of an Understandable Consensus Algorithm (Raft)

Paxos works, but almost nobody can explain it cleanly. Raft gets the same guarantees by splitting consensus into leader election, log replication, and safety, the algorithm now running underneath etcd, Consul, and Kubernetes.

Diego Ongaro, John Ousterhout · USENIX ATC 2014

intermediate 6 minPro
Black Belt· Pro
Paxos Made Simple
Systems & Distributed

Paxos Made Simple

Getting a cluster to agree on one value, safely, even when proposers collide and messages get lost, is the oldest hard problem in distributed systems. Lamport's own plain-language rewrite of his notoriously opaque algorithm is still the clearest path into how Raft, Chubby, and Spanner actually agree on anything.

Leslie Lamport · ACM SIGACT News 2001

advanced 5 minPro
Attention Is All You Need: The Transformer Architecture Explained
AI & ML

Attention Is All You Need: The Transformer Architecture Explained

RNNs process language one word at a time, which is slow to train and struggles with long-range dependencies. This paper introduces the Transformer, which replaces recurrence entirely with self-attention and became the architecture behind every modern large language model.

Ashish Vaswani, Noam Shazeer et al. · NeurIPS 2017

advanced 5 minPro
A Critique of ANSI SQL Isolation Levels
Databases

A Critique of ANSI SQL Isolation Levels

The 1995 paper that showed the SQL standard's isolation levels were ambiguous, and defined snapshot isolation. Replay anomalies at each level.

Hal Berenson, Phil Bernstein et al. · ACM SIGMOD 1995

advanced 4 minPro
The Log-Structured Merge-Tree (LSM-Tree)
Databases

The Log-Structured Merge-Tree (LSM-Tree)

Why write-heavy databases like RocksDB and Cassandra keep data in memory and merge it to disk in batches. Play with a tiny LSM-tree and watch the merges.

Patrick O'Neil, Edward Cheng et al. · Acta Informatica 1996

advanced 4 minPro
HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm
Algorithms

HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm

Count a billion unique visitors in about a kilobyte. See how leading zeros in hashes, many tiny registers and a harmonic mean make it work.

Philippe Flajolet, Éric Fusy et al. · AofA 2007

advanced 3 minPro
Conflict-Free Replicated Data Types
Systems & Distributed

Conflict-Free Replicated Data Types

How to let every replica accept writes and still merge without conflicts: strong eventual consistency, grow-only counters and add-wins sets, with a three-replica playground.

Marc Shapiro, Nuno Preguiça et al. · SSS 2011

advanced 4 minPro
Spanner: Google's Globally-Distributed Database
Databases

Spanner: Google's Globally-Distributed Database

How Google built a database that is consistent across the planet: TrueTime's honest clock interval, commit wait and Paxos groups, with a clock-skew simulator.

James C. Corbett, Jeffrey Dean et al. · OSDI 2012

advanced 4 minPro
Scaling Memcache at Facebook
Systems & Distributed

Scaling Memcache at Facebook

How Facebook made a plain cache scale: look-aside caching, lease tokens against thundering herds and stale sets, gutter pools and log-driven invalidation.

Rajesh Nishtala, Hans Fugal et al. · NSDI 2013

advanced 4 minPro
Maglev: A Fast and Reliable Software Network Load Balancer
Networking

Maglev: A Fast and Reliable Software Network Load Balancer

How Google balances nearly all of its incoming traffic in software on ordinary servers, and the Maglev hashing algorithm that makes backends take turns.

Daniel E. Eisenbud, Cheng Yi et al. · NSDI 2016

advanced 4 minPro
Training Compute-Optimal Large Language Models
AI & ML

Training Compute-Optimal Large Language Models

The Chinchilla paper: with a fixed compute budget, model size and training data should grow together. See why a 70B model beat a 280B one.

Jordan Hoffmann, Sebastian Borgeaud et al. · NeurIPS 2022

advanced 4 minPro
Training Language Models to Follow Instructions with Human Feedback
AI & ML

Training Language Models to Follow Instructions with Human Feedback

The InstructGPT paper behind modern chat models: supervised examples, a reward model learned from human rankings, and reinforcement learning. Rank answers and train a scorer.

Long Ouyang, Jeff Wu et al. · NeurIPS 2022

advanced 4 minPro
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
AI & ML

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Exact attention that is several times faster by moving less data: tiling, online softmax and the GPU memory hierarchy, with two calculators.

Tri Dao, Daniel Y. Fu et al. · NeurIPS 2022

advanced 4 minPro
Efficient Memory Management for Large Language Model Serving with PagedAttention
AI & ML

Efficient Memory Management for Large Language Model Serving with PagedAttention

How vLLM serves 2 to 4 times more requests by paging the KV cache like an operating system. Fit requests into the same memory and see the waste vanish.

Woosuk Kwon, Zhuohan Li et al. · SOSP 2023

advanced 4 minPro