How We Got Here

The first artificial neuron. A single-layer network that could learn to classify simple patterns. It worked for AND and OR gates, but failed on XOR - a problem so basic it nearly killed the field. Marvin Minsky proved this limitation in 1969, and funding dried up. The first "AI Winter" began.

The breakthrough that made deep networks possible. Instead of guessing how to adjust weights, backprop calculates exactly how much each weight contributed to the error, then adjusts proportionally. Like tracing a mistake back through a chain of decisions to find where things went wrong.

A deep convolutional neural network trained on two NVIDIA GTX 580 GPUs destroyed the ImageNet competition by over 10 percentage points. This wasn't an incremental improvement - it was a paradigm shift. Everyone realized: deep learning works, and GPUs are the key. Funding flooded back. The "AI Spring" began.

Mikolov proved that words could be represented as vectors where arithmetic works on meaning. The famous example: king - man + woman = queen. Words aren't just labels anymore - they have geometry. Similar meanings cluster together. This was the birth of embeddings, the foundation of every vector database and semantic search system.

Eight researchers at Google wrote a 15-page paper that would reshape the entire field. Their radical idea: throw away RNNs entirely. No recurrence, no convolutions. Build the entire model from attention layers stacked on attention layers. The result trained 10x faster than RNNs because everything runs in parallel - RNNs had to process tokens one by one, but transformers process ALL tokens simultaneously.

In RNN-era attention, the decoder looked at the encoder. In self-attention, every token in a sequence looks at every OTHER token in the same sequence. "The cat sat on the mat because it was tired" - self-attention learns that "it" refers to "cat" not "mat" by computing relevance scores between all word pairs. Each token asks: "Who else in this sentence matters to me?"

OpenAI made a bold bet: take a transformer decoder, pre-train it on a huge pile of books (unsupervised - just predict the next word), then fine-tune it for specific tasks. 117 million parameters. The key insight wasn't the architecture - it was the training recipe. Pre-train once on general text, then cheaply adapt to any task. One model, many uses.

Google's counter-move. While GPT reads left-to-right (decoder), BERT reads in BOTH directions (encoder). Training trick: randomly mask 15% of words and make the model predict them. "The [MASK] sat on the mat" - BERT learns that "cat" fits better than "car." BERT dominated NLP benchmarks for years. The encoder vs. decoder debate begins.

An LLM confidently generates plausible-sounding text that is completely false. Ask about a company's refund policy and it invents one. Ask for a legal citation and it fabricates case numbers. For consumer chatbots this is annoying; for enterprise applications (legal, medical, financial) it's a dealbreaker. The industry needed a way to ground LLMs in actual data.

The paper that launched a thousand startups. The core idea: instead of relying on what the LLM memorized during training, retrieve relevant documents from a knowledge base and include them in the prompt. Step 1: User asks a question. Step 2: Retriever finds relevant documents. Step 3: Documents + question go to the LLM. Step 4: LLM generates an answer grounded in real data. Hallucination drops dramatically.

The paper that launched the agent era. Instead of just generating text, let the model THINK about what to do (reason), then DO it (act), then OBSERVE the result. Thought: "I need to find the weather in Tokyo." Action: search("Tokyo weather"). Observation: "72F, cloudy." Thought: "Now I can answer the user." This loop - think, act, observe, repeat - is the foundation of every AI agent.

The API feature that made agents practical. Instead of parsing free-text "actions" from the model output, LLM providers added native function calling. You define tools as JSON schemas, the model generates structured function calls, your code executes them, and results go back to the model. OpenAI launched it in June 2023. Anthropic and Google followed. The model doesn't just write text anymore - it programs.

The biggest bottleneck in LLM serving was memory management. Each request needs its own KV-cache (key-value memory from attention), and different requests have different lengths. Traditional serving wasted 60-80% of GPU memory on fragmentation. vLLM borrowed an idea from operating systems: paging. Just like virtual memory maps non-contiguous physical pages to contiguous virtual addresses, PagedAttention maps KV-cache blocks flexibly. Result: 24x throughput improvement.

Static batching waits for all requests in a batch to finish before starting new ones. If one request generates 500 tokens and another generates 10, the short one waits for the long one. Continuous batching processes token-by-token across requests - when a short request finishes, a new request immediately takes its slot. Combined with vLLM, this made LLM serving practical at scale.