Neural Networks Had No Memory
By the early 1990s, backpropagation had proven that multi-layer networks could learn. But there was a fundamental problem: these networks had no sense of time. A standard feedforward neural network processes each input independently. Show it the word 'bank' and it has no idea whether you are talking about a river bank or a financial bank, because it has no memory of the words that came before.
Language, speech, music, stock prices, weather. All of these are sequential. The meaning of each element depends on what came before it. To handle sequences, researchers built Recurrent Neural Networks (RNNs). Networks with loops that let information persist from one step to the next. The output from processing word 3 feeds back into the network when processing word 4. In theory, this gives the network memory.
In practice, RNNs forgot almost everything. The problem was the vanishing gradient, the same issue that plagued deep feedforward networks, but worse. In an RNN, each time step is like adding another layer. Processing a 50-word sentence is like having a 50-layer deep network. Gradients from the end of the sentence almost never made it back to influence the processing of early words. The network could remember the last 5-10 words, but anything further back was effectively invisible.
LSTM: Teaching Networks to Remember
In 1997, Sepp Hochreiter and Jurgen Schmidhuber published a paper introducing Long Short-Term Memory (LSTM) networks. The core idea was elegant: instead of hoping that gradients survive the journey through many time steps, build an explicit memory mechanism with gates that control what to remember and what to forget.
An LSTM cell has three gates, each one a mini neural network that outputs values between 0 and 1:
Forget gate: looks at the current input and the previous hidden state, then decides what to erase from the cell memory. Output near 0 = forget this. Output near 1 = keep this.
Input gate: decides what new information to write into the cell memory. It has two parts: a gate that controls how much to write, and a candidate layer that proposes what to write.
Output gate: decides what part of the cell memory to expose as the output for this time step. Not everything stored in memory is relevant to the current output.
The key insight is the cell state, a highway that runs through the entire sequence with minimal interference. Information can flow along this highway unchanged for hundreds of time steps. The gates add or remove information from this highway selectively. This solves the vanishing gradient problem because gradients can flow along the cell state highway without being repeatedly multiplied by small numbers.
A Concrete Example
Consider the sentence: 'The cat, which sat on the mat and watched the birds fly by the window while the dog slept nearby, was hungry.' A regular RNN would probably forget what 'was hungry' refers to by the time it reaches the end. The subject ('cat') is 20 words back. An LSTM can remember it because the forget gate keeps the subject in the cell state while the intervening clause passes through. When the network reaches 'was hungry,' the subject information is still available.
Why LSTMs Dominated for 20 Years
LSTMs became the go-to architecture for anything involving sequences. Google used LSTMs for Google Translate starting in 2016. Apple used them for Siri. Amazon used them for Alexa. Virtually every speech recognition system, language model, and time series predictor from 1997 to 2017 was built on LSTMs or their close cousin, GRUs (Gated Recurrent Units, a simplified version with two gates instead of three).
The LSTM architecture worked well enough for a remarkable range of tasks: machine translation, speech recognition, sentiment analysis, named entity recognition, text generation, handwriting recognition, and music composition. It was not perfect at any of these, but it was good enough to power real products serving billions of users.
The limitations were real, though. LSTMs process sequences one step at a time. You cannot parallelize across time steps because each step depends on the previous one. This made training slow on long sequences. And despite solving the vanishing gradient problem in theory, LSTMs still struggled with very long-range dependencies (hundreds or thousands of tokens). These limitations would eventually lead to the transformer architecture that replaced them. But for two decades, LSTMs were the best tool anyone had.
LSTMs did not solve natural language understanding. They solved the memory problem well enough to build useful products. That practical usefulness, not theoretical elegance. Is why they dominated for so long.
Neural Language Models: Teaching Machines to Predict Words
In 2003, Yoshua Bengio published a paper that seems obvious in hindsight but was revolutionary at the time: 'A Neural Probabilistic Language Model.' The idea was to use a neural network to predict the next word in a sentence, and in the process, learn useful representations of words.
Before Bengio's work, language models were statistical. They counted how often word sequences appeared in large text corpora and used those counts to estimate probabilities. 'The cat sat on the ___'. Look up how often 'mat' follows 'the' after 'on' in your corpus. These n-gram models worked, but they had a fatal flaw: they could only look at a fixed window of previous words (usually 2-5), and they could not generalize. If the model had seen 'the dog sat on the mat' a thousand times but never 'the cat sat on the mat,' it could not make the connection that cats and dogs behave similarly in sentences.
The Embedding Breakthrough
Bengio's key contribution was the word embedding. Instead of representing words as one-hot vectors (a vector of all zeros with a single 1, treating every word as equally different from every other word), he represented each word as a dense vector of real numbers. Say, 100 dimensions. These vectors were learned during training, and words used in similar contexts ended up with similar vectors.
This meant the model could generalize. If it learned that 'the dog sat on the mat' is a valid sentence, it automatically knew that 'the cat sat on the mat' is likely valid too, because 'cat' and 'dog' have similar embedding vectors. The model did not need to see every possible word combination. It learned the underlying structure of language through these shared representations.
This idea, that words can be represented as points in a continuous vector space where proximity reflects semantic similarity. Became the foundation for everything that followed. Word2Vec (2013), GloVe (2014), and eventually the contextual embeddings in transformers (2017) all build on Bengio's 2003 insight. Every time you use an EmbeddingModel in a GenAI pipeline, you are using a descendant of this work.
The Path from LSTMs to Transformers
LSTMs and neural language models set up two critical ideas that the transformer architecture would later combine and supercharge:
Sequential modeling: processing text as an ordered sequence where context matters. LSTMs showed this was possible with gated memory.
Learned representations: representing words as dense vectors where similar words cluster together. Bengio's language model showed this was learnable.
The transformer (2017) would solve the LSTM's biggest limitation. Sequential processing. By replacing recurrence with self-attention, allowing the model to look at every word simultaneously. But it kept the learned representations. In fact, transformer embeddings are richer because they are contextual: the vector for 'bank' changes depending on whether the surrounding words are about rivers or money. LSTMs tried to achieve this through their hidden states, but transformers do it more effectively through attention.
What This Means for GenAI Engineers
If you are building RAG pipelines or AI applications today, the LSTM era matters for three practical reasons.
First, understanding why sequence matters helps you understand why chunk ordering and overlap matter in RAG. When you configure a Chunker with overlap=50 tokens, you are ensuring that context from the end of one chunk appears at the start of the next, the same continuity problem that LSTMs were designed to solve. If your chunks have no overlap, you risk splitting a critical piece of information across two chunks where neither has enough context.
Second, the embedding models in modern RAG pipelines are direct descendants of Bengio's neural language model. When you add an EmbeddingModel to your pipeline and it converts 'What is the refund policy?' into a 1536-dimensional vector, it is using learned representations that trace back to the 2003 paper. The model learned that 'refund policy' and 'return rules' should have similar vectors, so both queries retrieve the same relevant chunks from your VectorDB.
Third, the LSTM-to-transformer transition illustrates a pattern that keeps repeating in AI: the next big architecture solves the previous one's bottleneck. LSTMs solved RNN's memory problem. Transformers solved LSTM's parallelization problem. Understanding this pattern helps you evaluate new architectures critically. Ask 'what bottleneck does this solve?' instead of just following the hype.
Try It in the Lab
In ScaleDojo's GenAI Pipeline Lab, the EmbeddingModel component is a direct descendant of these ideas. When you choose between text-embedding-3-small and text-embedding-3-large, you are choosing between different learned representation spaces. Smaller and cheaper vs. larger and more nuanced. Try Act 1 levels to see how embedding choice affects retrieval quality and cost.
Further Reading
Hochreiter & Schmidhuber (1997): 'Long Short-Term Memory,' the original LSTM paper
Bengio et al. (2003): 'A Neural Probabilistic Language Model' (JMLR)
Chris Olah's blog post 'Understanding LSTM Networks' (2015), the single best visual explanation of LSTMs ever written
Andrej Karpathy's 'The Unreasonable Effectiveness of Recurrent Neural Networks' (2015), fun demos of what character-level LSTMs can generate
Cho et al. (2014): 'Learning Phrase Representations using RNN Encoder-Decoder,' the GRU paper, a simpler alternative to LSTMs
