There’s no single ancestral model from which GPT, Gemini, Claude, and their rivals descend like digital offspring. But there is a mother architecture: the transformer. It comes from an eight-page paper published in 2017 by a team of Google researchers, with one of the best titles in the history of computer science:
“Attention Is All You Need.”
The major LLM families share that architectural inheritance. Their details differ. Their scale differs even more. But the organizing idea—attention mechanisms working alongside feed-forward layers and information about token positions—is the one Vaswani et al. laid out nine years ago. The story of how transformers won is the origin story of modern generative AI.
What Came Before
Before transformers, recurrent neural networks, including Long Short-Term Memory networks, were the leading approaches to modeling text sequences. These processed text one token1 at a time, carrying a hidden state forward like a rolling summary of what they had seen. Information connecting distant parts of a sentence had to survive that chain of updates. The longer the chain, the harder those dependencies could be to learn.
German, with its verb-at-the-end grammar, springs to mind. Consider the proverbial professor who begins a sentence, rambles through an entire lecture, then rattles off the waiting verbs at the end. In programmer terms, he pushes the verbs onto a stack and eventually pops them all off—long after his audience has lost track of what they belong to. This linguistic tall tale illustrates the bookkeeping problem RNNs run into with natural language.
Attention offered a shorter route. Instead of requiring information to be preserved through every step, it let a model draw directly on relevant positions elsewhere in a sequence. Attention wasn’t new in 2017; earlier systems already combined it with recurrent networks. The transformer’s breakthrough was making attention the central mechanism: it could replace recurrence and convolution. Hence the title of the paper.
In self-attention, the model computes how much information from other tokens should contribute to each token’s representation. A pronoun can draw on information from a noun fifty tokens earlier without relying on that information surviving fifty recurrent updates. The weights aren’t a tidy diagram of grammatical relationships, but they give the model a direct way to connect distant parts of the text.
Autoregressive generation, like when ChatGPT responds to a message, proceeds sequentially, token by token, and causal attention prevents a position from looking ahead at later tokens. But removing the recurrent processing chain made training much more parallelizable. The original paper demonstrated both better translation results and substantially shorter training times. The advantages were obvious.
The State of Statelessness
A chat with an LLM feels continuous. For this to happen, earlier messages must remain available as context for the model. An LLM doesn’t learn the conversation the same way it learned during training.
In a straightforward chat implementation, the entire conversation is sent back to the model at each turn. Without caching, the model processes that growing history again and again, increasing both computation and the wait before the answer begins. Prefix caching lets the system reuse earlier attention calculations and process only the new material.
The model’s learned parameters, its working inference state, and the application’s stored conversation are different things. The conversational memory of a chatbot is therefore a property of the system around the model, not just the model itself. What a chatbot can do depends on infrastructure as well as its underlying architecture.
A Question of monoculture
The transformer’s dominance raises a reasonable concern: how much of the field is relying on the same architectural bet?
If transformers have a fundamental limitation in reasoning, grounding, or some other capability, current LLMs might hit a ceiling. New approaches are therefore being developed and tested constantly. Some of these show real promise.
TypeSafe’s Jev, for example, targets structured decisions rather than open-ended text generation. At this point, it’s an interesting alternative to chat-style generation with real advantages for certain use cases, but it’s not (yet) a replacement for the general-purpose LLM.
The success of LLMs so far doesn’t establish how much further transformers can go. But neither does their age establish that we’ve exhausted their potential. The question isn’t whether attention works—it does—but how much further we can take it.
- A token is a piece of text that a model processes as a unit—often a word or part of one. How text is converted to and from tokens isn’t important here. ↩︎
Leave a Reply