Transformers

A transformer is the neural network design behind modern language models like GPT and Claude. Text is split into tokens, word pieces, and each becomes a vector of numbers. The key step is attention: for every token, the model scores how relevant every other token is, then blends in information from them weighted by those scores, so “it” can pick up meaning from the noun it refers to many words back. Stacking many layers of attention and small feed-forward networks lets the model build up rich context. Unlike older recurrent networks, all tokens are processed in parallel during training, which is what let these models scale. The cost is that attention grows with the square of the input length.