AI Core

What is a Transformer?

A Transformer is a neural network architecture that uses self-attention mechanisms to process sequences in parallel rather than sequentially. Introduced in the 2017 "Attention Is All You Need" paper, it's the foundation of all modern LLMs.

Why It Matters

Before Transformers, sequence models (RNNs, LSTMs) processed text word-by-word, making them slow and unable to capture long-range dependencies. Transformers revolutionized NLP by enabling parallel processing and better context understanding—making models like GPT-4 possible.

Key Innovation: Self-Attention

Self-attention allows every token to "look at" every other token in the sequence, weighted by relevance.

# Example: "The cat sat on the mat because it was tired"
When processing "it", attention weights:
"cat" → 0.7 (high attention - "it" refers to cat)
"mat" → 0.1 (low attention)
"sat" → 0.05

Transformer Variants

Encoder-only

BERT, for understanding tasks (classification, NER)

Decoder-only

GPT, Claude, Llama—for text generation

Encoder-Decoder

T5, for translation and seq-to-seq tasks

Related Terms

← Back to Glossary