How Transformers Work
Attention, embeddings, and the transformer architecture
Skills taught: llms.architecture
The Transformer Architecture
Transformers, introduced in the 2017 paper Attention Is All You Need, replaced recurrent networks as the dominant architecture for language. They use self-attention to process all tokens in parallel and weigh their relevance to each other — enabling massive scale and parallelism.
- Self-Attention
- A mechanism that lets each token look at all other tokens and compute how relevant each is for predicting the next token. The word 'bank' attends heavily to 'river' (not 'money') in 'The bank of the river'. This is calculated as Q·K^T (query vs key dot products).
- Embedding
- A dense vector (list of floats) representing a token or piece of text. Similar meanings have geometrically similar vectors. The embedding for 'king' - 'man' + 'woman' ≈ 'queen' in many models.
- Context Window
- The maximum number of tokens the transformer can attend to at once. Because attention is O(n²) in token count, longer contexts use exponentially more compute — which is why longer prompts cost more.
Encoder vs Decoder vs Encoder-Decoder
GPT (Decoder-only) vs BERT (Encoder-only)
- GPT-style (Decoder-only): Generates text left-to-right with causal masking. Each token only sees past tokens.
- BERT-style (Encoder-only): Bidirectional — sees all tokens at once. Great for understanding tasks.
# Get text embeddings via API (OpenAI example)
from openai import OpenAI
client = OpenAI() # reads OPENAI_API_KEY
response = client.embeddings.create(
input="Machine learning is transforming industries",
model="text-embedding-3-small" # 1536-dimensional vector
)
vector = response.data[0].embedding
print(f"Dimensions: {len(vector)}") # 1536
print(f"First 5 values: {vector[:5]}")
# Compute cosine similarity between two texts
import numpy as np
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
v1 = client.embeddings.create(input="dog", model="text-embedding-3-small").data[0].embedding
v2 = client.embeddings.create(input="cat", model="text-embedding-3-small").data[0].embedding
v3 = client.embeddings.create(input="automobile", model="text-embedding-3-small").data[0].embedding
print(cosine_similarity(v1, v2)) # ~0.85 (similar)
print(cosine_similarity(v1, v3)) # ~0.65 (less similar)
Why Context Length Matters for Cost
Attention computation is O(n²) in the number of tokens. Doubling the context roughly quadruples the attention cost. Modern techniques like sliding window attention, flash attention, and grouped-query attention mitigate this, but long contexts still cost more and are slower to process.
# Rough cost comparison for Anthropic Claude models (Nov 2026 pricing)
PRICES = {
"claude-haiku-5-5": {"in": 0.25, "out": 1.25}, # $ per 1M tokens
"claude-sonnet-5-5": {"in": 3.00, "out": 15.00},
"claude-opus-5-5": {"in": 15.00, "out": 75.00},
}
def estimate_cost(model, input_tokens, output_tokens):
p = PRICES[model]
cost = (input_tokens / 1_000_000) * p["in"] + (output_tokens / 1_000_000) * p["out"]
return f"${cost:.4f}"
# Summarise a 10-page document (~4000 input tokens, ~400 output tokens)
print(estimate_cost("claude-haiku-5-5", 4000, 400)) # $0.0015 — pennies
print(estimate_cost("claude-sonnet-5-5", 4000, 400)) # $0.0180
print(estimate_cost("claude-opus-5-5", 4000, 400)) # $0.0900
Tip: You do not need to understand transformer internals to build great LLM applications, but it helps to understand WHY context limits exist and why longer contexts cost more.
Frequently Asked Questions
What is the difference between GPT and BERT?
GPT is a decoder-only model (generates text left-to-right with causal masking). BERT is encoder-only (bidirectional, sees all context). GPT-style models power all major chat LLMs.
What is flash attention?
Flash Attention is an I/O-aware algorithm that computes attention without materialising the full n×n attention matrix in memory, dramatically reducing memory use and enabling longer contexts on the same hardware.
Do I need to understand transformers to use LLM APIs?
No — but it helps. Knowing that attention is quadratic explains why long prompts cost more. Knowing about embeddings helps you understand semantic search and RAG. The API abstracts all the internals.