Ankit Mishra's Blog

Understanding SnapKV: Compressing LLM Memory From First Principles

2026-06-23T00:00:00Z

Here is the GitHub repo that you can check out: GitHub Repository

KV Cache is one of the most critical optimization techniques in modern Large Language Models. However, it also creates one of the biggest memory bottlenecks in AI inference.

Most resources explain KV Cache compression with complex math, but the core intuition is surprisingly simple.

In this article, we will break down why KV Cache exists, the massive memory problem it creates, and how a technique called SnapKV solves this by keeping only the tokens that truly matter.

Want visuals? We have plenty of them.

Want intuition first? Keep reading.

Let’s begin.

Step 1: Why Does KV Cache Exist?

Before understanding KV Cache, remember one fundamental rule:

LLMs generate text one token at a time.

If the model is given "My name is" and needs to generate "Ankit", it doesn’t produce the entire answer at once. Instead, it predicts one token, then the next.

The Problem: Repeating The Same Work

Imagine you’re reading a book and someone asks: “What was the first word on page 1?” Every time they ask a new question, you reread the entire book from the beginning. That would be incredibly inefficient.

Without KV Cache, an LLM does exactly this.

"My"
      ↓
   Compute

"My name"
      ↓
Compute Everything Again

"My name is"
      ↓
Compute Everything Again

Generate Next Token
      ↓
Compute Everything Again

The model repeatedly processes the exact same information for previous tokens.

The Solution: KV Cache

Notice something important: The words "My", "name", and "is" never change. Once we’ve processed them, their information remains exactly the same.

KV Cache simply saves the Keys (K) and Values (V) of every previously processed token so they can be reused.

Compute Once → Reuse Forever

This prevents the model from doing the same attention computations over and over again, drastically speeding up generation.

Step 2: The New Problem Created By KV Cache

While KV Cache makes generation faster, it introduces a massive new problem: Growth of Memory.

As the conversation gets longer:
Longer Prompt → More Tokens → More KV Pairs → More Memory

Eventually, the GPU runs out of memory.

The key insight is that KV Cache grows linearly with context length, while model weights stay fixed. Once context windows reached 32k, 128k, and 1M tokens, researchers realized KV Cache was becoming the biggest memory consumer in the inference stack.

Step 3: Do All Tokens Matter Equally?

So far, we know that KV Cache stores every token forever.

But does the model actually care about every token?

Imagine the conversation contains:

My name is Ankit.
I live in Mumbai.
I work in AI.
I love Machine Learning.

If the user asks: "What is my name?"

The model mostly cares about the tokens "Ankit" and "name". Tokens like "Mumbai", "AI", and "work" have almost nothing to do with the question.

Think of attention as a spotlight:

My                 ██
name               ███████
is                 ███
Ankit              ███████████████████
I                  █
live               █
Mumbai             █

The Big Observation

Even though the KV Cache contains all tokens, the model only heavily uses a small subset of them.

A token being present in the context does not mean it is equally important for future predictions.

Step 4: How SnapKV Solves the Memory Crisis

SnapKV starts from this surprisingly simple observation:

If a token is almost never attended to, storing its Key and Value forever is wasteful.

Instead of keeping every token, SnapKV asks:
Can we identify which tokens are likely to matter later and discard the rest?

The First-Principles Insight

SnapKV asks a specific question. Not “Which token received attention once?”, but rather:

Which tokens keep receiving attention over and over again?

If future tokens continue to look at the token "Ankit" repeatedly, it means its Key and Value contain vital information. We must keep them!

On the other hand, if a token like "Mumbai" receives almost no attention from any future query, storing it simply wastes KV-cache memory.

Step 5: The SnapKV Algorithm

Think of SnapKV like a library. If you have 1,000 books but students repeatedly borrow only 20 of them, you wouldn’t keep all 1,000 on the front desk. You’d keep the popular ones nearby and move the rest.

Tokens that are repeatedly “checked out” by attention are kept. Tokens that nobody looks at are removed.

The SnapKV process looks like this:

Observe Attention: Let the model process tokens and see where the attention flows.
Compute Importance: Track how much total attention each token receives.
Sort Tokens: Rank them by their importance scores.
Keep Top-K Tokens: Retain the most frequently attended KV pairs.
Delete Remaining: Drop the rest from memory.

Step 6: Pruning and the Visual Proof

When you map out the attention scores visually over a long context, a clear pattern emerges.

Most tokens show tiny blips of attention. But a few tokens—names, entities, system instructions—create MASSIVE SPIKES. These are the “Important Anchors” that the model constantly refers back to.

By identifying these spikes and deleting the flatline tokens, we can compress the KV Cache significantly.

But What if a Deleted Token Becomes Important Later?

Researchers measured this exact scenario. Surprisingly, the Quality Drop is very small, while the memory savings are massive. Because attention repeatedly focuses on those same anchor tokens anyway, dropping the noise barely affects generation quality.

Step 7: Visualizing the Matrix Compression

Let’s look at the actual matrix mechanics of what SnapKV does.

In the raw attention matrix, bright spots represent high attention, and dark areas represent low attention. SnapKV translates this directly into a pruning strategy.

ORIGINAL KV CACHE
K Shape: [1, 12, 26, 64]
V Shape: [1, 12, 26, 64]

COMPRESSED KV CACHE
K Shape: [1, 12, 7, 64]
V Shape: [1, 12, 7, 64]

We’ve reduced the sequence length from 26 tokens to just 7 tokens, keeping only what truly matters.

Mental Model Summary

Traditional KV Cache: Store the entire textbook.
SnapKV: Store the summary notes + a few important pages.

It’s not perfect, but it’s drastically smaller and nearly as effective. We simply observed the attention scores, computed importance, kept the Top-K tokens, and pruned the cache.

Here is the GitHub repo that you can check out: GitHub Repository

Comment down your thoughts below, if you found it helpful!

References

SnapKV: LLM Knows What You are Looking for Before Generation

Best,
Ankit

Understanding LLM.int8(): Making Large Models Fit

2026-06-12T00:00:00Z

Here is the GitHub repo that you can check out: GitHub Repository

LLM.int8() from First Principles. Most people understand that large language models have billions of parameters, but don’t know exactly how we compress them to run on normal GPUs without destroying their intelligence.

To understand how modern models can fit on consumer hardware, we need to understand Quantization and specifically LLM.int8().

It essentially solved the 8-bit inference bottleneck.

But before we get into the solution, we must understand the core problem.

Step 1: The Memory Gets Too Big

When a model processes text, its intelligence is stored inside billions of weight values.

Historically, most neural networks use FP32 (32-bit floating point). That means every single number occupies 4 bytes of memory.

For a tiny 4-parameter matrix, this is 16 bytes. No problem.

But look at how the memory requirements scale for modern LLMs:

Model Size	Precision	Memory Required (Weights Only)
7 Billion	FP32	~28 GB
70 Billion	FP32	~280 GB

Notice: Just the weights of a 70B model require 280 GB of VRAM. Most consumer GPUs cannot even hold a fraction of this memory.

The Rule:

If your model grows, the memory required explodes.

Here is the craziest part: Do we really need 32 bits for every single parameter?

This is exactly the question LLM.int8() asks.

Step 2: The Quantization Question

Instead of storing every weight using 32 bits, we can store them using fewer bits, like INT8 (8 bits).

By moving from FP32 (4 bytes) to INT8 (1 byte), we immediately reduce the memory by 4x. A 280 GB model becomes 70 GB.

But an INT8 value cannot directly store floating-point numbers such as 0.12 or 0.91. INT8 only stores integers from -128 to 127.

So we store an integer representation and a scaling factor:
0.56 -> 56 with a scale of 0.01.

This is called Naive Quantization.

Step 3: Why Naive Quantization Fails

Until now, quantization looked almost magical. We get 4x less memory with a tiny error. So what’s the catch?

Suppose our weights are [0.12, 0.56, 0.91, 0.33, 100].

Notice the 100. It is an outlier.

Because quantization maps the largest value to the maximum INT8 value (127), the scale becomes 100 / 127 ≈ 0.787.

This single giant value forces a massive scale factor. The smaller values (like 0.12) are completely crushed and lose precision.

The Outlier Problem

In large language models, a tiny fraction of values are orders of magnitude larger than the rest. If we force everything into INT8, these outliers stretch the scale so much that the model loses its intelligence. Researchers tried removing the outliers, but the model quality immediately dropped.

Step 4: The Magic Trick - LLM.int8() Solution

Instead of sending every value through INT8 quantization, LLM.int8() separates the values into two distinct groups:

Normal Values: [0.12, 0.56, 0.91, 0.33]
Outlier Values: [100]

The normal values are processed in INT8. They quantize very well because they live in a similar numerical range.

The outlier values are kept in higher precision (FP16). This avoids introducing large quantization errors.

By treating these two groups differently, we never force the normal values to compete with the giant outlier for the quantization scale. We get nearly the memory savings of INT8 with the numerical accuracy of FP16!

Step 5: Split Matrix Multiplication

In practice, Transformers spend their time performing massive matrix multiplications (Activation Vector × Weight Matrix).

LLM.int8() performs two separate, smaller matrix multiplications:

INT8 Path: Computes the contribution from normal dimensions.
FP16 Path: Computes the contribution from outlier dimensions.

Once both paths finish, their outputs are perfectly added together.

This final output is extremely close to the original FP32 computation while using significantly less memory. Use INT8 wherever possible and keep higher precision only where it is actually needed.

Step 6: The Benchmark Results

When we benchmark the FP32 vs INT8 models for a standard architecture, the results speak for themselves:

By applying LLM.int8(), we save 75% memory while introducing only a small reconstruction error. When testing generation, the model still understands English, grammar, and context beautifully.

Applying it in Code

Here is how you can use the HuggingFace transformers and bitsandbytes libraries to instantly leverage LLM.int8() for any model:

!pip install transformers
!pip install accelerate
!pip install bitsandbytes

from transformers import AutoModelForCausalLM
from transformers import AutoTokenizer

model_name = "gpt2"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(model_name)

print(model)

GPT2LMHeadModel(
  (transformer): GPT2Model(
    (wte): Embedding(50257, 768)
    (wpe): Embedding(1024, 768)
    (drop): Dropout(p=0.1, inplace=False)
    (h): ModuleList(
      (0-11): 12 x GPT2Block(...)
    )
    (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
  )
  (lm_head): Linear(in_features=768, out_features=50257, bias=False)
)

Checking Memory

total_params = sum(
    p.numel()
    for p in model.parameters()
)

memory_mb = (
    total_params * 4
) / (1024 * 1024)

print(f"Total Parameters: {total_params:,}")
print(f"Estimated FP32 Memory: {memory_mb:.2f} MB")

Total Parameters: 124,439,808
Estimated FP32 Memory: 474.70 MB

Running Inference

text = "My name is"

inputs = tokenizer(
    text,
    return_tensors="pt"
)

output = model.generate(
    **inputs,
    max_new_tokens=20
)

print(tokenizer.decode(output[0]))

My name is John. I'm a man of God. I'm a man of God. I'm a man

Loading Int8 Version

from transformers import BitsAndBytesConfig

# Creating Configuration 
bnb_config = BitsAndBytesConfig(
    load_in_8bit=True
)

# Loading Model
int8_model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config
)

This line: load_in_8bit=True is literally everything we learned.

Now Comparing Memory

total_params = sum(
    p.numel()
    for p in int8_model.parameters()
)

fp32_memory = (total_params * 4) / (1024 * 1024)
int8_memory = (total_params * 1) / (1024 * 1024)

print(f"FP32 Estimate: {fp32_memory:.2f} MB")
print(f"INT8 Estimate: {int8_memory:.2f} MB")
print(f"Reduction: {fp32_memory/int8_memory:.1f}x")

FP32 Estimate: 474.70 MB
INT8 Estimate: 118.68 MB
Reduction: 4.0x

Generating Text Again

output = int8_model.generate(
    **inputs,
    max_new_tokens=20
)

print(tokenizer.decode(output[0]))

My name is John. I'm a writer, and I'm a writer. I'm a writer. I'm

Conclusion at the End:

Before INT8

My name is John. I’m a man of God. I’m a man of God. I’m a man

After INT8

My name is John. I’m a writer, and I’m a writer. I’m a writer. I’m

Model still understands English
Model still understands grammar
Model still understands context
There is a tiny error at the end, which is what quantization is all about!

import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np

# Plotting the memory comparison
plt.figure(figsize=(8, 5))
models = ['FP32 Model', 'INT8 Model']
memories = [fp32_memory, int8_memory]

plt.bar(models, memories, color=['#64748b', '#3b82f6'], edgecolor='black', width=0.5)
plt.title('HuggingFace GPT-2 Memory Footprint', fontsize=16, fontweight='bold', pad=15)
plt.ylabel('Memory (MB)', fontsize=12)

for i, v in enumerate(memories):
    plt.text(i, v + (max(memories)*0.02), f'{v:.2f} MB', ha='center', fontweight='bold', fontsize=12)

plt.tight_layout()
plt.show()

Here is the GitHub repo that you can check out: GitHub Repository

Comment down your thoughts below, if you found it helpful !

References

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

Best,
Ankit

Understanding Flash Attention: Making Transformers Fast

2026-06-09T00:00:00Z

Here is the GitHub repo that you can check out: GitHub Repository

Flash Attention from First Principles. Most people understand Self-Attention, but don’t know exactly why it is so incredibly slow for long documents.

To understand how modern models can process millions of tokens at once, we need to understand Flash Attention.

It essentially solved the quadratic memory bottleneck.

But before we get into the solution, we must understand the core problem.

Step 1: The Matrix Gets Too Big

When a model processes text, it compares every single word to every other word using the Self-Attention mechanism.

This creates a massive Attention Matrix. Look at how the math problem scales:

Number of Words	Math Problem Size (Attention Matrix)
1,000 words	1,000,000 numbers
2,000 words	4,000,000 numbers
4,000 words	16,000,000 numbers
8,000 words	64,000,000 numbers

Notice: Every time you double the words, the memory you need multiplies by 4. This gets out of hand very fast.

The Math Term: O(N²) (Quadratic Scaling)

The Rule:
If your sequence gets longer, the Attention Matrix explodes. You need an insane amount of memory.

Here is the craziest part: Why are we forcing the GPU to create a gigantic 64-million-number matrix, if we are just going to throw it away a second later?

Step 2: The FlashAttention Question

Standard Attention creates 64,000,000 numbers, saves them to the GPU’s slow main memory (HBM), reads them back to do the Softmax math, and then deletes them.

Moving all those numbers back and forth is the real reason Transformers are slow. FlashAttention asks: Can we skip the slow memory completely?

Step 3: The Old Matrix vs. The Chunked Matrix

To solve this, we can introduce Tiling (Chunking).

Standard Attention:
Builds the massive matrix all at once in the slow memory.

Tiled Attention:
Does the math in pieces. It does a small piece, then another small piece, and so on. All of this fits inside the ultra-fast memory (SRAM).

The Important Rule

We are not skipping any steps.
We are still comparing every single word against every other word. We are just doing it in small, manageable batches instead of one giant gulp. The final mathematical answer will be exactly the same.

Why does chunking matter so much?

The GPU’s fast memory (SRAM) is incredibly fast, but very small. By doing the math in chunks, we keep all the work inside the fast memory. We avoid the brutal speed limits of the massive, slow main memory.

Step 4: But Wait… Tiling Breaks Softmax

We are going to prove why we can’t just easily chop the math into small pieces. Normal Softmax is not the same as Tiled Softmax.

The Golden Rule of Softmax

Imagine we have raw scores for one word looking at four other words: [2, 5, 1, 8]

Normal Attention:
The Softmax function looks at all four numbers at the exact same time. It figures out the percentages based on the whole group. The percentages will always add up to exactly 100% (or 1.0).

The Chunking Mistake

Suppose we decide to be clever and chop the scores into smaller chunks (tiles) so they fit in the GPU’s super-fast memory.

Tile 1: [2, 5]
Tile 2: [1, 8]

It feels logical to do Softmax on Tile 1, then Softmax on Tile 2, and add them together. But watch what happens.

The 200% Bug

Because we chopped the keys, the Softmax in each chunk doesn’t know about the other chunk. It forces each tiny chunk to equal 100% all by itself.

The Correct Answer:
If we do Softmax([2, 5, 1, 8]) all at once, we get [0.002, 0.047, 0.001, 0.950].
If you add those up, you get 1.0 (100%). Everything is perfect.

How It Breaks Chunk by Chunk:

Tile 1:
We ask Softmax to process [2, 5]. It thinks 5 is the biggest number ever, so it gives it 95%. The result is [0.047, 0.953]. (Adds up to 100%).

Tile 2:
We ask Softmax to process [1, 8]. It gives 8 almost everything. The result is [0.001, 0.999]. (Adds up to 100%).

The Trap:
If we stick those answers back together: [0.047, 0.953, 0.001, 0.999].
Add them up, and the sum is 2.0!

We just distributed 200% attention. The math is completely broken. Because Tile 1 couldn’t see the massive 8 in Tile 2, it gave the 5 way too much importance.

Step 5: The Magic Trick - Online Softmax

We just saw that chopping Softmax into chunks causes a massive bug. So how do we fix it?

The FlashAttention Goal:
How can we process the numbers in small, fast chunks, but still get the exact same math as if we looked at all the numbers at once?

Instead of needing to see every single number at the same time, we use a clever math trick called Online Softmax. We only need to remember two numbers as we process the chunks:

The Maximum: What is the biggest score we have seen so far?
The Sum: What is the running total of our calculations?

By keeping track of just these two tiny numbers, we barely use any memory.

Why is it called “Flash” Attention?

Because the “Maximum” number constantly updates (or “flashes”) as we process new chunks. When we find a new biggest number, the algorithm uses a smart formula to reach back and instantly fix the percentages of the old chunks.

We never have to save the big matrix!

Step 6: The Flash Attention Forward Pass

The Old Way (Standard Attention)

The standard way is terrible because it relies on the slow main memory (HBM):

Move Q and K from slow memory to fast memory.
Do the math. Write the giant result back to slow memory.
Read the giant result from slow memory to do Softmax. Write it back.
Read it again to multiply by V. Write the final answer.

All we are doing is moving data back and forth. It is slow.

The New Way (FlashAttention)

FlashAttention does everything in one smooth motion using the super-fast memory (SRAM):

Load a Chunk: Bring a small piece of Q, K, and V into fast memory.
Do the Math: Multiply them together.
Update the Trackers: Update the Running Maximum and Running Sum.
Update the Output: Keep building the final answer.
Throw the Chunk Away: Clear the fast memory and load the next piece.

We never write the giant middle step to the slow memory. This is why it is so fast!

Step 7: The Benchmark Results

When we benchmark Flash Attention against standard attention, the results are massive. The larger the sequence length, the more standard attention slows down and runs out of memory, while Flash Attention maintains steady performance.

FlashAttention was the turning point that allowed us to move from 4K context windows to 128K, 1M, and beyond.

Here is the GitHub repo that you can check out: GitHub Repository

Comment down your thoughts below, if you found it helpful !

References

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Best,
Ankit

Transformer Architecture: From First Principles

2026-06-08T00:00:00Z

In our previous article, i broke down Self-Attention into its simplest mathematical form, discovering how Query, Key, and Value matrices route contextual information between words.

But Self-Attention is just an engine. To build a system capable of translating languages or generating code, you need the complete vehicle: The Transformer Architecture.

A Transformer contains many additional components: Positional Encoding, Multi-Head Attention, Feed-Forward Networks, Residual Connections, Layer Normalization, Masking, Cross-Attention, and the final prediction layers.

Instead of memorizing blocks from an architecture diagram, we’ll build the Transformer from first principles, understanding why each component exists, what problem it solves, and how the mathematics works underneath.

If you’re new to Self-Attention, I highly recommend reading the previous article first:
Self Attention: from first principle

The Core Bottleneck: Why Invent Transformers?

Before 2017, Sequence-to-Sequence (Seq2Seq) Recurrent Neural Networks (RNNs) were the absolute rulers of AI. RNNs processed text strictly sequentially:

"My" ──► [Step 1] ──► "name" ──► [Step 2] ──► "is" ──► [Step 3] ──► "Ankit"

If an RNN processed a 500-word paragraph, the context of the first word had to survive 499 mathematical updates before reaching the end. Information degraded rapidly. Gradients vanished. This was the Encoder Bottleneck.

The Transformer abandoned recurrence entirely. It ingests the entire sentence simultaneously, replacing time-stepping loops with parallel mathematics.

The Complete End-to-End Architecture Flow

[ Input Sentence ]

↓

Embedding
+ Positional Encoding

↓

[ Output So Far ]

↓

Embedding
+ Positional Encoding

↓

ENCODER

Multi-Head Self-Attention

↓

Add & Norm

↓

Feed-Forward Network

↓

Add & Norm

Context Matrix
(Z)

── K, V ──►

DECODER

Masked Self-Attention
(can’t see future words)

↓

Add & Norm

↓

Cross-Attention
← receives K, V from Encoder

↓

Add & Norm

↓

Feed-Forward Network

↓

Add & Norm

↓
Linear Projection
↓
Softmax (Probabilities)
↓
[ Next Word Output ]

Let’s break down exactly what happens to our data at every single stage of this pipeline.

Phase 1: Preparation (Tokenization-Embeddings-Positional Encoding)

When we pass the sentence "My name is Ankit" into the model, neural networks cannot read text. They only understand continuous numbers.

Step 1.1: Tokenization & Embeddings

The sentence is broken into IDs and mapped into a continuous representation matrix $X$.

$$\text{“My name is Ankit”} \ \longrightarrow \ [523, 1024, 98, 4567] \ \longrightarrow \ X \in \mathbb{R}^{4 \times d_{\text{model}}}$$

Step 1.2: Positional Encoding

Because the Transformer ingests all 4 tokens simultaneously, it has zero concept of word order. To the raw network, “My name is Ankit” and “Ankit is my name” are mathematically identical.

To fix this, we fuse a mathematical watermark—a Positional Encoding Vector—directly into our embeddings.

  [0.12,  0.33]  // Semantic Meaning ("My")
+ [0.84,  0.11]  // Sine Waveform (Position 1)
────────────────
= [0.96,  0.44]  // Position-Aware Vector

By adding specific sine and cosine wave frequencies, the model naturally learns to separate the underlying meaning of the word from its physical coordinate in the sentence.

Phase 2: The Encoder (Building Deep Understanding)

Our position-aware matrix $X$ now enters the Encoder. The Encoder’s solitary job is to analyze the input sentence and construct a deeply interconnected representation of it.

Step 2.1: Multi-Head Self-Attention

↳ Deep Dive: Read Self-Attention from First Principles

Language is deeply complicated. Ever wondered how the exact same sentence in English can have multiple interpretations?

“She saw the man with the telescope.”

By looking at this, a Transformer must generate not just one, but multiple different attention masks:

Dependency 1: (saw → man, man → telescope) — The man has the telescope.
Dependency 2: (she → telescope, saw → man) — She is using the telescope to see the man.

With one attention head, the model can focus on only one type of relationship at a time. But language has many relationships simultaneously.

The Limitation of a Single Head
Consider the sentence: "I love Apple Phone"
Things happening at once:

Apple → Phone (Brand / Tech)
Love → Apple (Sentiment)
I → Love (Subject - Verb)
Word Order (Positional structure)

A single attention head must average all of these into one focus. That loses information.

The Solution: Multi-Head Attention
Multi-Head Attention = Multiple independent attention mechanisms running in parallel.

Each head:

Sees the same sentence.
Uses its own learned projections ($W^Q, W^K, W^V$).
Focuses on a different relationship.

Think of it as multiple viewpoints looking at the exact same data. Each word embedding is copied into multiple heads, so each head creates its own independent geometry.

Head 1: Focuses on syntax.
Head 2: Focuses on semantics.
Head 3: Focuses on entities.
Head 4: Focuses on long-range relations.

No one tells them this explicitly. Training discovers it.

Why Splitting into Heads Helps (Important Intuition)
Instead of doing one massive 512-dimensional attention pass, we do 8 heads $\times$ 64-dimensional attention.

Why don’t all heads just learn the exact same thing?

Different random initialization: At the start, all $W$ matrices are random. They are literally looking in different directions in vector space.
Dimensional Bottleneck: Because each head only gets 64 dimensions, it cannot memorize everything. It is forced to specialize.

What Happens During the First Forward Pass?
Sentence: “Apple released a new phone and people love it.”

Because the weights are random, the early attention patterns are accidental:

Head 1 Attention: Apple → released → phone → it (Nearby words)
Head 2 Attention: love → people → it (Emotion-heavy words)
Head 3 Attention: Apple → phone (Capitalized / Noun words)

What Happens During Backpropagation?
Suppose the model makes a mistake in understanding entity reference (failing to realize “it” refers to “phone”). Loss is high.

Backprop asks: “Which parameters most influenced the wrong output?”

Each head gets different gradients because their outputs and contributions were different:

Head 2 (Sentiment) helped prediction a bit. Gradient reinforces emotional alignment.
Head 3 (Entities) helped entity resolution. Gradient strongly reinforces entity links.
Head 1 (Nearby words) didn’t help much. Gradient weakens or pushes it elsewhere.

Over millions of examples, gradients strengthen useful patterns and weaken useless ones. Just like CNNs naturally learn filters for edges and textures, different heads capture different aspects of the sentence.

The Linear Transformation ($W_O$)
Each head produces an output vector. All head outputs are concatenated and passed through one final linear layer ($W_O$). The job of $W_O$ is to make sure only the relevant head information is passed forward into the final combined representation.

Step 2.2: The “Add & Norm”

Immediately after attention routing, the mixed output passes through an Add & Norm stabilization layer.

$$\text{Output} = \text{LayerNorm}\Big(\text{Attention}(X) + X \Big)$$

1. The “Add” (Residual Connection) — Preventing Information Loss
We physically add the original input $X$ directly back to the Attention output vector.

  [ 0.22, 0.44 ]  // Processed Attention Output
+ [ 0.10, 0.80 ]  // Original Input X (Uncorrupted)
────────────────
= [ 0.32, 1.24 ]  // Bypassed Residual Vector

Why is this necessary?
When a Transformer is initialized, its weights are completely random. When you pass an input vector through multiple matrix multiplications, it gradually becomes mathematically distorted.

In the first few layers, this isn’t a huge issue. But as the network gets deeper, the original input starts getting severely corrupted. By the time it reaches the output layer, massive information loss has occurred.

If a specific attention layer fails to learn anything useful early in training, the raw features flow safely past it, structurally eliminating the Vanishing Gradient problem.

2. The “Norm” (Layer Normalization) — Stabilizing Training
Deep networks are notoriously unstable. Without normalization, some activation numbers grow exponentially large (Exploding Gradients), while others shrink to zero.

Why is this catastrophic for speed?
If you have unnormalized data with wild variance, you are forced to use a very Small Learning Rate to prevent the model from crashing. A smaller learning rate means the model will take a significantly longer time to converge to the minimum loss. Training becomes painfully slow.

Layer Normalization solves this by mathematically standardizing the variance within each token. Let’s see the exact internal computation:

// 1. Token Vector
x = [2.0, 4.0, 6.0, 8.0]

// 2. Compute Mean (μ)
μ = (2 + 4 + 6 + 8) / 4 = 5.0

// 3. Subtract Mean
x - μ = [-3.0, -1.0, 1.0, 3.0]

// 4. Divide by Standard Deviation (σ ≈ 2.23)
Normalized = [-1.34, -0.44, 0.44, 1.34]

By explicitly keeping the numbers tightly bounded around zero, the model avoids gradient explosions. This allows us to use a High Learning Rate, radically improving training speed and stability.

Step 2.3: Feed-Forward Network (The Concept Factory)

Self-Attention only routes information between tokens. It is fundamentally a linear operation. But to truly understand language, a model must be able to draw complex, non-linear boundaries.

The Feed-Forward Network (FFN) introduces Non-Linearity.

$$\text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2$$

Let’s look at the exact mathematical mechanics inside the Concept Factory:

// Step 1: Input vector (2-dim, simplified)
Input = [0.5, 0.3]

// W_1 projects it up to 5 dimensions (learned weighted sums)
Expanded = [4.2, -1.8, 3.1, -0.5, 8.9]

// Step 2: Apply ReLU  →  max(0, x)  →  Kill every negative
Before: [  4.2, -1.8,  3.1, -0.5,  8.9 ]
After:  [  4.2,    0,  3.1,    0,  8.9 ]  // ◄── -1.8 and -0.5 zeroed out

// Step 3: W_2 projects back down (5 → 3 dims)
// Each output is a dot product of ALL 5 ReLU values × a learned row of W_2

W_2 row 1 = [0.3, 0.1, 0.5, 0.2, 0.4]
Output[0]  = (4.2×0.3)+(0×0.1)+(3.1×0.5)+(0×0.2)+(8.9×0.4) = 1.26+0+1.55+0+3.56 = 6.37

W_2 row 2 = [0.2, 0.4, 0.1, 0.3, 0.2]
Output[1]  = (4.2×0.2)+(0×0.4)+(3.1×0.1)+(0×0.3)+(8.9×0.2) = 0.84+0+0.31+0+1.78 = 2.93

Final Output = [6.37, 2.93, ...]   // ◄── Back to standard dimension

$W_2$ is a learned matrix. Each output value is a dot product that mixes all 5 ReLU signals into a single summary number. The network learns exactly which combination of signals encodes useful concepts. That is how 2048 neurons get distilled back down to 512 dimensions.

(Note: The output of the FFN passes through a second Add & Norm layer before officially exiting the Encoder).

Phase 3: The Decoder (Generating the Answer)

The Encoder is finished. It outputs a highly dense, fully contextualized matrix called the Context Matrix ($Z$).

Now, the Decoder steps in. Its job is to take that Context Matrix ($Z$) and generate the output text autoregressively (one word at a time).

Step 3.1: Masked Self-Attention

↳ Deep Dive: Read Self-Attention from First Principles

Masked Multi-Head Attention is self-attention where a token is NOT allowed to look at future tokens.

Before we do the math, let’s understand the structural difference between the two halves of the Transformer:

Encoder: Sees the full input at once. Builds a global understanding. No masking.
Decoder: Generates output step by step. Uses masked self-attention to see only past outputs.

Why Do We Need Masking?
The Decoder’s job is to generate text one word at a time. Consider the sentence: "I love apple phone"

When predicting:

word 1 → "I"
word 2 → "love"
word 3 → "apple"
word 4 → "phone"

At the exact moment of predicting "apple", the model must NOT see "phone". Why? Because "phone" is the exact answer it is supposed to predict later!

Without Masking
During training, the Decoder processes all target words simultaneously for massive GPU efficiency. Suppose the Decoder sees the entire sequence "I love apple phone" at once. While predicting "apple", the attention mechanism would simply look ahead at "phone".

The model learns a lazy, catastrophic rule: “Just peek at the future to get the answer.”

At inference time—when you are actually chatting with the AI—future words do not exist yet. Because the model relied on cheating during training, it completely fails to generate anything useful.

The Math of Masking
To prevent cheating, we must enforce causality. The model applies a Mask to the raw $QK^T$ attention scores before applying Softmax. It explicitly overwrites all future coordinate scores with negative infinity ($-\infty$), creating a lower-triangular matrix:

      [ Word 1, Word 2, Word 3, Word 4 ]
Word 1 [   1.2,   -Inf,   -Inf,   -Inf ]
Word 2 [   0.4,    1.1,   -Inf,   -Inf ]
Word 3 [   0.9,    0.1,    2.2,   -Inf ]
Word 4 [   0.2,    0.5,    1.4,    0.8 ]

When this matrix passes through the Softmax function ($\frac{e^{x}}{\sum e^x}$), any value of $e^{-\infty}$ collapses exactly to $0$.

Future words are mathematically erased. Word 2 is only allowed to route information from Word 1 and itself. This strictly enforces causality, ensuring the model learns to truly predict the next word rather than simply copying it.

Step 3.2: Cross-Attention

↳ Deep Dive: Read Self-Attention from First Principles

Next, the Decoder must link its generated text with the original input sentence. It does this via Cross-Attention.

Unlike Self-Attention where $Q, K, V$ all come from the exact same sentence, Cross-Attention deliberately splits the source matrices.

The Math of the Bridge:

The Queries ($Q$) are generated from the Decoder’s current word.
The Keys ($K$) and Values ($V$) are pulled directly from the Encoder’s fully processed Context Matrix ($Z$).

  Decoder's generated text so far ──►  Generates Query (Q)
                                              │
  Encoder's Source text (Matrix Z) ──►  Generates Key (K) & Value (V)

First-Principle Example:
Suppose we are translating “I love Apple” to French. The Decoder has already generated “Je”. Now it must generate the next word.

The Query: The Decoder projects “Je” into a Query vector ($Q$). Mathematically, it asks: “I am a first-person pronoun. What context should I attach to?”
$Q_{\text{Je}} = [0.8, 0.2]$
The Keys: The Encoder’s Context Matrix contains the deeply processed Keys ($K$) for “I”, “love”, and “Apple”.
$K_{\text{I}} = [0.9, 0.1]$
$K_{\text{love}} = [0.1, 0.8]$
The Matching (Dot Product): The Query computes the dot product multiplication against all Encoder Keys to measure geometric similarity.
```
Q("Je") • K("I")    = (0.8 × 0.9) + (0.2 × 0.1) = 0.74  (High Alignment)
Q("Je") • K("love") = (0.8 × 0.1) + (0.2 × 0.8) = 0.24  (Low Alignment)
```
It discovers a massive mathematical alignment with $K_{\text{I}}$.

The Value (Softmax + Weighted Blend): The raw scores are converted to attention weights via Softmax, then used to blend all Value vectors together:

// Softmax on dot product scores
e^0.74 = 2.10  →  2.10 / 3.37 = 62%  weight on V("I")
e^0.24 = 1.27  →  1.27 / 3.37 = 38%  weight on V("love")

// Value vectors (simplified)
V("I")    = [1.0, 0.0]
V("love") = [0.0, 1.0]

// Final output = weighted blend
Output = 0.62 × [1.0, 0.0]  +  0.38 × [0.0, 1.0]
       = [0.62, 0.38]   // ◄── Carries 62% "I" meaning + 38% "love" meaning

The Decoder does not blindly pick one word. It blends the meaning of every source token, weighted by alignment strength. This [0.62, 0.38] context vector is then passed into the FFN.

Cross-Attention is a translator computing dot products against a dictionary. The Decoder maps its current state onto a weighted semantic blend of the entire source sentence.

After Cross-Attention, the Decoder runs the vectors through its own Feed-Forward Network and Add & Norm layers.

Phase 4: Output Translation (Linear & Softmax)

We are translating “I love Apple” to French. The Decoder has generated “Je”. Now the final layers must select the next French word.

The final context vector is pushed through a Linear Layer that scores every word in the 50,000-word French vocabulary. Each score is called a logit.

// Step 1: Raw Logits from the Linear Layer
aime  = 4.5    // "love" in French  ← likely next word
suis  = 2.1    // "am" in French
mange = 1.2    // "eat" in French
...            // 49,997 more words with tiny scores

// Step 2: Exponentiate every logit  e^x  (makes all values positive)
e^4.5 = 90.01   // aime
e^2.1 =  8.16   // suis
e^1.2 =  3.32   // mange
Sum   = 101.49  // (real sum spans all 50,000 words)

// Step 3: Divide each by Sum → final probability for every word
aime  =  90.01 / 101.49 = 89%  // ◄── Winner. Model outputs "aime"
suis  =   8.16 / 101.49 =  8%
mange =   3.32 / 101.49 =  3%

The model outputs "aime" with 89% confidence. The Decoder’s input is now "Je aime". On the next loop it predicts the French word for Apple.

This loop runs until the model outputs <EOS> (End Of Sequence), terminating generation. The final translation: “Je aime Apple”.

Summary of the Engine

Block	Mathematical Purpose
Positional Encoding	Fuses spatial time-step coordinates directly into the semantic vectors.
Multi-Head Attention	Analyzes the sequence from multiple structural perspectives in parallel.
Add & Norm	Bypasses failing layers to preserve gradients and stabilizes numerical variance.
Feed-Forward	Constructs higher-order concepts from the routed contextual evidence.
Masked Attention	Zeroes out future coordinates to prevent the decoder from cheating.
Cross-Attention	Routes the decoder’s current progress directly to the encoder’s source understanding.

This article i wrote specially for myself to grasping the entire concept in more better way and try to not forget.

Comment down your thoughts below, if you found it helpful !

References

Attention Is All You Need

Best,
Ankit

Understanding Self-Attention: From First Principles

2026-06-07T00:00:00Z

Self-Attention from First Principles. I have read a lot of articles and watched many videos, and I noticed that most people don’t explain this incredibly important topic in the easiest manner.

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Most resources require you to be good at Deep Learning and its mathematics to fully grasp the concepts.

But Self-Attention is actually simple if learned the right way.

At its core, Self-Attention relies on just three foundational components:

Dot Product (The Matching Tool)
Weights (The Learnable Transformation)
Data (The Values to Transmit)

Self-attention was the core innovation that enabled the Transformer architecture, which ultimately transformed modern AI, that we see today.

I won’t start with abstract formulas, instead,

I will break things down in the simplest way possible by giving you raw intuition first, building up to the complete mathematical matrix.

Want visuals? Check out The Illustrated Transformer By Jay Alammar.

Want code? Check out Self-Attention From Scratch By Sebastrian Raschka.

Want intuition first? Keep reading.

Let’s begin.

Step 1: We Have a Sentence

"My name is Ankit"

The Tokenizer converts these words into numerical IDs:

[523, 1024, 98, 4567]

These numbers are just IDs with no inherent meaning yet. Think of them as simple mappings:

My     = 523
name   = 1024
is     = 98
Ankit  = 4567

They are just labels, exactly like a Student ID or an Employee ID.

Step 2: Embedding Layer

Now we need to add meaning. Suppose our embedding dimension is 2 for learning purposes. The model uses a large lookup table called an embedding table:

Token ID	Embedding Vector
523	$[0.12, \phantom{-}0.33]$
1024	$[0.51, -0.20]$
98	$[0.22, \phantom{-}0.78]$
4567	$[0.88, \phantom{-}0.12]$

Mapping our tokens to vectors gives us this structured representation:

Word	Dimension 1	Dimension 2	Vector Representation ($\vec{x}$)
My	0.12	0.33	`[0.12, 0.33]`
name	0.51	-0.20	`[0.51, -0.20]`
is	0.22	0.78	`[0.22, 0.78]`
Ankit	0.88	0.12	`[0.88, 0.12]`

Question: What does a vector like `[0.12, 0.33]` actually mean?

Answer: Nobody knows exactly. This is a very important concept to understand.

People often assume:

0.12 = person score
0.33 = noun score

This is not true. The vector represents a compressed meaning.

Step 3: Put All Embeddings Together

Next, we stack these individual vectors together into a single data matrix, which we call $X$. Let’s break down the assembly of this matrix line by line, mapping each word to its specific row.

First, we take the vector for My:

X =

[
  [0.12, 0.33]
]

Next, we stack the vector for name directly underneath:

X =

[
  [0.12, 0.33],
  [0.51, -0.20]
]

Then, we append the vector for is:

X =

[
  [0.12, 0.33],
  [0.51, -0.20],
  [0.22, 0.78]
]

Finally, we add the vector for Ankit to complete our sequence:

X =

[
  [0.12, 0.33],
  [0.51, -0.20],
  [0.22, 0.78],
  [0.88, 0.12]
]

To visualize this explicitly, we can view the finished matrix with its corresponding word rows:

X =

[
  [My],
  [name],
  [is],
  [Ankit]
]

[
  [0.12, 0.33],
  [0.51, -0.20],
  [0.22, 0.78],
  [0.88, 0.12]
]

Shape of X

Rows = 4: Because we have 4 tokens (My, name, is, Ankit).
Columns = 2: Because our embedding dimension is 2.

Therefore, the shape of our matrix is:

$$X \in \mathbb{R}^{4 \times 2}$$

Visualization

Think of each word as a point in an embedding space.

The coordinates of the point are the values inside the embedding vector.

For example:

My → [0.12, 0.33]
name → [0.51, -0.20]
is → [0.22, 0.78]
Ankit → [0.88, 0.12]

In this toy 2D example, each embedding can be visualized as a point on a graph.

An Important Realization

At this stage:

My does NOT know that name exists.
Ankit does NOT know that My exists.

Each token is completely independent. We only have Word Meaning, not Sentence Meaning yet.

What Problem Does Attention Solve?

Currently, we know:

Ankit = [0.88, 0.12]

But the vector for Ankit has no idea it is appearing inside the specific sentence “My name is Ankit”.

The model needs Ankit to learn that it is contextualized by and related to My, name, and is.

This is the exact motivation for Self-Attention.

Before Q, K, V: Raw Intuition

Let’s ask a fundamental question:

If I am the word “Ankit”, how can I determine which other words in this sentence are important to me?

A simple, intuitive approach is to compare my embedding with every other embedding using a mathematical operation called the dot product:

Ankit $\leftrightarrow$ My
Ankit $\leftrightarrow$ name
Ankit $\leftrightarrow$ is

This gives us our very first attention intuition: measuring similarity between words.

The Math: Computing Dot Products Manually

Let’s compute the similarity scores for the word Ankit on paper.

Our Embeddings Reference

My: [0.12, 0.33]
name: [0.51, -0.20]
is: [0.22, 0.78]
Ankit: [0.88, 0.12]

What is a Dot Product?

For two vectors $\vec{a} = [a_1, a_2]$ and $\vec{b} = [b_1, b_2]$, the dot product is calculated by multiplying corresponding elements and summing them up:

$$\vec{a} \cdot \vec{b} = a_1b_1 + a_2b_2$$

Dot Product 1: Ankit $\cdot$ My

$$[0.88, 0.12] \cdot [0.12, 0.33]$$

Step 1: $0.88 \times 0.12 = 0.1056$
Step 2: $0.12 \times 0.33 = 0.0396$
Step 3: $0.1056 + 0.0396 = 0.1452$

$$\text{Ankit} \cdot \text{My} = 0.1452$$

Dot Product 2: Ankit $\cdot$ name

$$[0.88, 0.12] \cdot [0.51, -0.20]$$

Step 1: $0.88 \times 0.51 = 0.4488$
Step 2: $0.12 \times (-0.20) = -0.0240$
Step 3: $0.4488 - 0.0240 = 0.4248$

$$\text{Ankit} \cdot \text{name} = 0.4248$$

Dot Product 3: Ankit $\cdot$ is

$$[0.88, 0.12] \cdot [0.22, 0.78]$$

Step 1: $0.88 \times 0.22 = 0.1936$
Step 2: $0.12 \times 0.78 = 0.0936$
Step 3: $0.1936 + 0.0936 = 0.2872$

$$\text{Ankit} \cdot \text{is} = 0.2872$$

Dot Product 4: Ankit $\cdot$ Ankit

$$[0.88, 0.12] \cdot [0.88, 0.12]$$

Step 1: $0.88 \times 0.88 = 0.7744$
Step 2: $0.12 \times 0.12 = 0.0144$
Step 3: $0.7744 + 0.0144 = 0.7888$

$$\text{Ankit} \cdot \text{Ankit} = 0.7888$$

Final Similarity Scores For “Ankit”

Compared With	Dot Product Score
Ankit	0.7888
name	0.4248
is	0.2872
My	0.1452

In this toy embedding space, Ankit is mathematically closest to itself, followed by name, is, and lastly My.

Why Does the Dot Product Measure Similarity?

Aligned Vectors: If two vectors point in a highly similar direction, their dot product yields a large positive value.
Opposite Vectors: If two vectors point in completely opposite directions, their dot product yields a negative value.
Orthogonal Vectors: If two vectors point at a $90^\circ$ angle to each other, their dot product is exactly $0$ (completely unrelated).

Large Positive Score $\rightarrow$ Similar direction

Near Zero Score $\rightarrow$ Unrelated

Negative Score $\rightarrow$ Opposite direction

The Problem With the Current Equation

If we calculate attention scores directly using our raw input embeddings, we are effectively doing a pure matrix multiplication:

$$\text{Score} = XX^T$$

This leads to three fundamental problems:

Problem 1: No Learnable Parameters

This is just a mathematical operation. With no learnable parameters, we can’t train it.

Take the sentence: "I love Apple Watch"

The model might not be able to understand that Apple Watch is a single entity. It may interpret Apple as a fruit and Watch as a clock or the verb see.

Problem 2: Fixed Geometry

Because attention simply asks: “Which vectors are already close to me in this fixed space?”, there is no learning, no adjustment, and no control.

The embedding space is created before attention (via word2vec, GloVe, or an embedding layer). This space already has its own bias:

Apple is close to Fruit
Apple is somewhat close to Phone
But the Fruit similarity is often stronger globally.

Problem 3: Symmetrical Similarity

In a simple dot product, the similarity between Apple and Phone is the exact same as Phone and Apple.
They will both be pulled together, shifting the overall distribution. We may want Apple to shift towards Phone, but we don’t necessarily want Phone to shift towards Apple.

The “Apple Watch” Problem: Visualized

Let’s say our embedding space looks like this:

Apple ──── Fruit   (very close)
  │
  │
  │
Phone              (farther away)

Now, self-attention computes similarity. The result? Apple attends more to Fruit meaning, even though Fruit is not in the sentence. The relation to Phone is weaker geometrically.

Why does this confusion happen?
Because attention says: “I trust the existing embedding geometry.” But that geometry was trained on the entire internet, not this specific sentence.

So:

It cannot reinterpret Apple.
It cannot shift meaning.
It cannot prefer Phone over Fruit.

Without learnable parameters, attention cannot change meaning — it can only reveal existing similarity.

How Learnable Parameters Fix This

Learnable parameters change the geometry. Without them, geometry is fixed.

What $W_Q$, $W_K$, $W_V$ Really Are

They are learned transformations that say: “For Attention, this is what matters. Ignore the rest.”

Learnable parameters DO NOT add knowledge. They reshape space.

In a Transformer attention layer, the main learnable parameters are weight matrices:

$W_Q$ = Weights for Queries (How to ask)
$W_K$ = Weights for Keys (How to match)
$W_V$ = Weights for Values (What meaning to pass)
$W_O$ = Weights after attention (Output projection)

Each of these is just a matrix of numbers. For understanding attention geometry, weights matter much more than biases.

Each weight matrix mathematically:

Rotates space
Stretches space
Compresses space

Revisiting the Sentence With Learnable Parameters

Sentence: "I love Apple Phone"

What training teaches the model:
From millions of examples, it learns that when words like phone, iphone, android, or love appear, Apple should move toward their meaning.

So during attention, a geometric change happens.

Original Space:

Apple ──── Fruit   (closer)
  │
  │
 Phone

After Learning Projection:

Apple ──── Phone   (closer)
  \
   \
  Fruit (pushed away)

This is the entire difference.

Without learnable parameters, self-attention can only measure similarity.
With learnable parameters, self-attention can create similarity.

Learnable parameters don’t change geometry per sentence. They learn how geometry should change WHEN CERTAIN CONTEXTS APPEAR.

Intuition Towards Self-Attention

If the model predicts poorly on "I love apple phone", that means Apple did not attend strongly enough to Phone, as it is strongly similar to Fruit.

So during backpropagation, weight updating happens.

Learning says:

Increase alignment between Apple (Query) and Phone (Key).
Decrease alignment with irrelevant words.

How? By slightly changing the numbers inside $W_Q$ and $W_K$.

The Mathematical Intuition

The model makes a wrong guess $\rightarrow$ sees the error $\rightarrow$ slightly adjusts the geometry. Next time, similar sentences work better. That’s all learning is.

Let’s unpack this slowly:

Step 1: What the model is trying to learn

Input: "I love apple phone"
The model’s job is to understand Apple as a brand, not as a fruit.
At the beginning (with random weights), the geometry is bad. Attention connects Apple weakly to Phone and strongly to Fruit.

Step 2: What “Wrong” Means

During training, the model is usually asked to predict the next word or fill a masked word:

Input: I love apple: _______
Target: phone

If the model fails to predict phone, that means attention did not focus enough on the right relationship. It may have considered Apple as Fruit.

Step 3: Adjust Weights

Now we adjust our 3 matrices: $W_Q$, $W_K$, and $W_V$.

Step 4: Geometry Adjustment

The system thinks: “When apple appears with phone, I should make them look more similar.”

What does it do? It rotates the Query direction of Apple and rotates the Key direction of Phone.

So next time, their dot product is bigger, and their attention weight increases.

Under the hood, the system knows:

“Output was wrong.”
“Which parameters influenced that output.”

So it nudges them: “Increase this similarity a bit, decrease that similarity a bit.”
This is gradient descent—trial $\rightarrow$ feedback $\rightarrow$ correction.

Why This Works Over Time

One sentence teaches very little. But after 10 million sentences, the model slowly learns:

In tech contexts $\rightarrow$ Apple aligns with Phone, Laptop.
In food contexts $\rightarrow$ Apple aligns with Fruit.

The model never learns "Apple = Phone". It learns "WHEN Apple should look like Phone".

The Self-Attention Formula

Now that we understand the intuition behind learnable parameters, let’s look at the actual mathematical formula that ties everything together:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Where our queries, keys, and values are generated by multiplying our input $X$ with our learned weight matrices:
$$Q = X \cdot W_Q \quad \text{(Queries)}$$
$$K = X \cdot W_K \quad \text{(Keys)}$$
$$V = X \cdot W_V \quad \text{(Values)}$$

Let’s break down exactly what each piece of this formula is doing:

$QK^T$ (The Matching Engine): This is the row-by-row dot product multiplication. It is where every Query asks every Key: “How relevant are you to me?”
$\sqrt{d_k}$ (The Scaler): This is a safety buffer. $d_k$ is the dimension size of our keys. If our dimensions get too large, the dot product scores blast through the roof, which breaks the math during training. Dividing by $\sqrt{d_k}$ keeps the numbers stable.
$\text{softmax}(\dots)$ (The Normalizer): This is the activation function that turns our raw matching scores into probabilities (ranging from 0 to 1).
$V$ (The Information Carrier): Finally, we multiply those percentages by the actual data payload matrix ($V$) to extract the exact contextual meaning we need.

Now, let’s look at the raw matrix mechanics of just that first crucial step: the matching engine, $QK^T$.

The Math of $QK^T$: Step-by-Step Matrix Mechanics

Let’s watch the magic happen mathematically. Let’s look exclusively at the matrix multiplication $Q K^T$. We will forget Softmax and $V$ for a brief moment just to master how queries match keys.

Assume our text matrix has been projected into $Q$ and $K$ spaces (using 2-dimensional representations for simplicity):

Q (Query Matrix)

[
  [1, 0], // My Query
  [0, 1], // name Query
  [1, 1], // is Query
  [2, 1]  // Ankit Query
]

K (Key Matrix)

[
  [1, 1], // My Key
  [0, 2], // name Key
  [1, 0], // is Key
  [2, 1]  // Ankit Key
]

Both matrices have a shape of $(4, 2)$ because we have $4 \text{ tokens} \times 2 \text{ dimensions}$.

To perform a dot product of every row in $Q$ with every row in $K$, we transpose $K$ so it has a shape of $(2, 4)$:

Kᵀ (Key Matrix Transposed)

[
  [1, 0, 1, 2],
  [1, 2, 0, 1]
]

Now we compute the matrix multiplication: $Q_{(4 \times 2)} \times K^T_{(2 \times 4)} = \text{Score}_{(4 \times 4)}$. Why $4 \times 4$? Because every single token compares itself with every single token!

Computing Row 1: The “My” Token Perspective

We take the Query vector for My ($[1,0]$) and cross-multiply it with all word Keys in $K^T$:

My Query $\cdot$ My Key: $[1,0] \cdot [1,1] = 1(1) + 0(1) = \mathbf{1}$
My Query $\cdot$ name Key: $[1,0] \cdot [0,2] = 1(0) + 0(2) = \mathbf{0}$
My Query $\cdot$ is Key: $[1,0] \cdot [1,0] = 1(1) + 0(0) = \mathbf{1}$
My Query $\cdot$ Ankit Key: $[1,0] \cdot [2,1] = 1(2) + 0(1) = \mathbf{2}$

Row 1 results in: [1, 0, 1, 2]

Computing Row 4: The “Ankit” Token Perspective

We take the Query vector for Ankit ($[2,1]$) and cross-multiply it with all word Keys in $K^T$:

Ankit Query $\cdot$ My Key: $[2,1] \cdot [1,1] = 2(1) + 1(1) = \mathbf{3}$
Ankit Query $\cdot$ name Key: $[2,1] \cdot [0,2] = 2(0) + 1(2) = \mathbf{2}$
Ankit Query $\cdot$ is Key: $[2,1] \cdot [1,0] = 2(1) + 1(0) = \mathbf{2}$
Ankit Query $\cdot$ Ankit Key: $[2,1] \cdot [2,1] = 2(2) + 1(1) = \mathbf{5}$

Row 4 results in: [3, 2, 2, 5]

The Complete $QK^T$ Score Matrix

Filling out the entire matrix gives us an actionable relationship grid:

QKᵀ (Attention Score Matrix)

[
  [           My (K), name (K), is (K), Ankit (K) ],
  [ My (Q),        1,        0,      1,         2 ],
  [ name (Q),      .,        .,      .,         . ],
  [ is (Q),        .,        .,      .,         . ],
  [ Ankit (Q),     3,        2,      2,         5 ]
]

💡 The Deep Reading Intuition:
Always read this matrix row by row. The row belongs to the Query (the word asking the question), and the columns belong to the Keys (the tokens responding).
Looking at the Ankit row, its scores are [3, 2, 2, 5]. This means that while processing the word Ankit, the model determines it is highly self-relevant ($5$), but its strongest structural connection out of the remaining tokens is to the word My ($3$).

What Happens Next: Softmax and the Value Matrix ($V$)

Once we have our score matrix, we finish the rest of the standard attention pipeline:

1. Scaling and Softmax

We turn these raw score matrices into percentages. For example, applying Softmax to the Ankit row [3, 2, 2, 5] converts those numbers into normalized probabilities that equal $100%$:

Ankit Attention Weights: [ My: 12%,  name: 5%,  is: 5%,  Ankit: 78% ]

2. Gathering the Values ($V$)

Now that we know exactly how much attention to pay to each word, we multiply these percentage weights by the Value Matrix ($V$).

Instead of passing the original data or keys forward, we pull information directly from $V$. If a token has an attention score of $78%$ on itself and $12%$ on My, its final output vector will be a blended compound containing $78%$ of its own value profile and $12%$ of My’s value profile.

Comment down your thoughts below, if you found it helpful !

References

Attention Is All You Need

Best,
Ankit

Embracing the Emptiness

2025-12-17T00:00:00Z

Hey, how are you. i hope you’re happy and doing well. I don’t write very often. When I do, it’s usually unplanned more like reflections or random thoughts.

Lately, there’s been a lot of silence. Not the dramatic kind, just the kind that exists when things stop demanding attention. Nothing is particularly wrong. Nothing is particularly right either.

At some point, I noticed that emptiness isn’t always a problem to solve. Sometimes it’s just a phase where things are being rearranged quietly, without any announcement.

Life keeps moving as expected. Time moves, People stay buzy, Plans continue. Yet internally, there’s a neutral space where reactions slow down and thoughts become more deliberate.

I used to think emptiness meant something was missing.
Now it feels more like something unnecessary has left.

There are no big conclusions here. No lessons wrapped in optimism. Just an observation.

So, this is a snapshot shaped over time, written down on 17th December, late at night. Let’s see if the perspective shifts.

Why MCP (Model Context Protocol) is Here to Stay!

2025-04-25T00:00:00Z

You’ve likely heard the buzz about MCP (Model Context Protocol) — and I firmly believe it’s here to stay.

But what exactly is MCP, and why does it matter? Let’s break it down simply.

What is MCP?

MCP is a fundamental shift in the software development ecosystem. It introduces a new standard that redefines how AI Agents interact with external systems, making integrations seamless, reusable, and efficient.

Why Did We Need MCP?

Let’s say you’re building an AI agent that needs to talk to:

Gmail
A custom database

Traditionally:

You manually write integrations for each API.
You define authentication, permissions, and logic from scratch.
Adding another agent? You repeat the entire process.

The Inefficiencies of the Old Way

Redundant integrations per agent
Inconsistent API logic (e.g., Gmail allows email deletion, but your use case might only need sending)
No easy way to scale, since logic has to be redefined again and again

How MCP Fixes This

MCP introduces a unified abstraction layer, acting as a standardized protocol between AI agents and APIs. This eliminates the need for redundant integration logic.

It’s like having a central translator that speaks both API and AI-agent.

Before MCP:

Each AI agent has to handle API interactions separately.

With MCP:

AI agents simply communicate with the MCP server, which manages the requests for them.

The Result

Faster Development
Reusable APIs
Standardized Permissions
Agent-Agnostic Workflows

Final Thoughts

MCP isn’t just another buzzword — it’s a foundational layer for building scalable AI ecosystems.

Whether you’re building agents for customer service, health records, or internal tooling — MCP will help you avoid technical debt and grow faster.

If this topic intrigued you, feel free to connect with me or share this post with your network.

Accuracy Without Context Is Like a Ship Without a Compass

2025-04-21T00:00:00Z

Ah yes, the classic beginner’s mindset: if the accuracy number is high, the model must be a genius! Logic? What’s that?

Accuracy Without Context Is Like a Ship Without a Compass

Many beginners believe that higher accuracy automatically indicates better model performance. However, there’s much more to the story of machine learning when applied in the real world.

The Scenario

Imagine you’re building a multi-class classification model to predict three types of fruits: Apples, Bananas, and Cherries.

Here are the model’s accuracy scores after testing:

Apple: 80% accuracy
Banana: 80% accuracy
Cherry: 60% accuracy

Seems obvious, right? Cherry is the worst-performing class. But hold on! Your answer might be correct for your college project or any hackathon, but this is not how real-world ML works!

There’s more to this story.

What’s Missing? The Baseline

What you’re missing is a baseline.

Without a comparison, accuracy numbers alone can be misleading. In machine learning, the baseline is not just a reference point, it’s the context that defines our understanding of model performance. Without it, accuracy becomes an illusion, masking the true strengths and weaknesses of our systems.

Manually Labelling by Humans

In this scenario, let’s set the baseline by asking team members to manually label the same fruit images.

Here’s how they performed:

Apple: 85% accuracy
Banana: 100% accuracy
Cherry: 61% accuracy

What We Learn from the Baseline

Now things get interesting. Based on this baseline, it’s clear that:

The model performs close to humans on cherries (60% vs. 61%).
It struggles significantly with bananas (80% vs. 100%), even though 80% might seem fine at first glance.
For apples, it’s almost there, but not quite matching human performance.

What’s the Lesson Here?

Accuracy doesn’t tell the full story. You need to measure against a baseline, whether it’s human performance or a simple method like random guessing, before deciding where your model truly struggles.

Many ML beginners tend to focus on the lowest accuracy class as the problem area, but context matters. Without a proper baseline, you might be focusing on the wrong issues.

Final Thoughts

So, the next time you’re evaluating a model, remember: accuracy is only as good as the context it’s placed in.