Vision and Language
A double helix
Why redraw the map
[Read More]
Gated Attention
One switch after softmax
A simple modification ignored for eight years
[Read More]
AI Infra Resource Map
Every trick is a trade
There is only one core thesis: the essence of every optimization is to trade one resource dimension for another. Only once you can see the full list of resources and their physical carriers can you judge what a trick is really swapping, and when it should not be used at...
[Read More]
Why Decode Is Slow
Shape, not the KV cache
A widely misunderstood bottleneck
[Read More]
When Is SFT Done?
Reading the signals
In LLM post-training, SFT (supervised fine-tuning) usually handles cold-start, behavior formatting, and task-protocol learning. But longer SFT is not always better, and lower loss is not always better. Judging whether SFT is “done” hinges not on how well it fits the training set, but on whether continued supervised imitation still...
[Read More]
Attention Is Not Matmul Bound
Where the time really goes
If someone asks which part of attention is slowest, the obvious guess is the matrix multiplication. It has by far the most floating-point operations, so surely it must dominate the runtime. On a modern GPU, that intuition is wrong. The two matmuls in attention are simply not where the time...
[Read More]
Mixture of Experts
Experts are computation paths
The first time people meet Mixture of Experts, they instinctively assume that since it is called “experts,” surely each expert specializes in a different task. The reading is natural, but in a real MoE system it simply does not hold.
[Read More]
Speculative Decoding
Same output, much faster
Speculative decoding (also called speculative sampling) is one of the most elegant tricks in the LLM inference-acceleration toolbox. In one line, it lets a large model run at speeds that were previously unimaginable, all without sacrificing any accuracy.
[Read More]
The Temperature Knob
How LLM sampling really works
The last layer of a language model does not emit a probability. It emits logits, a vector of real numbers. A probability distribution only appears after we push those logits through a softmax. The entire secret of temperature lives in the step just before the softmax: we divide every logit...
[Read More]
MLE, a Unifying View
From regression to GPT
A large fraction of machine learning can be viewed as:
[Read More]
Intro to LLMs
Introduction to Large Language Models (LLMs)
[Read More]
But, what is Attention in transformers?
Q,K,V, with code snippets
‘All’ you need to know about transformer
[Read More]
Intro to Recurrent Neural Networks (RNNs)
Part I
Intro to Recurrent Neural Networks (RNNs)
[Read More]