There is only one core thesis: the essence of every optimization is to trade one resource dimension for another. Only once you can see the full list of resources and their physical carriers can you judge what a trick is really swapping, and when it should not be used at all.

I. Three mental models

Model 1: resources have only three dimensions

Any hardware resource has, in essence, only three consumable or contended dimensions:

Latency usually cannot be eliminated directly. It can only be hidden with concurrency, and that is the whole of Model 3.

Model 2: Roofline, where compute meets bandwidth

The ridge point is $\text{AI}^{} = \text{Peak FLOPS} / \text{Peak BW}$. Below it ($\text{AI} < \text{AI}^{}$) you are memory-bound; above it, compute-bound. The H100’s ridge is about $295$ FLOP/byte and the B200’s about $281$, barely moving over the years. Because compute grows faster than bandwidth, the ridge rises rather than falls, pushing more and more kernels into the memory-bound region. This is the quantitative statement of the “memory wall.”

Large square matrices have high AI and are compute-bound; thin-long matrices and small-batch decode have low AI and are memory-bound. LLM decode is inherently memory-bound: each token requires reading all weights once, so $\text{AI} \approx \text{batch size}$, and only a large enough batch enters the compute-bound regime. That is exactly the physical motivation behind continuous batching and speculative decoding.

Model 3: hide latency with concurrency (Little’s Law)

Multi-warp hides memory latency, multi-stream hides kernel gaps, async copy hides H2D/D2H, overlap hides collective latency, double buffering hides load latency, and pipelining hides cross-stage dependencies. They are all structurally identical, all a matter of “do B while waiting for A.” The key corollary is that hiding latency only pays off when the hidden thing is actually the bottleneck. If communication is only 5% of the time, hiding it into compute saves at most 5%, and it may even net-lose by contending for SMs and registers or by splitting kernels. Always profile the bottleneck’s share first, then decide whether to hide it.

II. Compute and storage units

CUDA Core vs Tensor Core. The former is a scalar ALU; the latter is a matrix MMA unit that computes a whole small tile per instruction, with one-to-two orders of magnitude higher throughput, born only for GEMM/convolution.

An SM also contains independent units that can run in parallel with the main compute, and these form the hardware basis for overlap:

The memory hierarchy is the optimization-space pyramid (higher means faster and smaller). Keeping data one level higher for one more reuse is the essence of most kernel optimization.

III. The full list of schedulable resources

The left column is what you schedule at the framework / training-stack level; the right is the physical carrier and its bottleneck signal.

A few easily overlooked points stand out. The CPU launch thread is a real bottleneck during decode, where each step issues tens to hundreds of tiny kernels, and that is the entire reason CUDA Graph exists. The Copy Engine and the SMs are separate hardware, so copying does not consume compute, which is the physical prerequisite for “copy while computing.” L2 can be actively pinned with cudaAccessPolicyWindow, so it is not a fully automatic black box. NVLink-C2C (900 GB/s coherent memory) is far faster than PCIe (64 GB/s), so offload strategy differs accordingly. SHARP lets the switch do in-network reduction, completing half of an AllReduce’s additions inside the network. Power is a whole-card shared budget, so a fully loaded GEMM triggers downclocking. The register file is a hard constraint on occupancy, since using too much per thread leaves fewer warps resident.

IV. The trick panorama: what each one trades

Arranged by acting layer, from low-level kernel to macro-level serving. The interesting columns are the last two: which bottleneck it removes, and when it is instead harmful.

Mapping back to the three models: bandwidth-savers (fusion / Flash / tiling / quantization) raise AI, pushing memory-bound kernels toward compute-bound; latency-hiders (stream / async / overlap / PP) are Little’s Law, valid only if the hidden thing is the bottleneck; launch/sync reducers (CUDA Graph, persistent kernels, sync elimination) remove CPU-side and synchronization serialization; compute-changers (quantization / Tensor Core / sparsity) raise peak FLOPS or lower the byte cost per FLOP; and distributed tricks (TP/PP/DP/EP/SP) trade communication for single-card capacity and compute.

V. Real vendor hardware (mid-2026)

A note on reading specs: compute figures here default to dense values. Vendor spec sheets often default to 2:4 sparsity, roughly double the dense number. NVIDIA’s “FP4 20 PFLOPS” for B200 is the sparse value, and the dense figure is about 9 to 10. Always align conventions when comparing across vendors.

NVIDIA data center

The H20 is counterintuitive: its compute is cut to about 15% of the H100, yet its 4 TB/s bandwidth is actually higher, so for memory-bound decode its value is not bad at all, a direct projection of the Roofline. Blackwell’s 5th-gen Tensor Core adds FP4/FP6 microscaling, and the 2nd-gen Transformer Engine manages low-precision scales automatically. The roadmap continues with Vera Rubin NVL144 (2026) at 3.6 EFLOPS FP4 inference and HBM4 at 13 TB/s, then Rubin Ultra NVL576 (2027) at 15 EFLOPS FP4.

NVIDIA workstation / consumer

Neither supports NVLink, so multi-card setups fall back to PCIe 5.0 (~64 GB/s one-way). For local TP/EP experiments, communication easily becomes the bottleneck, which is exactly what the resource map predicts.

Note: these notes are compiled from sources on the internet and are not my original work. I plan to rewrite them in my own words later.