CUDA Tile Optimization Slashes Flash Attention Performance
Large CUDA Tiles Reduce Flash Attention TFLOPS by 18‑43% Across Sequences
Bigger tiles should mean fewer memory accesses, faster attention, right? Not on NVIDIA GPUs. Across every sequence length tested, large CUDA tiles actually cratered Flash Attention throughput by 18 to 43 percent.
The culprit isn’t memory bandwidth; it’s compute inefficiency. Larger tiles amplify the cost of precise arithmetic: exponential functions, divisions, and register pressure that simultaneously nukes occupancy and stalls the machine. But the story doesn’t end with a tragic performance cliff.
A single switch, enabling flush-to-zero and approximate rounding in the exponential and division ops, rescues the large-tile approach, turning a 43% loss into a 20% gain. Here’s how tile size and math precision conspire to make or break Flash Attention on CUDA.
The result in TFLOPS are: Performance degraded by 18-43% across all sequence lengths. This is the trap, where large tiles make performance worse. - Compute bottleneck: With more elements per tile, inefficient operations (separate mul/add, precise math) become the bottleneck.
- Instruction overhead: More work per tile means more instructions before the next memory operation. Lesson: Tile size and compute efficiency are interdependent. Large tiles only help if the computation is efficient enough to keep up.
NCU insight (SeqLen=1,024, NVIDIA B200): - Registers/thread jump to 168 (+31%), reducing theoretical occupancy to 18.75% - Achieved occupancy drops to 16.5% - Compute throughput collapses to 17.4% (the trap) - Memory throughput falls to 7.4% - Grid size shrinks to 512 (fewer blocks from larger tiles) 2. The rescue with fast math One of the bottlenecks is special functions: exp2 (exponential) and truediv (division). By default, these are IEEE-754 precise--highly accurate, but slow.
For deep learning, we can trade a tiny bit of precision for massive speedups: Before (precise operations): p = ct.exp2(qk) alpha = ct.exp2(m_i - m_ij) acc = ct.truediv(acc, l_i) After (fast math): p = ct.exp2(qk, flush_to_zero=True) alpha = ct.exp2(m_i - m_ij, flush_to_zero=True) acc = ct.truediv(acc, l_i, flush_to_zero=True, rounding_mode=RMd.APPROX) What these flags do: flush_to_zero=True : Denormal numbers (extremely small values near zero) become exactly zero. This avoids slow microcode paths on the GPU.rounding_mode=RMd.APPROX : Skips iterative refinement after initial hardware approximation. With fast math, we've "rescued" the large tiles, and the results in TFLOPS are: We now match or exceed the small-tile baseline, with 10-20% gains for longer sequences.
NCU insight (SeqLen=1,024, NVIDIA B200): - Registers/thread: 168 (unchanged) - Theoretical/achieved occupancy: 18.75% / 16.6% (unchanged) - Compute throughput rebounds to 24.0% - Memory throughput improves to 12.9% 3.
Large tiles aren’t the problem. Inefficient math is. The moment you pair oversized tiles with IEEE‑754 precision, the GPU stalls, register pressure spikes, occupancy collapses, and compute throughput plummets.
The numbers don’t lie: 18‑43% slower across the board. But flip on `flush_to_zero` and `rounding_mode=APPROX`, and the same large tiles that were a liability become a lever. Performance rebounds, then surpasses the small‑tile baseline by 10‑20% on long sequences.
The lesson cuts deeper than Flash Attention. In CUDA, tile size and compute efficiency form a single lever. You cannot tune one in isolation.
Large tiles multiply the cost of every operation inside them. If that operation is precise but slow, you pay the penalty across thousands of elements. If it’s fast, fast enough to keep the execution units fed, the larger tile amortizes overhead and wins.
The GPU doesn’t care about your intention; it cares about instruction mix, register count, and occupancy. Fast math isn’t a hack. It’s the difference between a tile that chokes the SM and one that saturates it.
Every kernel designer should read the NCU counters the way a pilot reads a cockpit display. Registers per thread jumped 31%. Achieved occupancy fell to 16.5%.
Compute throughput collapsed to 17.4%. Those aren’t abstract metrics, they’re the GPU screaming that the current strategy is broken. The fix wasn’t shrinking tile size; it was unblocking the math.
The architecture rewards the bold, provided the bold are precise about where they sacrifice precision. For deep learning, the trade‑off is trivial. Denormals and iterative division refinements add no value to an attention matrix.
The model doesn’t feel the difference; the runtime does. This is the art of CUDA performance: knowing exactly which bits of accuracy you can discard to unlock the hardware’s full potential. Large tiles, fast math, and the courage to measure what matters.
That’s how you rescue Flash Attention, and every compute‑bound kernel that comes after it.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
Related Topic NVIDIA NeMo powers telco reasoning model for autonomous network workflows
Industry Applications
Related Topic China offers cheaper electricity to AI firms abandoning NVIDIA chips
Business & Startups
From Archives KV cache compaction cuts LLM memory 50×, chunked processing long contexts
From Archives AI system flags probable matches, narrows anonymous accounts to shortlist
From Archives Nvidia invests USD 4 B in photonics, taps Lumentum and Coherent optics for AI GPUs
Business & Startups
From Archives NVIDIA’s AODT Boosts 6G Development with Physics‑Accurate RAN Simulations
AI Tools & Apps
Common Questions Answered
How do large CUDA tiles impact Flash Attention's performance on NVIDIA GPUs?
Large CUDA tiles can actually degrade performance by 18-43% across different sequence lengths. This performance drop occurs because larger tiles create compute bottlenecks, with inefficient operations like separate multiply/add steps and precise math becoming performance constraints.
Why do larger tiles in Flash Attention not automatically improve computational efficiency?
Larger tiles introduce more instruction overhead and create computational inefficiencies that can throttle performance. As more elements are packed into each tile, the separate computational steps become less efficient, leading to a potential performance degradation instead of the expected performance gains.
What key lesson did researchers learn about tile size and compute efficiency in Flash Attention?
The study revealed that tile size and compute efficiency are deeply interdependent, and simply scaling up tile sizes does not guarantee better performance. Researchers discovered that large tiles only help if the computational characteristics can maintain efficiency, and naive scaling can quickly become a performance trap.
Further Reading
- Tuning Flash Attention for Peak Performance in NVIDIA CUDA Tile - NVIDIA Developer Blog
- Tuning Flash Attention for Peak Performance in NVIDIA CUDA Tile - NVIDIA Developer Forums
- How to Extend FlashAttention to Nearly Infinite HeadDim ... - FlashAttention GitHub
