NVIDIA CUDA Tile Primitives Speed Up Mamba-2

Jane Street's machine learning engineers spent part of their recent research cycle rewriting Mamba-2's forward pass in NVIDIA's new CUDA Tile programming model, known as cuTile. The trading firm, which leans on deep learning for quantitative research, worked directly with NVIDIA engineers to build and tune the kernels. CuTile itself is a Python-based DSL that lets developers write GPU code at the tile level rather than managing threads and warps by hand, with the compiler handling the work of mapping operations onto Tensor Cores and other hardware features.

Mamba-2 matters here because it sits at an awkward middle point in sequence modeling. Softmax attention trains fast thanks to matrix multiplication on Tensor Cores, but its compute and memory costs climb linearly with sequence length during inference. Recurrent networks go the other way: constant memory and compute regardless of sequence length, but training is slow and resistant to parallelization.

Mamba-2 tries to split the difference, keeping a fixed-size hidden state like an RNN while still aiming for throughput closer to attention-based models. Building that architecture efficiently took three custom kernels, each leaning on specific cuTile optimizations to get there.