性能

内存受限What does it mean to be memory-bound

内存受限的内核受限于 GPU 的内存带宽。

屋顶线图（如上图所示）可帮助识别程序性能是受计算能力、内存带宽还是其他因素的瓶颈影响。图表改编自 Williams, Waterman, and Patterson (2008)。

具体来说，这类内核的瓶颈在于 GPU 显存与流式多处理器 (Streaming Multiprocessor) 的本地缓存之间的带宽，因为 GPU 性能关注的问题通常涉及远超内存层次结构中任何更高级别的工作集大小。

内存受限的内核具有较低的算术强度（每移动一个字节的操作数更少），相对低于其屋顶线模型的脊点

从技术上讲，内存受限的定义仅针对单个内核，作为屋顶线模型的一部分，但稍作扩展后，它也可推广到涵盖构成典型工作负载的多个内核。

当代大语言模型推理工作负载在解码/输出生成阶段通常是内存受限的，此时权重必须在每次前向传播中加载一次。且这一过程每执行一次输出一个令牌，除非使用多令牌预测或推测解码技术。这使得计算内存受限的 Transformer 大语言模型推理中令牌之间的最小延迟（令牌间延迟或每个输出令牌的时间）易于计算。

假设模型有 5000 亿个参数，以 16 位精度存储，总计 1 TB。如果我们在单个 GPU 上运行推理，其内存带宽为 10 TB/s，我们可以每 100 毫秒加载一次权重，这就为我们的令牌间延迟设定了一个下限。通过将多个输入进行批处理，我们可以线性增加每个加载参数所执行的浮点操作数（算术强度），原则上直到达到计算受限点，且不会增加额外延迟，这意味着吞吐量随批大小线性提升。

有关 LLM 推理的更多信息，请参阅我们的 [LLM 工程师指南] (https://modal.com/llm-almanac/summary)。

英文原文(Modal GPU Glossary)

Kernels that are memory-bound are limited by the memory bandwidth of the GPU.

.](themed-image://roofline-model.svg)

Specifically, they are limited by the bandwidth between the GPU RAM and the local cache of the Streaming Multiprocessors, because the problems of interest for GPU performance generally have working set sizes much larger than any higher level of the memory hierarchy.

Memory-bound kernels have a lower arithmetic intensity (fewer operations per byte moved), relative to the ridge point of their roofline model.

Technically, memory-boundedness is only defined for a single kernel, as part of the roofline model, but with a bit of squinting it can be generalized to cover the multiple kernels that make up a typical workload.

Contemporary large language model inference workloads are often memory-bound during the decode/output generation stage, when the weights must be loaded once in each forward pass. That happens once per output token, unless multi-token prediction or speculative decoding are used, which makes it easy to calculate the minimum latency between tokens (intertoken latency or time per output token) for memory-bound Transformer large language model inference.

Assume the model has 500B parameters, stored in 16-bit precision, for a total of 1 TB. If we run inference on a single GPU with a memory bandwidth of 10 TB/s, we can load the weights once every 100 ms, and that puts a lower bound on our intertoken latency. By batching multiple inputs together, we can linearly increase the number of floating point operations done per parameter loaded (the arithmetic intensity), in principle up to the point of compute-boundedness, without incurring any additional latency, which implies that the throughput improves linearly in the batch size.

For more on LLM inference, see our LLM Engineer's Almanac.

相关词条