性能

计算受限的含义是什么What does it mean to be compute-bound

计算受限的内核 (Kernel) 受限于 CUDA 核心 (CUDA Core) 或张量核心 (Tensor Core) 的算术带宽 (arithmetic bandwidth)。

在上述的屋顶线图 (roofline diagram) 中，位于蓝线以下的内核 (kernel) 属于计算受限。图表改编自 Williams, Waterman, and Patterson (2008)。

计算受限内核的特征是具有高算术强度 (arithmetic intensity)（每加载或存储一字节内存需要执行大量算术运算）。其性能瓶颈在算术流水线利用率 (Utilization of arithmetic pipes)。

从技术角度而言，计算受限性仅针对单个内核 (kernel) 定义，作为屋顶线模型 (roofline model) 的一部分，但稍作引申后，可以将其推广到构成典型工作负载的多个内核 (kernel)。

大型扩散模型的推理工作负载通常是计算受限的。当代大型语言模型的推理工作负载在批量预填充/提示处理阶段通常是计算受限的。此时每个权重可以加载到共享内存 (shared memory) 中一次，然后多个令牌重复使用。

让我们基于 kipperrii 的 Transformer 推理算术框架做一个简单估算，对计算受限的 Transformer 语言模型推理的最小令牌间延迟（inter-token latency，即每个输出令牌的生成时间）进行简单估算。假设某个模型有 5000 亿参数，以 16 位精度存储，总计 1 TB。每个批处理元素需执行约1万亿次浮点运算（每个参数一次乘法和一次累加）。在具有 16 位矩阵运算的 1 petaFLOP/s 算术带宽 (arithmetic bandwidth) 的 GPU 上运行，在计算受限的假设下，每个批处理元素的最小令牌间延迟为 1 毫秒。

需要注意的是，要使该 GPU 在批次大小为 1 时达到计算受限，需要具备 1 PB/s 的内存带宽 (memory bandwidth)（以便在 1 毫秒内加载全部 1 TB 权重）。当代内存带宽 (memory bandwidth) 在 TB/s 量级，因此需要数百个输入组成的批处理才能提供足够的算术强度 (arithmetic intensity) 使执行过程进入计算受限状态。。

有关 LLM 推理的更多信息，请参阅我们的 LLM 工程师指南。

英文原文(Modal GPU Glossary)

Kernels that are compute-bound are limited by the arithmetic bandwidth of the CUDA Cores or Tensor Cores.

above, kernels underneath the blue line are compute-bound. Diagram adapted from Williams, Waterman, and Patterson (2008).](themed-image://roofline-model.svg)

Compute-bound kernels are characterized by high arithmetic intensity (many arithmetic operations per byte of memory loaded or stored). Utilization of arithmetic pipes is the limiting factor for a compute-bound kernel.

Technically, compute-boundedness is only defined for a single kernel, as part of the roofline model, but with a bit of squinting it can be generalized to cover the multiple kernels that make up a typical workload.

Large diffusion model inference workloads are generally compute-bound. Contemporary large language model inference workloads are often compute-bound during batch prefill/prompt processing, when each weight can be loaded into shared memory once and then used across many tokens.

Let's do a simple estimation, inspired by kipperrii's Transformer inference arithmetic framework, of the minimum latency between tokens (inter-token latency or time per output token) for compute-bound Transformer language model inference. Assume the model has 500B parameters, stored in 16-bit precision, for a total of 1 TB. This model will perform roughly one trillion floating point operations (one multiply and one accumulate per parameter) per batch element. Run on a GPU with one petaFLOP/s of arithmetic bandwidth for 16-bit matrix math, the minimum latency between tokens, assuming compute-boundedness, is one millisecond per batch element.

Note that for this GPU to be compute-bound at batch size one, it would need a memory bandwidth of 1 PB/s (so that it can load all 1 TB of weights in one ms). Contemporary memory bandwidths are in the TB/s range, and so batches of hundreds of inputs are required to provide sufficient arithmetic intensity for execution to be compute-bound.

For more on LLM inference, see our LLM Engineer's Almanac.

相关词条