性能

延迟隐藏latency hiding

延迟隐藏是一种通过并发运行多个长延迟操作来掩盖长延迟操作的策略。

高性能 GPU 程序通过交错执行多个线程 (thread) 来隐藏延迟。这使得程序即使面对较长的指令延迟也能保持高吞吐量。当一个线程束 (warp) 因慢速内存操作而停顿 (stall) 时，GPU 会立即切换到执行另一个就绪的线程束 (eligible warp) 的指令。

这种方式能让所有执行单元同时保持忙碌状态。例如，当一个线程束 (warp) 使用张量核心 (Tensor Core) 进行矩阵乘法运算时，另一个线程束可能在 CUDA 核心 (CUDA Core) 上执行算术运算（例如，对矩阵乘数进行量化或反量化），而第三个线程束可能正在通过加载/存储单元 (load/store unit) 获取数据。

具体来说，考虑以下流式汇编器 (Streaming Assembler) 中的简单指令序列。

LDG.E.SYS R1, [R0]        // memory load, 400 cycles
IMUL R2, R1, 0xBEEF       // integer multiply, 6 cycles
IADD R4, R2, 0xAFFE       // integer add, 4 cycles
IMUL R6, R4, 0x1337       // integer multiply, 6 cycles

若按顺序执行，完成该序列需要 416 个周期。我们可以通过并发操作来隐藏此延迟。如果我们假设每个周期可以发射一条指令，那么根据利特尔定律 (Little's Law)，如果我们运行 416 个并发线程 (thread)，平均每个周期仍能完成一次该指令序列，从而对使用 R6 中数据的使用者隐藏了内存延迟。

需要注意的是，指令发射的单位不是线程 (thread)，而是线程束 (warp)。每个线程束 (warp) 包含 32 个线程 (thread)，因此我们的代码片段需要 416 ÷ 32 = 13 个线程束 (warp)。当成功隐藏延迟时，GPU 的调度系统会维持这么多线程束 (warp) 在执行中，在某个线程束停滞时立即切换到其他线程束执行，确保执行单元在等待慢速操作完成期间不会闲置。

要深入了解张量核心 (Tensor Core) 出现前的 GPU 延迟隐藏技术，请参阅 Vasily Volkov 的博士论文。

英文原文(Modal GPU Glossary)

Latency hiding is a strategy to mask long-latency operations by running many of them concurrently.

Performant GPU programs hide latency by interleaving the execution of many threads. This allows programs to maintain high throughput despite long instruction latencies. When one warp stalls on a slow memory operation, the GPU immediately switches to execute instructions from another eligible warp.

This keeps all execution units busy concurrently. While one warp uses Tensor Cores for matrix multiplication, another might execute arithmetic on CUDA Cores (say, quantizing or dequantizing matrix multiplicands), and a third could be fetching data through the load/store units.

Concretely, consider the following simple instruction sequence in Streaming Assembler.

LDG.E.SYS R1, [R0]        // memory load, 400 cycles
IMUL R2, R1, 0xBEEF       // integer multiply, 6 cycles
IADD R4, R2, 0xAFFE       // integer add, 4 cycles
IMUL R6, R4, 0x1337       // integer multiply, 6 cycles

Executed sequentially, this would take 416 cycles to complete. We can hide this latency by operating concurrently. If we assume we can issue one instruction every cycle, then, by Little's Law, if we run 416 concurrent threads, we can still finish the sequence once per cycle (on average), hiding the latency of memory from consumers of the data in R6.

Note that threads are not the unit of instruction issuance, warps are. Each warp contains 32 threads, and so our fragment requires 416 ÷ 32 = 13 warps. When successfully hiding latency, the GPU's scheduling system maintains this many warps in flight, switching between them whenever one stalls, ensuring the execution units never idle while waiting for slow operations to complete.

For a deep dive into latency hiding on pre-Tensor Core GPUs, see Vasily Volkov's PhD thesis.

相关词条