设备硬件 · SM

流式多处理器Streaming Multiprocessor

当我们对 GPU 进行编程时，会生成供其流式多处理器 (Streaming Multiprocessor) 执行的指令序列。

H100 流式多处理器（SM）的内部架构图。图中绿色部分为 GPU 核心，褐红色部分为其他计算单元，橙色部分为调度单元，蓝色部分为内存。修改自 NVIDIA 的 H100 白皮书。

NVIDIA GPU 的流式多处理器（SMs）大致类似于 CPU 的核心。也就是说，SMs 既执行计算，又在寄存器中存储可供计算使用的状态，并配有相关缓存。与 CPU 核心相比，GPU SMs 是简单、功能较弱的处理器。SMs 中指令的执行采用流水线方式（类似于 20 世纪 90 年代以来的几乎所有 CPU），但没有推测执行或指令指针预测（这与所有当代高性能 CPU 不同）。

不过，GPU 的流式多处理器（SMs）能够并行执行更多线程。

作为对比：一颗 AMD EPYC 9965 CPU 的最大功耗为 500 W，拥有 192 个核心，每个核心最多可以同时为两个线程执行指令，总共可并行执行 384 个线程，每个线程的运行功耗约为 1.25 W。

而一颗 H100 SXM GPU 最大功耗为 700 W，配置 132 个 SM，每个 SM 有四个线程束调度器 (Warp Scheduler)，每个调度器每个时钟周期可以向 32 个线程（也称为一个线程束 (Warp)）并行发出指令，总共可并行运行超过 128 × 132 > 16,000 个线程，每个线程的功耗约为 5 cW。请注意，这是真正的并行：16,000 个线程中的每一个线程都可以在每个时钟周期执行操作。

GPU 的流式多处理器（SMs）还支持大量并发线程 —— 即指令交错执行的执行线程。

H100 上的单个 SM 最多可并发执行多达 2048 个并发线程，这些线程分布在 64 个线程组中，每个线程组包含 32 个线程。配置 132 个 SM 后，GPU 总共可支持超过 250,000 个并发线程。

CPU 也能并发运行大量线程。但是线程束 (Warp) 之间的切换速度仅需一个时钟周期（比 CPU 的上下文切换快 1000 倍以上），这一能力由 SM 的线程束调度器 (Warp Scheduler) 支持。丰富的可用线程束 (Warp) 资源与快速的线程束切换 (Warp Switch) 速度有助于掩盖内存读取、线程同步或其他高开销指令带来的延迟 (Latency Hiding)，从而确保由 CUDA 核心 (CUDA Core) 和张量核心 (Tensor Core) 提供的算术带宽 (Arithmetic Bandwidth) 得到充分利用。

这种延迟隐藏 (Latency Hiding) 正是 GPU 优势的核心。 CPU 则通过维护大型硬件管理缓存和复杂的指令预测来向终端用户和程序员隐藏延迟，而这些额外硬件会占用 CPU 的硅片面积、功耗和散热预算，从而限制了可用于计算的资源比例。

与 CPU 相比，GPU 将更多的芯片面积用于计算（绿色部分），而用于控制和缓存（橙色和蓝色部分）的面积则更少。修改自 Fabien Sanglard 博客中的图表，该图表本身可能修改自 CUDA C 编程指南中的图表。

对于神经网络推理或数据库顺序扫描这类程序或函数，程序员可以相对容易地描述缓存的行为（例如，存储每个输入矩阵的一部分并将其在缓存中保留足够长时间以计算相关输出），其结果是显著更高的吞吐量。

英文原文(Modal GPU Glossary)

When we program GPUs, we produce sequences of instructions for its Streaming Multiprocessors to carry out.

.](themed-image://gh100-sm.svg)

Streaming Multiprocessors (SMs) of NVIDIA GPUs are roughly analogous to the cores of CPUs. That is, SMs both execute computations and store state available for computation in registers, with associated caches. Compared to CPU cores, GPU SMs are simple, weak processors. Execution in SMs is pipelined within an instruction (as in almost all CPUs since the 1990s) but there is no speculative execution or instruction pointer prediction (unlike all contemporary high-performance CPUs).

However, GPU SMs can execute more threads in parallel.

For comparison: an AMD EPYC 9965 CPU draws at most 500 W and has 192 cores, each of which can execute instructions for at most two threads at a time, for a total of 384 threads in parallel, running at about 1.25 W per thread.

An H100 SXM GPU draws at most 700 W and has 132 SMs, each of which has four Warp Schedulers that can each issue instructions to 32 threads (aka a warp) in parallel per clock cycle, for a total of 128 × 132 > 16,000 parallel threads running at about 5 cW apiece. Note that this is truly parallel: each of the 16,000 threads can make progress with each clock cycle.

GPU SMs also support a large number of _concurrent_ threads -- threads of execution whose instructions are interleaved.

A single SM on an H100 can concurrently execute up to 2048 threads split across 64 thread groups of 32 threads each. With 132 SMs, that's a total of over 250,000 concurrent threads.

CPUs can also run many threads concurrently. But switches between warps happen at the speed of a single clock cycle (over 1000x faster than context switches on a CPU), again powered by the SM's Warp Schedulers. The volume of available warps and the speed of warp switches help hide latency caused by memory reads, thread synchronization, or other expensive instructions, ensuring that the arithmetic bandwidth provided by the CUDA Cores and Tensor Cores is well utilized.

This latency-hiding is the secret to GPUs' strengths. CPUs seek to hide latency from end-users and programmers by maintaining large, hardware-managed caches and sophisticated instruction prediction. This extra hardware limits the fraction of their silicon area, power, and heat budgets that CPUs can allocate to computation.

, itself likely modified from a diagram in the CUDA C Programming Guide.](themed-image://cpu-vs-gpu.svg)

For programs or functions like neural network inference or sequential database scans for which it is relatively straightforward for programmers to express the behavior of caches — e.g. store a chunk of each input matrix and keep it in cache for long enough to compute the related outputs — the result is much higher throughput.

相关词条