主机软件

cuBLAScuBLAS

cuBLAS (CUDA Basic Linear Algebra Subroutines，CUDA 基础线性代数子程序) 是 NVIDIA 对基础线性代数子程序 (BLAS) 标准的高性能实现。它是一个专有软件库，提供了针对常见线性代数运算高度优化的内核 (kernel)。

开发者无需从头编写和优化像矩阵乘法这样的常见运算，而是可以直接从他们的主机代码中调用 cuBLAS 函数。该库包含大量内核，每个内核都针对特定的数据类型（例如 FP32、FP16）、矩阵大小和流式多处理器 (Streaming Multiprocessor, SM) 架构进行了精细调优。在运行时，cuBLAS 会使用内部启发式算法（具体细节未公开）选择性能最佳的内核及其最优启动参数。因此，cuBLAS 成为了在 NVIDIA GPU 上进行大多数高性能 (high-performance) 数值计算的基础，并被 PyTorch 等深度学习框架广泛使用，用于加速其核心运算，同时使用的还有更专门的内核 (kernel) 库，例如 cuDNN。

使用 cuBLAS 时最常见的一个错误来源是矩阵数据布局。由于历史原因，并且为了保持与原始 BLAS 标准（用 Fortran 编写）的兼容性，cuBLAS 期望矩阵采用列优先 (column-major order)。这与 C、C++ 和 Python 中常用的行优先相反。此外，BLAS 函数不仅需要知道运算规模（例如 M, N, K），还需要知道如何在内存中定位每一列的起始位置——这由前导维度（leading dimension，如lda）指定。前导维度是连续列之间的步长：当处理整个已分配矩阵时，前导维度等于行数；而处理子矩阵时，前导维度则为从中提取子矩阵的更大父矩阵的行数。

幸运的是，对于GEMM（通用矩阵乘法）等计算密集型内核，无需将矩阵从行优先重新排序为列优先。相反，我们可以利用数学上的恒等式：如果 C = A @ B，那么 C^T = B^T @ A^T。核心思路是：以行优先存储的矩阵，其内存布局与按列优先存储的转置矩阵完全相同。因此，如果我们向 cuBLAS 提供我们的行优先矩阵 A 和 B，但交换它们在函数调用中的顺序（并调整维度参数），cuBLAS 将计算 C^T 并以列优先输出它。当这段内存按行优先解读时，恰好就是我们所需的矩阵 C。这种技巧可通过以下函数示例说明：

#include <cublas_v2.h>

// 在行主序矩阵上使用 cublasSgemm 执行单精度运算 C = alpha * A @ B + beta * C
void sgemm_row_major(cublasHandle_t handle, int M, int N, int K,
                     const float *alpha,
                     const float *A, const float *B,
                     const float *beta,
                     float *C) {

  // A 是 M x K (行主序), cuBLAS 将其视为 A^T (K x M, 列主序),
  //   A^T 的主维度是 K
  // B 是 K x N (行主序), cuBLAS 将其视为 B^T (N x K, 列主序),
  //   B^T 的主维度是 N
  // C 是 M x N (行主序), cuBLAS 将其视为 C^T (N x M, 列主序),
  //   C^T 的主维度是 N

  // 注意交换了 A 和 B 的位置，以及交换了 M 和 N 的位置
  cublasSgemm(handle, CUBLAS_OP_N, CUBLAS_OP_N,
              N, M, K,
              alpha,
              B, N,  // B^T 的主维度
              A, K,  // A^T 的主维度
              beta,
              C, N); // C^T 的主维度
}

此示例的一个完整可运行版本可在 Godbolt 上找到。

CUBLAS_OP_N 标志用于指示内核按矩阵的原始形式使用（从其视角来看，无需额外的转置操作）。

要使用 cuBLAS 库，必须在编译时进行链接（例如，在使用 nvcc 编译时使用 -lcublas 标志）。其函数通过 cublas_v2.h 头文件暴露。

有关 cuBLAS 的更多信息，请参阅官方 cuBLAS 文档。

英文原文(Modal GPU Glossary)

cuBLAS (CUDA Basic Linear Algebra Subroutines) is NVIDIA's high-performance implementation of the Basic Linear Algebra Subprograms (BLAS) standard. It is a proprietary software library that provides highly optimized kernels for common linear algebra operations.

Instead of writing and optimizing common operations like matrix multiplication from scratch, developers can call cuBLAS functions from their host code. The library contains a wide array of kernels, each fine-tuned for specific data types (e.g. FP32, FP16), matrix sizes, and Streaming Multiprocessor (SM) architectures. At runtime, cuBLAS uses (unknown) internal heuristics to select the most performant kernel and its optimal launch parameters. As a result, cuBLAS is the foundation for most high-performance numerical computing on NVIDIA GPUs and is used extensively by deep learning frameworks like PyTorch to accelerate their core operations, along with more specialized kernel libraries like cuDNN.

The single most common source of error when using cuBLAS is the matrix data layout. For historical reasons, and to maintain compatibility with the original BLAS standard (which was written in Fortran), cuBLAS expects matrices to be in column-major order. This is the opposite of the commonly used row-major order in C, C++, and Python. Furthermore, a BLAS function needs to know not just the size of the operation (e.g., M, N, K), but also how to find the start of each column in memory. This is specified by the leading dimension (e.g. lda). The leading dimension is the stride between consecutive columns. When working with an entire allocated matrix, the leading dimension is just the number of rows. However, if working with a submatrix, the leading dimension would be the number of rows in the larger, parent matrix from which the submatrix is taken.

Fortunately, for computationally intensive kernels like GEMM, it is not necessary to reorder matrices from row-major to column-major. Instead, we can use the mathematical identity that if C = A @ B, then C^T = B^T @ A^T. The key insight is that a matrix stored in row-major order has the exact same memory layout as its transpose stored in column-major order. Therefore, if we provide our row-major matrices A and B to cuBLAS but swap their order in the function call (along with their dimensions), cuBLAS will compute C^T and output it in column-major order. This resulting block of memory, when interpreted in row-major, is exactly the matrix C that we want. This technique is demonstrated in the following function:

#include <cublas_v2.h>

// performs single-precision C = alpha * A @ B + beta * C
// on row-major matrices using cublasSgemm
void sgemm_row_major(cublasHandle_t handle, int M, int N, int K,
                     const float *alpha,
                     const float *A, const float *B,
                     const float *beta,
                     float *C) {

  // A is M x K (row-major), cuBLAS sees it as A^T (K x M, column-major),
  //   the leading dimension of A^T is K
  // B is K x N (row-major), cuBLAS sees it as B^T (N x K, column-major),
  //   the leading dimension of B^T is N
  // C is M x N (row-major), cuBLAS sees it as C^T (N x M, column-major),
  //   the leading dimension of C^T is N

  // note the swapped A and B, and the swapped M and N
  cublasSgemm(handle, CUBLAS_OP_N, CUBLAS_OP_N,
              N, M, K,
              alpha,
              B, N,  // leading dimension of B^T
              A, K,  // leading dimension of A^T
              beta,
              C, N); // leading dimension of C^T
}

A complete, runnable version of this example is available on Godbolt.

The CUBLAS_OP_N flag instructs the kernel to use the matrices as provided (without an additional transpose operation from its perspective).

To use the cuBLAS library, it must be linked (e.g. using the flag -lcublas when compiling with nvcc). Its functions are exposed via the cublas_v2.h header.

For more information on cuBLAS, see the official cuBLAS documentation.

相关词条