This appendix lists the most important of the many CUDA support documents and also the libraries available from the NVIDIA website and elsewhere.

The solution of partial differential equations in two and three-dimensions using stencil iteration (Jacobi’s method) is discussed and illustrated for Laplace’s equation. A very simple kernel gives about a factor of 100 speed-up compared to the host CPU.The very slow convergence of the Jacobi method can be addressed by using solutions on lower resolution grids to initialise higher resolution grids. A convergence check using the maximum change per iteration is also illustrated. Digital image processing is another example of stencil use and a number of digital image filters are shown including the Sobel filter for edge finding and the median filter for noise reduction. The fast GPU-based median filter uses one thread per image pixel and is implemented using an optimal Batcher network.

Chapter 3 contains a comprehensive description of CUDA cooperative groups including the powerful features for explicit warp-level programming. Warp-level programming is becoming more prominent with recent CUDA developments such as the warp matrix functions introduced to support tensor core hardware. The various types of thread groupings are discussed and illustrated in examples.We show a revised reduce kernel which uses warp-level intrinsic functions instead of shared memory.This kernel is further improved by using 128-bit vector loading of data from global GPU memory to kernel local register-based variables. A variation of this example using coalesced thread groups is also shown.The conditions for avoiding deadlock when working with warp-level thread divergence are explored with differences between the older and newer generations of GPU explained. An example using the new the cg::reduce function is shown; this function has hardware support for CC ≥ 8 devices.

Appendix H briefly lists and discusses the available NVIDIA and third-party tools to support AI and Python.

Chapter 3 contains a comprehensive description of CUDA cooperative groups including the powerful features for explicit warp-level programming. Warp-level programming is becoming more prominent with recent CUDA developments such as the warp matrix functions introduced to support tensor core hardware. The various types of thread groupings are discussed and illustrated in examples.We show a revised reduce kernel which uses warp-level intrinsic functions instead of shared memory.This kernel is further improved by using 128-bit vector loading of data from global GPU memory to kernel local register-based variables. A variation of this example using coalesced thread groups is also shown.The conditions for avoiding deadlock when working with warp-level thread divergence are explored with differences between the older and newer generations of GPU explained. An example using the new the cg::reduce function is shown; this function has hardware support for CC ≥ 8 devices.

This appendix gives a careful description of integer and IEEE floating point number formats and the ranges of valid values.

Chapter 5 continues the theme of digital image manipulation and considers transformations such as rotation or scaling with required pixel interpolation to create the most accurate final result. The GPU hardware texture units are used for this and their features are discussed. The cx utilities provided with our code include wrappers that significantly simplify the creation of CUDA textures. Curiously, these hardware texture units are rarely discussed in other CUDA tutorial material for scientific applications but we find they can give a 5-fold performance boost. We show how OpenCV can be used to provide a simple GUI interface for viewing the transformed images with very little coding effort. We end the chapter with a fully working 3D image registration program using affine transformations applied to volumetric MRI data sets. The 3D affine transformations are about 1500 times faster on the GPU than on the host CPU and a full registration between two MRI images of size 256 × 256 × 256 takes about one second.

Appendix I discusses a number of keyC++ concepts used throughout the book

Chapter 10 describes the various tools available for both profiling kernel performance and debugging code.For profiling we discuss both the older CUDA nvprof command line profiler and associated NVVP GUI and the newer Nsight Systems and Nsight Compute profilers which have many options. In particular, Nsight Compute can give detail of the performance within an individual kernel which was not possible before. Our discussion of debugging is based on the tools in Microsoft Visual Studio both for conventional C++ debugging and to enhance CUDA plugins for kernel debugging. The CUDA (Next-Gen) toolset allows line by line monitoring of individual threads during kernel execution.

This chapter discusses the tensor core hardware available on newer GPUs. This hardware is designed to perform fast mixed precision matrix multiplications and is intended for applications in AI.However, CUDA exposes their use to programmers with the warp matrix function library. These functions support tiled matrix multiplication using 16 × 16 tiles.We provide examples of their use to improve on the early matrix multiplication example in Chapter 2.We also show how reduction operations can be performed using tensor codes as a potential non-AI application.

Appendix G lists and discusses the contents of our cx support header files. The cx.h header includes basic definitions for type and thrust support and is likely to be used by most programs. The other header files provide more specialised support for timers, binary IO and CUDA textures. This appendix is where detailed information on the use of CUDA textures can be found. It is thought that some version of Morton indexing is used for 2D texture layout and that described.