Nicholas Wilt - The CUDA Handbook: A Comprehensive Guide to GPU Programming

A Comprehensive Guide to GPU Programming

Nicholas Wilt

Upper Saddle River, NJ Boston Indianapolis San Francisco
New York Toronto Montreal London Munich Paris Madrid
Capetown Sydney Tokyo Singapore Mexico City

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and the publisher was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals.

The author and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein.

The publisher offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales, which may include electronic versions and/or custom covers and content particular to your business, training goals, marketing focus, and branding interests. For more information, please contact:

U.S. Corporate and Government Sales
(800) 382-3419

For sales outside the United States, please contact:

International Sales

Visit us on the Web: informit.com/aw

Cataloging in Publication Data is on file with the Library of Congress.

Copyright 2013 Pearson Education, Inc.

All rights reserved. Printed in the United States of America. This publication is protected by copyright, and permission must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. To obtain permission to use material from this work, please submit a written request to Pearson Education, Inc., Permissions Department, One Lake Street, Upper Saddle River, New Jersey 07458, or you may fax your request to (201) 236-3290.

ISBN-13: 978-0-321-80946-9
ISBN-10: 0-321-80946-7
Text printed in the United States on recycled paper at RR Donelley in Crawfordsville, Indiana.
First printing, June 2013.

For Robin

If you are reading this book, I probably dont have to sell you on CUDA. Readers of this book should already be familiar with CUDA from using NVIDIAs SDK materials and documentation, taking a course on parallel programming, or reading the excellent introductory book CUDA by Example (Addison-Wesley, 2011) by Jason Sanders and Edward Kandrot.

Reviewing CUDA by Example, I am still struck by how much ground the book covers. Assuming no special knowledge from the audience, the authors manage to describe everything from memory types and their applications to graphics interoperability and even atomic operations. It is an excellent introduction to CUDA, but it is just that: an introduction. When it came to giving more detailed descriptions of the workings of the platform, the GPU hardware, the compiler driver nvcc, and important building block parallel algorithms like parallel prefix sum (scan), Jason and Edward rightly left those tasks to others.

This book is intended to help novice to intermediate CUDA programmers continue to elevate their game, building on the foundation laid by earlier work. In addition, while introductory texts are best read from beginning to end, The CUDA Handbook can be sampled. If youre preparing to build or program a new CUDA-capable platform, a review of ). Other chapters give detailed descriptions of the software architecture, GPU subsystems such as texturing and the streaming multiprocessors, and applications chosen according to their data access pattern and their relative importance in the universe of parallel algorithms. The chapters are relatively self-contained, though they do reference one another when appropriate.

The latest innovations, up to and including CUDA 5.0, also are covered here. In the last few years, CUDA and its target platforms have significantly evolved. When CUDA by Example was published, the GeForce GTX 280 (GT200) was new, but since then, two generations of CUDA-capable hardware have become available. So besides more detailed discussions of existing features such as mapped pinned memory, this book also covers new instructions like Fermis ballot and Keplers shuffle and features such as 64-bit and unified virtual addressing and dynamic parallelism. We also discuss recent platform innovations, such as the integration of the PCI Express bus controller into Intels Sandy Bridge CPUs.

However you choose to read the bookwhether you read it straight through or keep it by your keyboard and consult it periodicallyits my sincerest hope that you will enjoy reading it as much as I enjoyed writing it.

I would like to take this opportunity to thank the folks at NVIDIA who have been patient enough to answer my questions, review my work, and give constructive feedback. Mark Harris, Norbert Juffa, and Lars Nyland deserve special thanks.

My reviewers generously took the time to examine the work before submission, and their comments were invaluable in improving the quality, clarity, and correctness of this work. I am especially indebted to Andre Brodtkorb, Scott Le Grand, Allan MacKinnon, Romelia Salomon-Ferrer, and Patrik Tennberg for their feedback.

My editor, the inimitable Peter Gordon, has been extraordinarily patient and supportive during the course of this surprisingly difficult endeavor. Peters assistant, Kim Boedigheimer, set the standard for timeliness and professionalism in helping to complete the project. Her efforts at soliciting and coordinating review feedback and facilitating uploads to the Safari Web site are especially appreciated.

My wife Robin and my sons Benjamin, Samuel, and Gregory have been patient and supportive while I brought this project across the finish line.

Nicholas Wilt has been programming computers professionally for more than twenty-five years in a variety of areas, including industrial machine vision, graphics, and low-level multimedia software. While at Microsoft, he served as the development lead for Direct3D 5.0 and 6.0, built the prototype for the Windows Desktop Manager, and did early GPU computing work. At NVIDIA, he worked on CUDA from the beginning, designing and often implementing most of CUDAs low-level abstractions. Now at Amazon, Mr. Wilt is working in cloud computing technologies relating to GPUs.

Much ink has been spilled describing the GPU revolution in computing. I have read about it with interest because I got involved very early. I was at Microsoft in the mid-1990s as development lead for Direct3D when Intel and AMD were introducing the first multimedia instruction sets to accelerate floating point computation. Intel had already tried (unsuccessfully) to forestall the migration of clock cycles for 3D rasterization from their CPUs by working with Microsoft to ship rasterizers that used their MMX instruction set. I knew that effort was doomed when we found that the MMX rasterizer, running on a yet-to-be-released Pentium 2 processor, was half as fast as a humble S3 Virge GX rasterizer that was available for sale.

For Direct3D 6.0, we worked with CPU vendors to integrate their code into our geometry pipeline so developers could transparently benefit from vendor-optimized code paths that used new instruction sets from Intel and AMD. Game developers embraced the new geometry pipeline, but it did not forestall the continued migration of clock cycles from the CPU to the GPU, as the new instruction sets were used to generate vertex data for consumption by GPUs hardware geometry pipelines.