Alexander Supalov - Inside the Message Passing Interface: Creating Fast Communication Libraries

Alexander Supalov

Inside the Message Passing Interface

ISBN 978-1-5015-1554-5

e-ISBN (PDF) 978-1-5015-0687-1

e-ISBN (EPUB) 978-1-5015-0678-9

Library of Congress Control Number: 2018952175

Bibliographic information published by the Deutsche Nationalbibliothek

The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografie;

detailed bibliographic data are available on the Internet at http://dnb.dnb.de.

2018 Alexander Supalov

Published by Walter de Gruyter Inc., Boston/Berlin

www.degruyter.com

To my beloved wife and wonderful children

De|G PRESS, the startup born out of one of the worlds most venerable publishers, De Gruyter, promises to bring you an unbiased, valuable, and meticulously edited work on important topics in the fields of business, information technology, computing, engineering, and mathematics. By selecting the finest authors to present, without bias, information necessary for their chosen topic for professionals , in the depth you would hope for, we wish to satisfy your needs and earn our five-star ranking.

In keeping with these principles, the books you read from De|G PRESS will be practical, efficient and, if we have done our job right, yield many returns on their price.

We invite businesses to order our books in bulk in print or electronic form as a best solution to meeting the learning needs of your organization, or parts of your organization, in a most cost-effective manner.

There is no better way to learn about a subject in depth than from a book that is efficient, clear, well organized, and information rich. A great book can provide life-changing knowledge. We hope that with De|G PRESS books you will find that to be the case.

I would like to thank Stuart Douglas and Jeffrey Pepper of De Gruyter for approaching me with an offer that prompted the writing of this book. My thanks also go to Jeffrey Pepper and Jaya Dalal for their efforts on the manuscript, to Mary Sudul for diligent copyediting, and to MacPS for their production work.

Jeff Squyres and Kenneth Raffenetti helped me with certain aspects of Open MPI and MPICH, respectively, saving quite a bit of time that would have otherwise gone into issue investigation.

Many anonymous reviewers provided valuable feedback on my conference submissions, helping me fine tune the text of the book. I extend my gratitude to them, too.

Implementation of the Message Passing Interface (MPI) is deemed one of the more exalted of systems programmers occupations: somewhat below the presumed complexity of an operating system (OS) with all its drivers, akin to a compiler and its runtime libraries, and more demanding than application programming. This is partly explained by the nature of the MPI itself: as just another software library in form, it sits between the MPI application (and thus the application level) and the hardware (and thus at and below the operating system level), striving to achieve the best possible performance. Of course, the important services of the operating system are called upon when necessary, however, now and then an MPI implementor must go down to the wire, bypassing the operating system either directly or with the help of other specific systems libraries that in turn talk directly to the respective hardware components.

This book seeks to demystify this endeavor, often perceived as open to just a few exceptional daredevils who dwell in ivory towers, spending uncounted taxpayer money running expensive custom built parallel instal-lations for purposes far too remote for anyone in the street. Well, going backward through this statement, this is all wrong. First of all, High Performance Computing (HPC), of which the MPI is an integral part, has come to affect everyone, anytime, and everywhere. Looking up tomorrows weather forecast, driving a car, boarding a plane, searching the Web, or buying goods onlineyou rely on HPC and likely the MPI more often than you may think. Second, parallelism has reached the computer you use for your everyday business and leisure activi-ties. The laptop of today surpasses supercomputers of the 1970s, and kids toys carry around more computing power nowadays than the whole Apollo 11 mission that landed men on the Moon back in the 1960s! And third, any skill, however exalted, can be learned given the will and capacity to learn.

Communication permeates the modern world. This is why learning how to create fast, efficient, compact, and scalable communication libraries is important now and is going to be even more important in the future. Even though MPI per se may be unique in its laser focus on cutting edge technical HPC, the methods and ap-proaches used during MPI design, implementation, and testing are going to be pertinent in all other areas of communication software. This book will provide you with a better understanding of the MPI internals and their ifs and whys. It will also let you extract more performance out of your HPC applications. Beyond that, it will show you how to approach comparable tasks in any programming language on any device, whether that is a supercomputer running Linux, a laptop running Windows, or your smartphone running Android or iOS.

This is a hands-on guide to writing a Message Passing Interface (Message Passing Interface Forum, 2015) implementation from scratch. This book also discusses the major MPI implementations, best optimization techniques, application relevant usage hints, and a historical retrospective of the MPI worldall based on the experience of a quarter century spent inside the MPI. It offers:

Intimate knowledge of the MPI internals presented in a hands-on fashion

Optimizations to achieve the maximum performance possible in the hardware

Deep understanding of the MPI advantages and drawbacks

This book is intended for MPI implementors (existing and hopeful), creators of alternative communication interfaces looking beyond whats on their plate, advanced MPI users seeking deeper understanding of the MPI internals, and programmers willing to learn more about software design, development, and optimization.

This book is neither an introduction to the HPC in general or the MPI in particular, nor a description of a particular MPI implementation, nor a vendor-specific MPI usage manual. There are a number of good information sources covering the aforementioned areas that you can read in order to get up to speed if necessary.

In addition to this, you should have a rather good grasp of systems programming, communication and networking technologies, and the MPI standard itself. Knowledge of the C programming language as well as of the Linux operating system is essential. Knowledge of the Bourne shell and Fortran programming languages may occasionally be helpful, too, as well as practical experience of creating sizeable software products and using the associated code management techniques and the program building tool chain.

I tried to keep this book as informal as possible. All you need to know is that source code will be highlighted using