Singh - Parallel And Distributed Computing

Parallel and Distributed Computing
Copyrighted Material
Copyright 2021-22 by Ajit Singh. All Rights Reserved.

No part of this publication may be reproduced, stored in a retrieval system or transmitted, in any form or by any means electronic, mechanical, photocopying, recording or otherwise without prior written permission from the author, except for the inclusion of brief quotations in a review.

For information about this title or to order other books and/or electronic media, contact the publisher.

Ajit Singh
ajit_singh24@yahoo.com
http://www.ajitvoice.wordpress.in

This piece of study of Parallel and Distributed Computing is an outcome of the encouragement, guidance, help and assistance provided to us by our colleagues, Sr. faculties, Tech-friends and our family members.

As an aknowledgement, I would like to take the opportunity to express my deep sense of gratitude to all those who played a crucial role in the successful completion of this book, especially to my sr. students; this book certainly has been benefited from discussions held with many IT professionals over the years it took us to write it.

My primary goal here is to provide a sufficient introduction and details of the Parallel and Distributed Computing so that the students can have an efficient knowledge about Parallel and Distributed Computing. Moreover, it presupposes knowledge of the principles and concepts of the C and FORTAN languages are required. On the same note, any errors and inaccuracies are my responsibility and any suggestions in this regard are warmly welcomed!

Finally, I would like to thank the Kindle Direct Publishing team and Amazon team for its enthusiastic online support and guidance in bringing out this book.
I hope that the reader will like this book and find it useful in learning the concepts of Parallel and Distributed Computing with practical implementation of MPI.
Thank You !! Ajit Singh Sharetheknowledge,
Strenghtenthesurrounding..... !!

The study/learning of Parallel & Distributed Computing is an essential part of any computer science education and of course for the B.Tech / MCA / M.Tech courses of several Universities across the world. This textbook is intended as a guide for an explanatory course of Parallel & Distributed Computing for the Graduate and Post Graduate Students of several universities across the world.

This book is an introduction to the complex and emerging world of the Parallel and Distributed Computing. It helps you understand the principles and acquire the practical skills of MPI programming using the C/FORTAN programming language. My aim is for you to gain sufficient knowledge and experience to perform simple useful programming tasks using the best up-to-date techniques and so I hope for it to be the easiest book from which you can learn the basics of MPI programming. It helps you understand the principles, algorithm & implementation of Parallel and Distributed Computing.

I have chosen the topics for this book to cover what is needed to get started with Parallel and Distributed Computing, not just what is easy to teach and learn. On the other hand, I wont waste your time with material of marginal practical importance. If an idea is explained here, its because youll almost certainly need it.

This book is emphatically focused on the concept. Understanding the fundamental ideas, principles, and techniques is the essence of a good programmer. Only well-designed code has a chance of becoming part of a correct, reliable, and maintainable parallel system. Through this book, I hope that you will see the absolute necessity of understanding Parallel and Distributed Computing.
I have taken a top-down approach, addressing the issues to be resolved in the design of parallel and distributed systems and describing successful approaches in the form of abstract models, algorithms and detailed case studies of widely used systems.

The book aims to provide an understanding of the principles on which the distributed computing are based; their architecture, algorithms and design; and how it meets the demands of contemporary distributed applications. I began with a set of chapters that together cover the building blocks for a study of parallel and distributed systems. The first few chapters provide a conceptual overview of the subject, outlining the characteristics of parallel and distributed systems and the challenges that must be addressed in their design: scalability, heterogeneity, security and failure handling being the most significant. These chapters also develop abstract models for understanding process interaction, failure and security.

"I hope you enjoy reading this book as much as I have enjoyed writing it.... Sl No Topic(s) Pg No
1 INTRODUCTION TO PARALLEL COMPUTING 9

Introduction
Background
Problem Solving in Parallel
Concept of Temporal Parallelism
Concept of Data Parallelism
Functional Parallelism
Classes of parallel computers
Performance Evaluation
The Concept of Concurrent and Parallel Execution

Granularity
Potential of Parallelism
Data Clustering
Minsky Conjecture
The Need of Parallel Computation
Levels of Parallel Processing
Bit Level
Instruction Level
Loop Level
Procedure Level
Program Level
Applications of Parallel Processing
Scientific Applications/Image Processing
Engineering Applications
Database Query/Answering Systems
AI Applications
Mathematical Simulation and Modeling Applications Supercomputers and grand challenge problems
Modem Parallel Computers
Future of parallel computing

2 CLASSIFICATION OF PARALLEL COMPUTERS 25

Introduction
Types of Classification
Flynns Classification

Instruction Cycle
Instruction Stream and Data Stream
Flynns Classification

Handlers Classification
Structural Classification
Shared Memory System/Tightly Coupled System

Uniform Memory Access Model
Non-Uniform Memory Access Model
Cache-only Memory Architecture Model
Loosely Coupled Systems

Classification Based on Grain Size Parallelism Conditions
Bernstein Conditions for Detection of Parallelism Parallelism Based on Grain Size

3 INTERCONNECTION NETWORK 40 Introduction
Network Properties
Design issues of Interconnection Network
Various Interconnection Networks
Concept of Permutation Network
Performance Metrics

4 PARALLEL COMPUTER ARCHITECTURE 55 Introduction
Pipeline Processing

Classification of Pipeline Processors
Instruction Pipelines
Arithmetic Pipelines

Performance and Issues in Pipelining
Vector Processing
Array Processing

Associative Array Processing
Multi-threaded Processors

5 PERFORMANCE ANALYSIS OF PARALLEL COMPUTING 70 Introduction
Definitions

Performance analysis
Performance analysis techniques
Performance analysis metrics

Efficiency
Speedup
Amdahl's Law
Gustafson Law
Gustafson-Barsis's Law

Superlinear Speedup and Efficiency
The Karp-Flatt Metric
The Isoefficiency Metric
Isoefficiency Relation
Cost and Scalability

6 PARALLEL COMPUTATIONAL MODEL 90 PRAM
CRCW
CREW
EREW,
Simulating CRCW on CREW & SREW
PRAM algorithms
P-Complete problems.

7 INTRODUCTION TO PARALLEL ALGORITHMS 98 PVM
MPI Paradigms
Simple parallel programs in MPI/PVM environments
Parallel algorithms on network
Addition of Matrices
Multiplication of Matrices
Parallel Programming Issues
Systolic Array

Next page