Aytas Yusuf - How to Use, Deploy and Maintain Big Data Systems

1. Chapter 4
2. Chapter 10

1. Chapter 2
2. Chapter 4
3. Chapter 5
4. Chapter 6
5. Chapter 7
6. Chapter 8
7. Chapter 9
8. Chapter 10
9. Chapter 11
10. Chapter 12
11. Appendix A
12. Appendix B

Yusuf Aytas

Dublin, Ireland

This edition first published 2021

2021 John Wiley and Sons, Inc.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by law. Advice on how to obtain permission to reuse material from this title is available at http://www.wiley.com/go/permissions.

The right of Yusuf Aytas to be identified as the author of this work has been asserted in accordance with law.

Registered Office

John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USA

Editorial Office

111 River Street, Hoboken, NJ 07030, USA

For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com.

Wiley also publishes its books in a variety of electronic formats and by printondemand. Some content that appears in standard print versions of this book may not be available in other formats.

Limit of Liability/Disclaimer of Warranty

The contents of this work are intended to further general scientific research, understanding, and discussion only and are not intended and should not be relied upon as recommending or promoting scientific method, diagnosis, or treatment by physicians for any particular patient. In view of ongoing research, equipment modifications, changes in governmental regulations, and the constant flow of information relating to the use of medicines, equipment, and devices, the reader is urged to review and evaluate the information provided in the package insert or instructions for each medicine, equipment, or device for, among other things, any changes in the instructions or indication of usage and for added warnings and precautions. While the publisher and authors have used their best efforts in preparing this work, they make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives, written sales materials or promotional statements for this work. The fact that an organization, website, or product is referred to in this work as a citation and/or potential source of further information does not mean that the publisher and authors endorse the information or services the organization, website, or product may provide or recommendations it may make. This work is sold with the understanding that the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for your situation. You should consult with a specialist where appropriate. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

Library of Congress CataloginginPublication Data Applied for:

ISBN: 9781119690924

Cover design by Wiley

Cover image: monsitj/iStock/Getty Images

• mer E. Avaroullar
• San Francisco, CA
• United States

• Louis Calisi
• Acton, MA
• United States

• Ender Demirkaya
• Seattle, WA
• United States

• Alperen Eraslan
• Ankara
• Turkey

• Ang Gao
• Dublin
• Ireland

• Zehra Kavasolu
• London
• United Kingdom

• David Kjerrumgaard
• Henderson, Nevada
• United States

• Ari Miller
• Los Angeles, CA
• United States

• Alper Mermer
• Manchester
• United Kingdom

• Will Schneider
• Waltham, MA
• United States

Big data has been one of the buzzwords of the last couple of decades. Although there is a certain hype around big data, it is now seen as increasingly essential for organizations. There are so many new systems popping up now and then. This book does not particularly focus on any of the new systems or projects, but rather focuses on how these systems are built and used. The book gives an overview of the systems and lays out the architecture. The book mentions and sometimes goes into detail about both new and old big data systems, such as Apache Hadoop and Apache Spark; it attempts to give an overall view over designing big data platforms regardless of the technology.

The book offers some general knowledge about designing big data platforms. If you are a frontend engineer, backend engineer, data engineer, data analyst, or data scientist, you will see different aspects of designing big data platforms. The book at times goes into technical detail on some subjects through code or design but this doesn't prevent the nonspecialist obtaining an understanding of the underlying concepts. If you are an expert on big data platforms, perhaps, the book can revisit things of which you might already be aware.

The book gives a general overview of big data technologies to design big data platforms. The book covers many interesting technologies, but it is not a reference book for any of the technologies mentioned. It dives deep into certain technologies but overall tries to establish a framework rather than focusing on certain tools.

At the beginning, the book tries to go over big data, big data platforms, and a simple data processing system. Later, it starts to go into discussing various aspects of big data such as storage, processing, discovery, security, and so forth. At the end of the book, it summarizes systems that it talked about and discusses some of the useful patterns for designing big data platforms. In Appendix A, the book discusses some of the other technologies and patterns that don't really fit the flow of the book. In Appendix B, the book discusses recipes where it presents a solution for a particular big data problem.

5 February 2021

Yusuf Aytas

Dublin

This book is an overview of many systems, patterns, and ideas, in an attempt to combine experiences from academia and industry with reference to many great articles, posts, and books in order to provide a consolidated view on designing big data platforms. While writing this book, I have gained enormous knowledge from many exceptional authors, engineers, analysts, scientists, and editors. I'm very grateful for all the information that people made available.