Sreeram Nudurupati - Essential PySpark for Scalable Data Analytics: A beginners guide to harnessing the power and ease of PySpark 3

A beginner's guide to harnessing the power and ease of PySpark 3

Sreeram Nudurupati

BIRMINGHAMMUMBAI

Copyright 2021 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Publishing Product Manager: Aditi Gour

Senior Editor: Mohammed Yusuf Imaratwale

Content Development Editor: Sean Lobo

Technical Editor: Manikandan Kurup

Copy Editor: Safis Editing

Project Coordinator: Aparna Ravikumar Nair

Proofreader: Safis Editing

Indexer: Sejal Dsilva

Production Designer: Joshua Misquitta

First published: October 2021

Production reference: 1230921

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80056-887-7

www.packt.com

Sreeram Nudurupati is a data analytics professional with years of experience in designing and optimizing data analytics pipelines at scale. He has a history of helping enterprises, as well as digital natives, build optimized analytics pipelines by using the knowledge of the organization, infrastructure environment, and current technologies.

Karen J. Yang is a software engineer with computer science training in programming, data engineering, data science, and cloud computing. Her technical skills include Python, Java, Spark, Kafka, Hive, Docker, Kubernetes, CI/CD, Spring Boot, machine learning, data visualization, and cloud computing with AWS, GCP, and Databricks. As an author for Packt Publishing LLC, she has created three online instructional video courses, namely Apache Spark in 7 Days, Time Series Analysis with Python 3.x, and Fundamentals of Statistics and Visualization in Python. In her technical reviewer role, she has reviewed Mastering Big Data Analytics with PySpark, The Applied Data Science Workshop, and, most recently, Essential PySpark for Scalable Data Analytics.

Ayan Putatunda has 11 years of experience working with data-related technologies. He is currently working as an engineering manager of data engineering at Noodle.ai, based in San Francisco, California. He has held multiple positions, such as tech lead, principal data engineer, and senior data engineer, at Noodle.ai. He specializes in utilizing SQL and Python for large-scale distributed data processing. Before joining Noodle.ai he worked at Cognizant for 9 years in countries such as India, Argentina, and the US. At Cognizant, he worked with a wide range of data-related tools and technologies. Ayan holds a bachelor's degree in computer science from India and a master's degree in data science from the University of Illinois at Urbana-Champaign, USA.