Jules S. Damji - Learning Spark: Lightning-Fast Data Analytics

by Jules S. Damji , Brooke Wenig , Tathagata Das , and Denny Lee

Copyright 2020 Databricks, Inc. All rights reserved.

Printed in the United States of America.

Published by OReilly Media, Inc. , 1005 Gravenstein Highway North, Sebastopol, CA 95472.

OReilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com .

• Acquisitions Editor: Jonathan Hassell
• Development Editor: Michele Cronin
• Production Editor: Deborah Baker
• Copyeditor: Rachel Head
• Proofreader: Penelope Perkins
• Indexer: Potomac Indexing, LLC
• Interior Designer: David Futato
• Cover Designer: Karen Montgomery
• Illustrator: Rebecca Demarest

• January 2015: First Edition
• July 2020: Second Edition

See http://oreilly.com/catalog/errata.csp?isbn=9781492050049 for release details.

The OReilly logo is a registered trademark of OReilly Media, Inc. Learning Spark, the cover image, and related trade dress are trademarks of OReilly Media, Inc.

The views expressed in this work are those of the authors, and do not represent the publishers views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

This work is part of a collaboration between OReilly and Databricks. See our statement of editorial independence.

978-1-492-05004-9

[GP]

Apache Spark has evolved significantly since I first started the project at UC Berkeley in 2009. After moving to the Apache Software Foundation, the open source project has had over 1,400 contributors from hundreds of companies, and the global Spark meetup group has grown to over half a million members. Sparks user base has also become highly diverse, encompassing Python, R, SQL, and JVM developers, with use cases ranging from data science to business intelligence to data engineering. I have been working closely with the Apache Spark community to help continue its development, and I am thrilled to see the progress thus far.

The release of Spark 3.0 marks an important milestone for the project and has sparked the need for updated learning material. The idea of a second edition of Learning Spark has come up many timesand it was overdue. Even though I coauthored both Learning Spark and Spark: The Definitive Guide (both OReilly), it was time for me to let the next generation of Spark contributors pick up the narrative. Im delighted that four experienced practitioners and developers, who have been working closely with Apache Spark from its early days, have teamed up to write this second edition of the book, incorporating the most recent APIs and best practices for Spark developers in a clear and informative guide.

The authors approach to this edition is highly conducive to hands-on learning. The key concepts in Spark and distributed big data processing have been distilled into easy-to-follow chapters. Through the books illustrative code examples, developers can build confidence using Spark and gain a greater understanding of its Structured APIs and how to leverage them. I hope that this second edition of Learning Spark will guide you on your large-scale data processing journey, whatever problems you wish to tackle using Spark.

Matei Zaharia, Chief Technologist,

Cofounder of Databricks, Asst. Professor at Stanford,

and original creator of Apache Spark