Svilen Gospodinov - Concurrent Data Processing in Elixir: Fast, Resilient Applications with OTP, GenStage, Flow, and Broadway

Copyright 2021 The Pragmatic Programmers, LLC. This book is licensed to the individual who purchased it. We don't copy-protect it because that would limit your ability to use it for your own purposes. Please don't break this trustyou can use this across all of your devices but please do not share this copy with other members of your team, with friends, or via file sharing services. Thanks.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and The Pragmatic Programmers, LLC was aware of a trademark claim, the designations have been printed in initial capital letters or in all capitals. The Pragmatic Starter Kit, The Pragmatic Programmer, Pragmatic Programming, Pragmatic Bookshelf and the linking g device are trademarks of The Pragmatic Programmers, LLC.

Every precaution was taken in the preparation of this book. However, the publisher assumes no responsibility for errors or omissions, or for damages that may result from the use of information (including program listings) contained herein.

The Pragmatic Bookshelf is an agile publishing company. Were here because we want to improve the lives of developers. We do this by creating timely, practical titles, written by programmers for programmers.

Our Pragmatic courses, workshops, and other products can help you and your team create better software and have more fun. For more information, as well as the latest Pragmatic titles, please visit us at http://pragprog.com.

Our ebooks do not contain any Digital Restrictions Management, and have always been DRM-free. We pioneered the beta book concept, where you can purchase and read a book while its still being written, and provide feedback to the author to help make a better book for everyone. Free resources for all purchasers include source code downloads (if applicable), errata and discussion forums, all available on the book's home page at pragprog.com. Were here to make your life easier.

Want to keep up on our latest titles and announcements, and occasional special offers? Just create an account on pragprog.com (an email address and a password is all it takes) and select the checkbox to receive newsletters. You can also follow us on twitter as @pragprog.

If you buy directly from pragprog.com, you get ebooks in all available formats for one price. You can synch your ebooks amongst all your devices (including iPhone/iPad, Android, laptops, etc.) via Dropbox. You get free updates for the life of the edition. And, of course, you can always come back and re-download your books when needed. Ebooks bought from the Amazon Kindle store are subject to Amazon's polices. Limitations in Amazon's file format may cause ebooks to display differently on different devices. For more information, please see our FAQ at pragprog.com/#about-ebooks. To learn more about this book and access the free resources, go to https://pragprog.com/book/sgdpelixir, the book's homepage.

Thanks for your continued support,

Andy Hunt
The Pragmatic Programmers

The team that produced this book includes: Dave Rankin (CEO) Janet Furlow (COO) Tammy Coron (Managing Editor) Series editor: Bruce A. Tate Jacquelyn Carter (Development Editor) Karen Galle (Copy Editor) Potomac Indexing, LLC (Indexing) Gilson Graphics (Layout) Andy Hunt and Dave Thomas (Founders)

For customer support, please contact .

For international rights, please contact .

Elixir started as a learning exercise. I wanted to dive into the Erlang Virtual Machine, and creating a new language would also allow me to explore how programming languages actually work. It was a period where multiple mistakes were made but they were essential in discovering what Elixir should be.

Once the general vision of the language was established, I got the approval to start working on it part time. I personally had the ambition of using it for building scalable and robust web applications. However, I didnt want Elixir to be tied to the web. My goal was to design a general purpose and extensible programming language, allowing developers to bring it to new domains and create a diverse ecosystem.

The Erlang platform has been used extensively for networking and distributed systems since its inception. This meant Elixir would naturally be a good fit in those domains, and building data-processing and data-ingestion pipelines were a natural extension of those areas. In fact, projects like Riak have already explored using the Erlang platform for working with data at large scales, although they were always presented as complete solutions, and I wanted Elixir to focus on providing a bottom-up approach.

After Elixir v1.0 was released, the Elixir team and I finally started turning our attention to this problem. At the same time, projects like Reactive Extensions a new behaviourbuilt on top of GenServerfor exchanging events with back-pressure between Elixir processes.

To provide a more practical example of how GenStage could be used at scale, we also built Flow,