Chip Huyen - Designing Machine Learning Systems: An Iterative Process for Production-Ready Applications

by Chip Huyen

Copyright 2022 Huyen Thi Khanh Nguyen. All rights reserved.

Printed in the United States of America.

Published by OReilly Media, Inc. , 1005 Gravenstein Highway North, Sebastopol, CA 95472.

OReilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com .

• Acquisitions Editor: Nicole Butterfield
• Development Editor: Jill Leonard
• Production Editor: Gregory Hyman
• Copyeditor: nSight, Inc.
• Proofreader: Piper Editorial Consulting, LLC
• Indexer: nSight, Inc.
• Interior Designer: David Futato
• Cover Designer: Karen Montgomery
• Illustrator: Kate Dullea

• May 2022: First Edition

See http://oreilly.com/catalog/errata.csp?isbn=9781098107963 for release details.

The OReilly logo is a registered trademark of OReilly Media, Inc. Designing Machine Learning Systems, the cover image, and related trade dress are trademarks of OReilly Media, Inc.

The views expressed in this work are those of the author, and do not represent the publishers views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

978-1-098-10796-3

[LSI]

Ever since the first machine learning course I taught at Stanford in 2017, many people have asked me for advice on how to deploy ML models at their organizations. These questions can be generic, such as What model should I use? How often should I retrain my model? How can I detect data distribution shifts? How do I ensure that the features used during training are consistent with the features used during inference?

These questions can also be specific, such as Im convinced that switching from batch prediction to online prediction will give our model a performance boost, but how do I convince my manager to let me do so? or Im the most senior data scientist at my company and Ive recently been tasked with setting up our first machine learning platform; where do I start?

My short answer to all these questions is always: It depends. My long answers often involve hours of discussion to understand where the questioner comes from, what theyre actually trying to achieve, and the pros and cons of different approaches for their specific use case.

ML systems are both complex and unique. They are complex because they consist of many different components (ML algorithms, data, business logic, evaluation metrics, underlying infrastructure, etc.) and involve many different stakeholders (data scientists, ML engineers, business leaders, users, even society at large). ML systems are unique because they are data dependent, and data varies wildly from one use case to the next.

For example, two companies might be in the same domain (ecommerce) and have the same problem that they want ML to solve (recommender system), but their resulting ML systems can have different model architecture, use different sets of features, be evaluated on different metrics, and bring different returns on investment.

Many blog posts and tutorials on ML production focus on answering one specific question. While the focus helps get the point across, they can create the impression that its possible to consider each of these questions in isolation. In reality, changes in one component will likely affect other components. Therefore, its necessary to consider the system as a whole while attempting to make any design decision.

This book takes a holistic approach to ML systems. It takes into account different components of the system and the objectives of different stakeholders involved. The content in this book is illustrated using actual case studies, many of which Ive personally worked on, backed by ample references, and reviewed by ML practitioners in both academia and industry. Sections that require in-depth knowledge of a certain topice.g., batch processing versus stream processing, infrastructure for storage and compute, and responsible AIare further reviewed by experts whose work focuses on that one topic. In other words, this book is an attempt to give nuanced answers to the aforementioned questions and more.