Practical Simulations for Machine Learning
by Paris Buttfield-Addison , Mars Buttfield-Addison , Tim Nugent , and Jon Manning
Copyright 2022 Secret Lab. All rights reserved.
Printed in the United States of America.
Published by OReilly Media, Inc. , 1005 Gravenstein Highway North, Sebastopol, CA 95472.
OReilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.
- Acquisitions Editor: Rebecca Novack
- Development Editor: Michele Cronin
- Production Editor: Christopher Faucher
- Copyeditor: Piper Editorial Consulting, LLC
- Proofreader: Audrey Doyle
- Indexer: nSight, Inc.
- Interior Designer: David Futato
- Cover Designer: Karen Montgomery
- Illustrator: Kate Dullea
Revision History for the First Edition
- 2022-06-07: First Release
See http://oreilly.com/catalog/errata.csp?isbn=9781492089926 for release details.
The OReilly logo is a registered trademark of OReilly Media, Inc. Practical Simulations for Machine Learning, the cover image, and related trade dress are trademarks of OReilly Media, Inc.
The views expressed in this work are those of the authors and do not represent the publishers views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.
978-1-492-08992-6
[LSI]
Preface
Welcome to Practical Simulations for Machine Learning! This book combines two of our favorite things: video game engines and artificial intelligence. We hope you enjoy reading it as much as we enjoyed writing it.
Specifically, this book explores the use of Unity, a product that used to be called a game engine but now likes to be called a platform for creating and operating interactive, real-time 3D content. Thats a lot of words, but they basically boil down to this: Unity is a platform for building things in 3D, and though it has traditionally been used for video game development, it can be used to build anything that can be represented in 3D, by using a combination of 3D graphics, physics simulations, and inputs of some kind.
By combining a platform for creating and operating interactive, real-time 3D content with machine learning tools, you can use the 3D world you create to train a machine learning model, kind of like its the real world. Its not actually like the real world, but its fun to imagine, and there are some legitimately useful connections to the real world (such as being able to generate both data for use in real-world machine learning applications, as well as models that can be transposed to physical, real-world objects, like robots).
Tip
When we say real-world, we actually mean physical.
Combining Unity with machine learning is a great way to create both simulations and synthetic data, which are the two different topics we cover in this book.
Resources Used in This Book
We recommend following along with the book by writing code yourself as you progress through each chapter.
If you become stuck, or just want to archive a copy of our version of the code, you can find what you need via our website.
For some activities we work through in the book, youll need a copy of the resources to get certain assets, so we do recommend you download it.
Audience and Approach
We wrote this book for programmers and software engineers who are interested in machine learning, but are not necessarily machine learning engineers. If you have a passing interest in machine learning, or are starting to work more in the machine learning space, then this book is for you. If youre a game developer, who kind of already knows Unity, or another game engine, and wants to learn machine learning (for either games or some other application) then this book is for you too.
If youre already a machine learning expert, this book is for you as well, but in a different way: we dont go too deep on the whys and hows of machine learning. So, if you already know whats going on deep within PyTorch, and similar frameworks, youll do just fine here. And if you dont already know whats deep within the world of machine learning, youll be fine too because everything is very accessible. The point of simulations and synthesis with Unity is that you dont need to know the ins and outs of whats going on. It all kind of just works (famous last words, we know).
Anyway, this book is for you if youre coming from software, machine learning, or games. Theres something for everyone here. We teach you just enough Unity and just enough machine learning to be dangerous and well provide you with jumping-off points to learn more about the paths that youre interested in.
Organization of This Book
This book is divided into three parts.
, introduces the topics of simulation and synthesis, and eases you in gently with a simple activity based on each.
, is dedicated to simulation. This is the biggest part of the book, because simulations are a much, much bigger topic than synthesis. In this part, we go almost step-by-step through a collection of simulation activities, building additional concepts and approaches as we go. By the end of this part, youll have been exposed to many of the different paths through simulation that you can take.
, is dedicated to synthesis. This is a much smaller part than simulation, but is still crucial. Youll learn the fundamentals of creating synthetic data with Unity, and by the end youll be equipped to make basically any kind of synthesis you might need.
Using This Book
Weve structured this book around activities. We hope youll work through the activities with us, and add your own spin where youre so inclined (but dont feel like you have to).
We took an activity-based approach because we feel its the best way to learn the bits you need from both the Unity game engine, and the machine learning side of things. We didnt want to have to teach you everything about Unity, and theres no room in the book to unpack all the details of machine learning.
By going from activity to activity, we can introduce or exclude things as needed. We really hope you enjoy our choice of activities!
Our Tasks
For simulation, well be building:
A ball that can roll itself to a target, in (we know, it sounds too amazing to be true, but it is!)
A cube that can push a block into a goal area, in
A simple self-driving car, navigating a track, in
A ball that seeks a coin, trained by imitating human demonstrations, in