Tanay Agrawal
Hyperparameter Optimization in Machine Learning
Make Your Machine Learning and Deep Learning Models More Efficient
1st ed.
Logo of the publisher
Tanay Agrawal
Bangalore, Karnataka, India
Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub via the books product page, located at https://www.apress.com/978-1-4842-6578-9 . For more detailed information, please visit https://www.apress.com/source-code .
ISBN 978-1-4842-6578-9 e-ISBN 978-1-4842-6579-6
https://doi.org/10.1007/978-1-4842-6579-6
Tanay Agrawal 2021
Apress Standard
The use of general descriptive names registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Distributed to the book trade worldwide by Springer Science+Business Media New York, 1 New York Plaza, Suite 4600, New York, NY 10004-1562, USA. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com, or visit https://www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation.
This book is dedicated to my parents and my grandparents.
Foreword 1
I have to admit that tweaking parameters by hand was something that I really enjoyed when I trained my first ML models. I would change a parameter, run my training script, and wait to see if the evaluation score improved. One of those guilty pleasures.
But as I spent more time in the ML world, I understood that there are other, more impactful areas where I could spend my time. I realized that I could (and should) outsource parameter tweaking somewhere.
I learned about random search and started using it in my projects. At some point, I felt I could do better than random search and started reading about more advanced hyperparameter optimization algorithms and libraries.
A lot of articles I found where pretty shallow and basic, but I remember reading this deep, hands-on yet easy-to-follow article about Hyperopt, one of the most popular HPO libraries. It was written by Tanay Agrawal. That article probably still is one of the more valuable articles Ive ever read on the subject. I mentioned it in one of my blog posts and this is how we met.
When Tanay told me that he was writing a book about hyperparameter optimization, without hesitation, I proposed to review it. I am not going to lie, I really wanted to read it before anyone else! To my surprise, Tanay agreed and I was able to give a few notes here and there.
This book truly is a deep dive into the theory and practice of hyperparameter optimization. I really like how it explains theory deeply but not in an overly complex way. The practical examples are centered on the libraries and frameworks that are heavily used today, which makes this book current and, most importantly, useful.
I recommend this book to any ML practitioner who wants to go beyond the basics and learn the why, how, and what of hyperparameter optimization.
Jakub Czakon
Senior Data Scientist
Neptune.ai
Foreword 2
In this book, Tanay takes you on an interactive journeyin the most literal sense, as each line of code can be run in a notebookof the depths of hyperparameters. It helps anyone to quickly get started on tuning and improving their deep learning project with any library they choose to use.
The author mindfully covers the inner workings of hyperparameters in ML models in a thorough but accessible fashion, which will allow you to understand and build upon them using different libraries. The book also demystifies the blackest of the black box: hyperparameter optimization in automated machine learning.
Its a friendly guide to a complicated subject, and yet its full of cutting-edge gems that even advanced practitioners will love.
Akruti Acharya
Data Scientist
Curl HG
Introduction
Choosing the right hyperparameters when building a machine learning model is one of the biggest problems faced by data science practitioners. This book is a guide to hyperparameter optimization (HPO). It starts from the very basic definition of hyperparameter and takes you all the way to building your own AutoML script using advance HPO techniques. This book is intended for both students and data science professionals.
The book consists of five chapters. Chapter discuss Bayesian optimization, related libraries, and AutoML.
The intent of this book is for readers to gain an understanding of the HPO concepts in an intuitive as well as practical manner, with code implementation provided for each section. I hope you enjoy it.
Acknowledgments
I would like to thank Kailash Ahirwar (CTO, MateLabs) for being a great mentor. A big thanks to teams from both MateLabs and Curl HG for their constant support. I am grateful to Akruti Acharya and Jakub Czakon for their insightful inputs while writing this book. I would also like to thank Paankhi Agrawal, Sahil Sharma, Dr. H.-Y. Amanieu, Anubhav Kesari, Abhishek Kumar, Amog Chandrashekar, and others. This book wouldnt have been possible without all the love and encouragement from my family.