David Sweet - Experimentation for Engineers: From A/B testing to Bayesian optimization

Three stages of an A/B test: Design, Measure, and Analyze

Four iterations of a Bayesian optimization. In frames (a)(d), we run four iterations of the optimization. By frame (d), the parameter value (black dots) has stopped changing.

From A/B testing to Bayesian optimization

David Sweet

To comment go to liveBook

Manning

Shelter Island

For more information on this and other Manning titles go to

www.manning.com

For online information and ordering of these and other Manning books, please visit www.manning.com. The publisher offers discounts on these books when ordered in quantity.

For more information, please contact

Special Sales Department

Manning Publications Co.

20 Baldwin Road

PO Box 761

Shelter Island, NY 11964

Email:

2023 by Manning Publications Co. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps.

Recognizing the importance of preserving what has been written, it is Mannings policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine.

ISBN: 9781617298158

To B and Iz.

When I first entered the industry, I had the training of a theoretician but was presented with the tasks of an engineer. As a theoretician, I had worked with models using pen-and-paper or simulation. Where the model had a parameter, Ithe theoreticianwould try to understand how the model would behave with different values of it. But now Ithe engineerhad to commit to a single value: the one to use in a production system. How could I know what value to choose?

The short answer I received from more experienced practitioners was, Just try something. In other words, experiment. This set me off on a course of study of experimentation and experimental methods, with a focus on optimizing engineered systems.

Over the years, the methods applied by the teams I have been on, and by engineers in trading and technology generally, have become ever more precise and efficient. They have been used to optimize the execution of stock trades, market making, web search, online advertising, social media, online news, low-latency infrastructure, and more. As a result, trade execution has become cheaper and more fairly priced. Users regularly claim that web search and social media recommendations are so good that they worry their phones might be eavesdropping on them (theyre not).

Statistics-based experimental methods have a relatively short history. Sir R. A. Fisher published the seminal work, The Design of Experiments, in 1935less than a century ago. In it he discussed the class of experimental methods in which wed place an A/B test (chapter 2). In 1941, H. Hotelling wrote the paper Experimental determination of the maximum of a function, in which he discussed the modeling of a response surface (chapter 4). Response surface methodology was further explored by G. Box and K. P. Wilson. In 1947, A. Wald published the book Sequential Analysis, which studies the idea of analyzing experimental data measurement by measurement (chapter 3), rather than waiting until all measurements are available (as you would in an A/B test).

While this research was being done, the methods were being applied in industry: first in agriculture (Fishers methods), then in chemical and process industries (response surface methods). Later (from the 1950s to the 1980s) experimentation merged with statistical process control to give us the quality movements in manufacturing, exemplified by Toyotas Total Quality Management, and later, popularized by Six Sigma.

From the 1990s onward, internet companies have experienced an explosion of opportunity for experimentation as users have generated views, clicks, purchases, likescountless interactionsthat could be easily modified and measured with software on centralized web servers. In 2005, C.-C. Wang and S. R. Kulkarni wrote Bandit problems with side observations, which combined sequential analysis and supervised learning into a method now called a contextual bandit (chapter 5).

In 1975, J. Mockus wrote On the Bayes methods for seeking the extremal point, the foundation for Bayesian optimization (chapter 6), which takes an alternative approach to modeling a response surface and combines it with ideas from sequential analysis. This method was developed over the decades since by many researchers, including D. Jones et al., who wrote Efficient global optimization of expensive black-box functions, which, in 1998, applied some modern ideas to the method, making it look much more like the approach presented in this book.

In 2017, Vasant Dhar asked me to talk to his Trading Strategies and Systems class about high-frequency trading (HFT). He was gracious enough to allow me to focus specifically on the experimental optimization of HFT strategies. This was valuable to me because it gave me an opportunity to organize my thoughts and understanding of the topicto pull together the various bits and pieces that Id collected over the years. Slowly, those notes have grown into this book.

I hope this book saves you some time by putting all the bits and pieces Ive collected in one place and stitching them together into a single, coherent unit.

I am grateful to so many people for their hard work, for their support, and for their faith that this book could be brought into existence.