Ashwin Rao - Foundations of Reinforcement Learning with Applications in Finance

1. ?

Foundations of Reinforcement Learning with Applications in Finance aims to demystify Reinforcement Learning, and to make it a practically useful tool for those studying and working in applied areas especially finance.

Reinforcement Learning is emerging as a powerful technique for solving a variety of complex problems across industries that involve Sequential Optimal Decisioning under Uncertainty. Its penetration in high-profile problems like self-driving cars, robotics, and strategy games points to a future where Reinforcement Learning algorithms will have decisioning abilities far superior to humans. But when it comes getting educated in this area, there seems to be a reluctance to jump right in, because Reinforcement Learning appears to have acquired a reputation for being mysterious and technically challenging.

This book provides a lucid and insightful understanding of the topic by emphasizing the foundational mathematics and implementing models and algorithms in well-designed Python code, along with robust coverage of several financial trading problems that can be solved with Reinforcement Learning. This book has been created after years of iterative experimentation on the pedagogy of these topics while being taught to university students as well as industry practitioners.

Features

• Focus on the foundational theory underpinning Reinforcement Learning and software design of the corresponding models and algorithms.

• Suitable as a primary text for courses in Reinforcement Learning, but also as supplementary reading for applied/financial mathematics, programming, and other related courses.

• Suitable for a professional audience of quantitative analysts or data scientists.

• Blends theory/mathematics, programming/algorithms and real-world financial nuances while always striving to maintain simplicity and to build intuitive understanding.

First edition published 2023

by CRC Press

6000 Broken Sound Parkway NW, Suite 300, Boca Raton, FL 33487-2742

and by CRC Press

4 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN

2023 Ashwin Rao, Tikhon Jelvis

CRC Press is an imprint of Taylor & Francis Group, LLC

Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers.

For permission to photocopy or use material electronically from this work, access

Trademark notice: Product or corporate names may be trademarks or registered trademarks and are used only for identification and explanation without intent to infringe.

ISBN: 978-1-032-12412-4 (hbk)

ISBN: 978-1-032-13429-1 (pbk)

ISBN: 978-1-003-22919-3 (ebk)

DOI: 10.1201/9781003229193

Typeset in TeXGyrePagella font

by KnowledgeWorks Global Ltd

Publisher's note: This book has been prepared from camera-ready copy provided by the authors.

We (Ashwin and Tikhon) have spent all of our educational and professional lives operating in the intersection of Mathematics and Computer ScienceAshwin for more than 3 decades and Tikhon for more than a decade. During these periods, we've commonly held two obsessions. The first is to bring together the (strangely) disparate worlds of Mathematics and Software Development. The second is to focus on the pedagogy of various topics across Mathematics and Computer Science. Fundamentally, this book was born out of a deep desire to release our twin obsessions so as to not just educate the next generation of scientists and engineers but to also present some new and creative ways of teaching technically challenging topics in ways that are easy to absorb and retain.

Apart from these common obsessions, each of us has developed some expertise in a few topics that come together in this book. Ashwin's undergraduate and doctoral education was in Computer Science, with specializations in Algorithms, Discrete Mathematics and Abstract Algebra. He then spent more than two decades of his career (across the Finance and Retail industries) in the realm of Computational Mathematics, recently focused on Machine Learning and Optimization. In his role as an Adjunct Professor at Stanford University, Ashwin specializes in Reinforcement Learning and Mathematical Finance. The content of this book is essentially an expansion of the content of the course CME 241 he teaches at Stanford. Tikhon's education is in Computer Science, and he has specialized in Software Design, with an emphasis on treating software design as mathematical specification of what to do versus computational mechanics of how to do. This is a powerful way of developing software, particularly for mathematical applications, significantly improving readability, modularity and correctness. This leads to code that naturally and clearly reflects mathematical structures, thus blurring the artificial lines between Mathematics and Programming. He has also championed the philosophy of leveraging programming as a powerful way to learn mathematical concepts. Ashwin has been greatly influenced by Tikhon on this philosophy and both of us have been quite successful in imparting our students with deep understanding of a variety of mathematical topics by using programming as a powerful pedagogical tool.

In fact, the key distinguishing feature of this book is to promote learning through an appropriate blend of A) intuitive understanding of the concepts, B) mathematical rigor and C) programming of the models and algorithms (with sound software design principles that reflect the mathematical specification). We've found this unique approach to teaching facilitates strong retention of the concepts because students are active learners when they code everything they are learning, in a manner that reflects the mathematical concepts. We have strived to create a healthy balance between content accessibility and intuition development on one hand versus technical rigor and completeness on the other hand. Throughout the book, we provide proper mathematical notation, theorems (and sometimes formal proofs) as well as well-designed working code for various models and algorithms. But we have always accompanied this formalism with intuition development using simple examples and appropriate visualization.

We want to highlight that this book emphasizes the foundational components of Reinforcement LearningMarkov Decision Processes, Bellman Equations, Fixed-Points, Dynamic Programming, Function Approximation, Sampling, Experience-Replay, Batch Methods, Value-based versus Policy-based Learning, balancing Exploration and Exploitation, blending Planning and Learning etc. So although we have covered several key algorithms in this book, we do not dwell on specifics of the algorithmsrather, we emphasize the core principles and always allow for various types of flexibility in tweaking those algorithms (our investment in modular software design of the algorithms facilitates this flexibility). Likewise, we have kept the content of the financial applications fairly basic, emphasizing the core ideas, and developing working code for simplified versions of these financial applications. Getting these financial applications to be effective in practice is a much more ambitious endeavorwe don't attempt that in this book, but we highlight what it would take to make it work in practice. The theme of this book is understanding of core concepts rather than addressing all the nuances (and frictions) one typically encounters in the real world. The financial content in this book is a significant fraction of the broader topic of Mathematical Finance, and we hope that this book provides the side benefit of a fairly quick yet robust education in the key topics of Portfolio Management, Derivatives Pricing and Order-Book Trading.