IEEE Press. - Reinforcement Learning and Approximate Dynamic Programming for Feedback Control

IEEE Press

445 Hoes Lane

Piscataway, NJ 08854

IEEE Press Editorial Board 2012

John Anderson, Editor in Chief

Kenneth Moore, Director of IEEE Book and Information Services (BIS)

Cover Illustration: Courtesy of Frank L. Lewis and Derong Liu

Cover Design: John Wiley & Sons, Inc.

Copyright 2013 by The Institute of Electrical and Electronics Engineers, Inc.

Published by John Wiley & Sons, Inc., Hoboken, New Jersey. All rights reserved

Published simultaneously in Canada

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com . Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permission .

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com .

Library of Congress Cataloging-in-Publication Data:

Reinforcement learning and approximate dynamic programming for feedback control / edited by Frank L. Lewis, Derong Liu.

p. cm.

ISBN 978-1-118-10420-0 (hardback)

1. Reinforcement learning. 2. Feedback control systems. I. Lewis, Frank L.

II. Liu, Derong, 1963

Q325.6.R464 2012

003.5dc23

2012019014

Preface

Modern day society relies on the operation of complex systems including aircraft, automobiles, electric power systems, economic entities, business organizations, banking and finance systems, computer networks, manufacturing systems, and industrial processes. Decision and control are responsible for ensuring that these systems perform properly and meet prescribed performance objectives. The safe, reliable, and efficient control of these systems is essential for our society. Therefore, automatic decision and control systems are ubiquitous in human engineered systems and have had an enormous impact on our lives. As modern systems become more complex and performance requirements more stringent, improved methods of decision and control are required that deliver guaranteed performance and the satisfaction of prescribed goals.

Feedback control works on the principle of observing the actual outputs of a system, comparing them to desired trajectories, and computing a control signal based on that error, which is used to modify the performance of the system to make the actual output follow the desired trajectory. The optimization of sequential decisions or controls that are repeated over time arises in many fields, including artificial intelligence, automatic control systems, power systems, economics, medicine, operations research, resource allocation, collaboration and coalitions, business and finance, and games including chess and backgammon. Optimal control theory provides methods for computing feedback control systems that deliver optimal performance. Optimal controllers optimize user-prescribed performance functions and are normally designed offline by solving HamiltonJacobiBellman (HJB) design equations. This requires knowledge of the full system dynamics model. However, it is often difficult to determine an accurate dynamical model of practical systems. Moreover, determining optimal control policies for nonlinear systems requires the offline solution of nonlinear HJB equations, which are often difficult or impossible to solve. Dynamic programming (DP) is a sequential algorithmic method for finding optimal solutions in sequential decision problems. DP was developed beginning in the 1960s with the work of Bellman and Pontryagin. DP is fundamentally a backwards-in-time procedure that does not offer methods for solving optimal decision problems in a forward manner in real time.

The real-time adaptive learning of optimal controllers for complex unknown systems has been solved in nature. Every agent or system is concerned with acting on its environment in such a way as to achieve its goals. Agents seek to learn how to collaborate to improve their chances of survival and increase. The idea that there is a cause and effect relation between actions and rewards is inherent in animal learning. Most organisms in nature act in an optimal fashion to conserve resources while achieving their goals. It is possible to study natural methods of learning and use them to develop computerized machine learning methods that solve sequential decision problems.

Reinforcement learning (RL) describes a family of machine learning systems that operate based on principles used in animals, social groups, and naturally occurring systems. RL methods were used by Ivan Pavlov in the 1860s to train his dogs. RL refers to an actor or agent that interacts with its environment and modifies its actions, or control policies, based on stimuli received in response to its actions. RL computational methods have been developed by the Computational Intelligence Community that solve optimal decision problems in real time and do not require the availability of analytical system models. The RL algorithms are constructed on the idea that successful control decisions should be remembered, by means of a reinforcement signal, such that they become more likely to be used another time. Successful collaborating groups should be reinforced. Although the idea originates from experimental animal learning, it has also been observed that RL has strong support from neurobiology, where it has been noted that the dopamine neurotransmitter in the basal ganglia acts as a reinforcement informational signal, which favors learning at the level of the neurons in the brain. RL techniques were first developed for Markov decision processes having finite state spaces. They have been extended for the control of dynamical systems with infinite state spaces.

One class of RL methods is based on the actorcritic structure, where an actor component applies an action or a control policy to the environment, whereas a critic component assesses the value of that action. Actorcritic structures are particularly well adapted for solving optimal decision problems in real time through reinforcement learning techniques. Approximate dynamic programing (ADP) refers to a family of practical actorcritic methods for finding optimal solutions in real time. These techniques use computational enhancements such as function approximation to develop practical algorithms for complex systems with disturbances and uncertain dynamics. Now, the ADP approach has become a key direction for future research in understanding brain intelligence and building intelligent systems.