Sayon Dutta - Reinforcement Learning With TensorFlow: A Beginner’s Guide to Designing Self-Learning Systems With TensorFlow and OpenAI Gym

The architecture of asynchronous one-step Q-learning is very similar to DQN. An agent in DQN was represented by a set of primary and target networks, where one-step loss is calculated as the square of the difference between the state-action value of the current state s predicted by the primary network, and the target state-action value of the current state calculated by the target network. The gradients of the loss is calculated with respect to the parameters of the policy network, and then the loss is minimized using a gradient descent optimizer leading to parameter updates of the primary network.

The difference here in asynchronous one-step Q-learning is that there are multiple such learning agents, for instance, learners running and calculating this loss in parallel. Thus, the gradient calculation also occurs in parallel in different threads where each learning agent interacts with its own copy of the environment. The accumulation of these gradients in different threads over multiple time steps are used to update the policy network parameters after a fixed time step, or when an episode is over. The accumulation of gradients is preferred over policy network parameter updates because this avoids overwriting the changes perform by each of the learner agents.

Moreover, adding a different exploration policy to different threads makes the learning diverse and robust. This improves the performance owing to better exploration, because each of the learning agents in different threads is subjected to a different exploration policy. Though there are many ways to do this, a simple approach is to use different sample of epsilon for different threads while using -greedy .

The pseudo-code for asynchronous one-step Q-learning is shown as follows. Here, the following are the global parameters:

• : the parameters (weights and biases) of the policy network
• : parameters (weights and biases) of the target network
• T: overall time step counter

So far in this unit of discussing reinforcement learning application research domains, we saw how reinforcement learning is disrupting the field of robotics, autonomous driving, financial portfolio management, and solving games of extremely high complexity, such as Go. Another important domain which is likely to be disrupted by reinforcement learning is advertisement technology.

Before getting into the details of the problem statement and it's solution based on reinforcement learning, let's understand the challenges, business models, and bidding strategies involved, which will work as a basic prerequisite in understanding the problem that we will try to solve using a reinforcement learning framework. The topics that we will be covering in this chapter are as follows:

• Computational advertising challenges and bidding strategies

• Real-time bidding by reinforcement learning in display advertising

Robotics is associated with a high level of complexity in terms of behavior, which is difficult to hand engineer nor exhaustive enough to approach a task using supervised learning. Thus, reinforcement learning provides the kind of framework to capture such complex behavior.

Any task related to robotics is represented by high dimensional, continuous state, and action spaces. The environmental state is not fully observable. Learning in simulation alone is not enough to say the reinforcement learning agent is ready for the real world. In the case of robotics, a reinforcement learning agent should experience uncertainty in the real-world scenario but it's difficult and expensive to obtain and reproduce.

Robustness is the highest priority for robotics. In normal analytics or traditional machine learning problems, minor errors in data, pre-processing, or algorithms result in a significant change in behavior, especially for dynamic tasks. Thus, robust algorithms are required that can capture the real-world details. The next challenge for robot reinforcement learning is the reward function. Since the reward function plays the most important role in optimized learning, generating a domain specific reward function is needed that helps the learning agent to adapt better to the real world as quickly as possible. Thus, domain knowledge is the key behind devising a good reward function, which is again a hard task in robot machine learning.

Here, were will discuss the types of tasks in the field of robotics that can be achieved by the reinforcement learning algorithms we have studied in this book, and try connect them together to build a promising approach.