Paper notes: Dueling Network architectures for deep reinforcement learning

Dueling Network architectures for deep reinforcement learning

ICML Best Paper

Absrtact: The contribution point of this paper is mainly in the DQN network structure, the features of convolutional neural network are divided into two paths, namely: the state value function and the State-dependent action Advantage function. .

The main feature of this design is generalize learning across actions without imposing any change to the underlying reinforcement learning Algori tHM.

the results of this paper show that this kind of design structure can get better strategy evaluation when many similar values are found in the actions. (Policy evaluation).

　　Introduction : The article first began to explain the recent deep learning network structure of the rapid development, listed a number of successful cases. It also mentions the existing DRL development, which relies on the design of the new RL algorithm rather than the new network result. Based on the network structure, this paper proposes a new structure and the existing RL algorithm to better combine.

The q-network of the traditional single process and the new proposed dueling Q-network are as follows:

　　

　　

As can be seen, the dueling network presented in this paper divides the subsequent output into two branches to predict the state-value and the advantages for each actions respectively; The Green Output module performs the following formula (9) to combine the two outputs. Both network structures are q-values that output each action.

Intuitively, the dueling architecture can learn which State is valuable (which states is (or is not) valuable), without having to learn the effects of each action on each of the States. This is particularly effective in situations where the actions do not affect the environment. To illustrate this point, we can consider the salient figure shown in Figure 2.

This figure shows the value and advantage saliency maps for two different time steps. At a time step, the value network Steam takes note of the road, especially the horizon, where new vehicles appear. He also noticed the score. The advantage stream does not care about visual input on the other hand, because when no vehicle is present, you can choose the action at will, and the environment has little effect. But the second figure shows that

Paper notes: Dueling Network architectures for deep reinforcement learning