A brief talk on function estimation problem in reinforcement learning-functions approximation in RL

The following is a brief discussion of the function estimation in reinforcement learning, where the basic principles of reinforcement learning, common algorithms and the mathematical basis of convex optimization are not discussed. Let's say you have a basic understanding of reinforcement learning (reinforcement learning).

Overview value function estimation Incremental/Gradient descent method batch processing method deep reinforcement Learning Analysis (DQN) double dqn with priority playback double DQN (prioritized Replay) dueling DQN nonparametric Estimation method Direct policy search non-modular Type of strategy search stochastic strategy reinforce G (PO) MDP TRPO actor-critic deterministic strategy DPG and DDPG model-based strategy search GPS Pilco

Overview

For the reinforcement learning problem of the state space, we need to use the method of function estimation to express various mapping relationships. Function estimation method can be divided into parameter estimation and nonparametric estimation, in which parametric estimation is divided into linear parametric estimation and nonlinear parametric estimation. In this paper, we mainly discuss parametric estimation. For weaker readers, you can refer to this more basic article. Value function Estimation

The process of estimating the value function can be seen as a supervised learning process where data and tag pairs are (st,ut) (S T, U t) (S_{t}, U_{t}). The objective functions of the training are: argminθ (q (s,a) −q^ (s,a,θ)) orargminθ (V (s) −v^ (s,θ)) arg minθ (Q (S, a) −q ^ (s, a, θ)) or arg minθ (V (s) −v ^ (s, θ)) \arg\min_{\theta} (q (S,a)-\hat{q} (S,a,\theta)) \quad \text{or} \quad \arg\min_{\theta} (V (s)-\hat {v} (S,\theta))

Incremental/Gradient-descent method

The basic principle of the

gradient descent can refer to unconstrained programming method in convex optimization problem. Here we ask for the smallest deviation, so the gradient descent method is used: Θt+1=θt+αdtθt + 1 =θt +αd t \theta_{t+1}=\theta_{t}+\alpha d_{t} here dt D t D_{t} is the direction of the deviation descent, which should be −∇θ (ut−v^ (st,θt))