Intensive Learning (David Silver) 4: Model-Free Learning

0. Why model-free learning?

PS: The value of the iteration in the course is the value function; Zhou Zhihua Teacher's watermelon book is the state value function; The state-action value function is iterated in the course.

1. Monte Carlo method: Sum directly by sampling (V (s) = s (s)/n (s), where s (s) = s (s) + g (t), G (t) =r (t+1) +r (t+2) + ... ）

1.1. Monte Carlo Increment calculation method (V (s) = V (s) + A * (G (t)))

2. TD Algorithm (V (s) =)

3, MC and TD comparison

1) TD online real-time learning; MC can only learn in bulk

2) TD do not complete the sequence, do not need to know the full result; MC opposite

3) in the TD calculation project, the partial data is used to estimate V, which is biased; the return (MC) formula is unbiased; the return (MC) is random, the variance is strong, the TD is random and the variance is small.

4) MC is not sensitive to the initial value, and convergence for function approximation; TD is sensitive to the initial value, and the function approximation is not necessarily convergent.

5) High efficiency of TD, fast convergence and good convergence of MC

6) The MC converges toward the least squares; TD converges to MDP

7) TD explores Markov nature and is more effective in Markov environment; MC does not explore Markov nature and is more effective in non-Markov environment

4. Comparison of DP/MC/TD

1) Bootstrap: MC does not Bootstrap, Dp/td Bootstrap (bootstrap: Not with real final feedback as the goal, but with indirect intermediate results)

2) Sampling: MC/TD sampling; DP non-sampling

5. Three methods of unified discussion

From whether to bootstrap and backup number considerations:

1) Use only the sample backup value, bootstrap, is TD

2) Use only the sample backup value, not bootstrap, is MC

3) Use full backup value, Bootstrap, DP

4) Use full backup value, not bootstrap, is exhaustive search

6, TD (N) is the integration of TD and MC: TD (0) is n=1, forward step; TD (Infinity) is to see the end

7. TD (Lambda) is a weighted combination of TD (0)/TD (1)/td (2); So the TD algorithm will be more robust

8, the TD (lambda) of the back algorithm and qualification traces

Intensive Learning (David Silver) 4: Model-Free Learning