Intensive Learning (David Silver) 4: Model-Free Learning

Source: Internet
Author: User

0. Why model-free learning?

PS: The value of the iteration in the course is the value function; Zhou Zhihua Teacher's watermelon book is the state value function; The state-action value function is iterated in the course.

1. Monte Carlo method: Sum directly by sampling (V (s) = s (s)/n (s), where s (s) = s (s) + g (t), G (t) =r (t+1) +r (t+2) + ... )

1.1. Monte Carlo Increment calculation method (V (s) = V (s) + A * (G (t)))

2. TD Algorithm (V (s) =)

3, MC and TD comparison

1) TD online real-time learning; MC can only learn in bulk

2) TD do not complete the sequence, do not need to know the full result; MC opposite

3) in the TD calculation project, the partial data is used to estimate V, which is biased; the return (MC) formula is unbiased; the return (MC) is random, the variance is strong, the TD is random and the variance is small.

4) MC is not sensitive to the initial value, and convergence for function approximation; TD is sensitive to the initial value, and the function approximation is not necessarily convergent.

5) High efficiency of TD, fast convergence and good convergence of MC

6) The MC converges toward the least squares; TD converges to MDP

7) TD explores Markov nature and is more effective in Markov environment; MC does not explore Markov nature and is more effective in non-Markov environment

4. Comparison of DP/MC/TD

1) Bootstrap: MC does not Bootstrap, Dp/td Bootstrap (bootstrap: Not with real final feedback as the goal, but with indirect intermediate results)

2) Sampling: MC/TD sampling; DP non-sampling

5. Three methods of unified discussion

From whether to bootstrap and backup number considerations:

1) Use only the sample backup value, bootstrap, is TD

2) Use only the sample backup value, not bootstrap, is MC

3) Use full backup value, Bootstrap, DP

4) Use full backup value, not bootstrap, is exhaustive search

6, TD (N) is the integration of TD and MC: TD (0) is n=1, forward step; TD (Infinity) is to see the end

7. TD (Lambda) is a weighted combination of TD (0)/TD (1)/td (2); So the TD algorithm will be more robust

8, the TD (lambda) of the back algorithm and qualification traces

Intensive Learning (David Silver) 4: Model-Free Learning

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.