Andrew Ng Open Course Notes-reinforcement learning and Control

Netease Open Course: 16th courses
Notes, 12

The previous supervised learning clearly tells you what the correct y is for a specified X.
But some sequential demo-making problems, such as playing chess or helicopter self-driving
I cannot know exactly how the next step is correct, because it is a continuous and serialized decision. For example, it is not good to know the previous choice until the final helicopter crash or the game is lost, but which part of the decision is wrong when there are so many steps in the middle? It can be seen that this is a complicated problem.

The basic idea of reinforcement learning is that, since you don't know how to make it right, try it and then gradually strengthen the behavior to get the correct results based on the feedback.
Interesting. Think about the process of learning,
For example, if you have learned to ride a bicycle, you don't know how to do it.
You can only try it. If you fall down, the subconscious will change the steps. If you can ride it up, it will strengthen the previous steps and slowly learn to ride a bike. This is a typical reinforcement learning.

 

Markov demo-processes

A Markov demo-process is a tuple, where:

A Markov decision process can be expressed as a 5-tuples.
S, set of States. Status set. For helicopter driving, it is the current position and direction

A, set of actions. A set of actions. For helicopter driving, it is the next step to take control, top, bottom, left, right, front and back, etc.

, State transition probabilities. Probability Distribution of State migration. Because the action is taken in the S state, the final state is random, which reflects the Markov process.
For example, if you move a helicopter to the right, but for example, it is possible to move the helicopter to the front right, or to the back right, or to the front because of wind or other noise, it must be expressed by probability.

, Discount factor. We will see later that this parameter is used to adjust the weight between the current decision and the future decision.

, Reward function. indicates the feedback of a behavior in S state, and returns the function ., Often, the return function only takes the state as the parameter and is understood as moving to the current state. What is the role of a return function in achieving a good result?

As you can imagine, the Markov decision-making process should be like this process.

In this process, we can see the sum of the Reward function. here we can see that the value is about 0.99, and the subsequent Reward function is multiplied, such as discount.
The earlier the decision is made, the more important it is. Is it reasonable?

 

That is,

So our goal of reinforcement learning is to maximize the expected value of the total payoff.

This is not hard to understand. The larger the value of the Reward function, the more correct it is.

For example, if you look at Ng's example, 12 grids, one of which is an obstacle. The final arrival of + 1 () is a success, and the arrival of-1 () is a failure.

Assume that the starting point is (3, 1) in the figure, and the action is N, that is, going north-facing.

Face up,

Maximum probability of reaching (0.8 ),
As a disturbance to noise, there is also a probability that 0.1 will reach (), or)
The probability of reaching other cells is 0.

Let's see how to set the Reward function value?

REACH (), that is, success, Reward function is 1
() Is an error, and the Reward function is-1.
The Reward function of other cells is set to-0.02.

This is a skill. Setting the remaining cells as a small negative reward means more electricity or energy consumption for navigation or robots.
If you want the reward function and the maximum value, you must minimize the number of steps.

 

Bellman equations

Defines the ing between S and A, that is, the action corresponding to in a certain state, called Policy

Obviously, our goal is to find the policy function that maximizes total payoff.

The starting point is S, the policy is, and the total payoff of the final process is called the value function. The Value Function

The above formula can be written as the following recursive form,

S1 is unknown and there are multiple possibilities. The above formula is written

This is called Bellman equation.
It is divided into two parts,
Where R (s) is immediate reward and returns immediately
Part 2: Future discounted rewards, which can also be written

What is the use of Bellman equation?
When the Reward function R (s) is known, it is difficult to directly find the function, because the S in each step is uncertain in the Process of Markov decision-making.

The Bellman equation can be used to list such an equation for every second,

As a variable, for n States, there will be n variables. There are n Bellman equations. by solving the equations, we can solve each

Let's look at the example above,
Indicates a policy. You can see that each grid is drawn and the direction of movement is selected, that is, action.

The Bellman equations are listed for status (3, 1), where all the elements in the frame are variables.

When the preceding equations are listed for each State, you can solve the variable values by solving the equations.

To obtain, as shown in

 

OK. Continue to define

Our goal is to find the largest

That is, because R (s) is a constant, another representation of the Bellman equation is obtained.

So we define, so that,

This is the best policy we are looking.

 

Value Iteration and Policy Iteration

Now we will introduce the algorithm to solve the above optimal problem. The introduced algorithm only targets limited states and actions MDPs.

Value Iteration

The algorithm is quite simple, that is, using the Bellman equation to constantly update v. it can be proved that V will converge to V *.

After obtaining v *, use

You can find

In the above example (3, 1), how to set the policy? Calculate the values to W or to N based on the formula above, and find that to W is a better policy.

There are two types of V update: asynchronous and synchronous,
After all the new values of S are calculated for synchronization, all values are updated at one time.
Asynchronous computing is an update, so the v calculation of s will use the V

 

Policy Iteration

Randomly specified, and then solved through the equations of the Bellman equation, updated according to the new ......

Both algorithms converge,

Policy iteration converges faster when the State is relatively small, but it cannot be used for the MDP of the huge state set because it requires solving the equations, and the overhead is too large.

Therefore, value iteration is often used for MDP of large State sets.

 

Learning a model for an MDP

The preceding algorithms are based on the assumption that the state transition probabilities and rewards functions are known.
But in many cases, they are unknown, so we need to estimate them
Here, the rewards function is also known, because it is provided by you, you should know, except for some special cases

So let's take a special look at how the state transition probabilities are estimated,

In fact, it is very simple. Just try it several times and then make statistics based on the actual situation,
And this P should be updated online and more accurate.

In addition, for case 0/0, replace 1/| S |

Using a similar procedure, if R is unknown, we can also pick our estimate of the expected immediate reward R (s) in State S to be the average reward
Observed in State S.

If R is unknown, we can also use the Mean observed in the experiment as the estimated value. I don't quite understand how can we observe the value of the Reward function in the experiment?