(六)Value Function Approximation-LSPI code (2)

標籤：

接上一篇，對LSPI演算法的code進行解釋

 

 1 # -*- coding: utf-8 -*- 2 """Contains main interface to LSPI algorithm.""" 3 ＃LSPI演算法的主要介面 4 from copy import copy 5  6 import numpy as np 7  8  9 def learn(data, initial_policy, solver, epsilon=10**-5, max_iterations=10):10     r"""Find the optimal policy for the specified data.11 　　＃對於特定的資料找到最優的策略12     Parameters輸入13     ----------
　　　　14 

＃data通常是一個採樣的列表，然而data的類型並不是很重要，會在求解器裡解決這些問題，比如當進行基於模型的學習時，我們輸入的是模型而不是資料！！

　　　　data:＃15         Generally a list of samples, however, the type of data does not matter16         so long as the specified solver can handle it in its solve routine. For17         example when doing model based learning one might pass in a model18         instead of sample data

　　　　

＃最初的策略會被保留？

19  　　initial_policy: Policy＃策略 
20  　　　　Starting policy. A copy of this policy will be made at the start of the 
21  　　　　method. This means that the provided initial policy will be preserved.

＃Solver ABC的一個子類，實現各種計算方法比如梯度下降，線性求解等等　　　　22     solver: Solver＃求解器23         A subclass of the Solver abstract base class. This class must implement24         the solve method. Examples of solvers might be steepest descent or25         any other linear system of equation matrix solver. This is basically26         going to be implementations of the LSTDQ algorithm.

　　　　＃策略權重更新的閾值？決定策略是否收斂，如果權重更新的大小小於這個值就認為是收斂了27     epsilon: float28         The threshold of the change in policy weights. Determines if the policy29         has converged. When the L2-norm of the change in weights is less than30         this value the policy is considered converged
　　　　＃最大的迭代次數31     max_iterations: int32         The maximum number of iterations to run before giving up on33         convergence. The change in policy weights are not guaranteed to ever34         go below epsilon. To prevent an infinite loop this parameter must be35         specified.36 37     Return＃返回38     ------
　　　　＃收斂了的策略，如果沒有收斂就返回最後一次計算的數值39     Policy40         The converged policy. If the policy does not converge by max_iterations41         then this will be the last iteration‘s policy.42 43     Raises＃一些錯誤的定義44     ------45     ValueError46         If epsilon is <= 047     ValueError48         If max_iteration <= 049 50     """51     if epsilon <= 0:＃檢查收斂閾值52         raise ValueError(‘epsilon must be > 0: %g‘ % epsilon)53     if max_iterations <= 0:＃檢查最大迭代次數54         raise ValueError(‘max_iterations must be > 0: %d‘ % max_iterations)55 56     # this is just to make sure that changing the weight vector doesn‘t57     # affect the original policy weights58     curr_policy = copy(initial_policy)＃這時為了保證更新的策略不會影響到最初的策略，所以我們複製出來一份最初策略59 60     distance = float(‘inf‘)＃距離初始化61     iteration = 0＃迭代次數初始化62     while distance > epsilon and iteration < max_iterations:＃當更新長度比較大，並且迭代次數沒達到最大值時進行迴圈63         iteration += 1＃迭代次數加１64         new_weights = solver.solve(data, curr_policy)＃用求解器求解最新的權重＠！只更新一次！65 66         distance = np.linalg.norm(new_weights - curr_policy.weights)＃計算新的權重和老權重的距離67         curr_policy.weights = new_weights更新權重68 69     return curr_policy返回計算後的策略

 

(六)Value Function Approximation-LSPI code (2)