Six Value Function APPROXIMATION-LSPI Code (2)

Next, the code of the LSPI algorithm is explained

1 #-*-coding:utf-8-*-2 """Contains main interface to LSPI algorithm."""3Main interface of #lspi algorithm4  fromCopyImportCopy5 6 ImportNumPy as NP7 8 9 defLearn (data, Initial_policy, Solver, Epsilon=10**-5, max_iterations=10):TenR"""Find the optimal policy for the specified data. One# Find the best strategy for your specific data A Parameters Input -     ----------
　　　　 - 

#data is usually a sample list, but the type of data is not very important, it solves these problems in the solver, for example, when we do model-based learning, we enter models instead of data!!

data:#        generally a list of samples, however, the type of data does not matter C7/>so long as the specified solver can handle it in its solve routine. for-        example when doing model based learning one might pass in a model
    instead of Sample Data

　　　　

# The initial strategy will be retained?

 + initial_policy:policy# Strategy
 - starting policy. A copy of this policy is made at the start of the
 + method. This means the provided initial policy would be preserved.

#solver a subclass of ABC to achieve various computational methods such as gradient descent, linear solution, etc. A solver:solver# Solver at A Subclass of the Solver abstract base class. This class must implement - The solve method. Examples of solvers might be steepest descent or - Any other linear system of Equation matrix solver. this is basically - Going to be implementations of the LSTDQ algorithm.

# Thresholds for policy weight updates? Determines whether the policy converges, if the size of the weight update is less than this value is considered to be convergent - epsilon:float - The threshold of the change in policy weights. Determines if the policy in Has converged. when the l2-norm of the weights are less than - This value of the policy is considered converged
# Maximum number of iterations to Max_iterations:int + The maximum number of iterations to run before giving - Convergence. the change in policy weights is not guaranteed to ever the go below Epsilon. To prevent a infinite loop This parameter must is * specified. $ Panax Notoginseng return# return - ------
# A convergent strategy that returns the last calculated value if there is no convergence the Policy + The converged policy. If the policy does not converge by Max_iterations A Then this would be is the last iteration's policy. the  + raises# Some of the wrong definitions -     ------ $ ValueError $ If Epsilon is <= 0 - ValueError - If max_iteration <= 0 the  -     """Wuyi     ifEpsilon <=0:# Check Convergence threshold value the         RaiseValueError ('Epsilon must be > 0:%g'%Epsilon) -     ifMax_iterations <=0:# Check the maximum number of iterations Wu         RaiseValueError ('max_iterations must be > 0:%d'%max_iterations) -  About     #This was just to make sure that changing the weight vector doesn ' t $     #affect the original policy weights -Curr_policy =copy (initial_policy) # At this point to ensure that the updated policy does not affect the original policy, so we copied out a copy of the original policy -  -Distance = float ('inf') # Distance initialization Aiteration =0# Number of iterations initialized +      whileDistance > Epsilon andIteration <max_iterations:# Loop When the update length is large and the iteration count does not reach the maximum value theiteration + = # number of iterations plus 1 -New_weights =solver.solve (data, Curr_policy) # solves the latest weights with the Solver @! Update only once!  $  theDistance = Np.linalg.norm (New_weights-curr_policy.weights) # Calculate the distance between the new weights and the old weights theCurr_policy.weights =new_weights Update Weights the  the     returnCurr_policy returning a calculated policy

Six Value Function APPROXIMATION-LSPI Code (2)