Gradient descent vs. random gradient descent (implementation of Python)

1 #-*-coding:cp936-*-2 ImportNumPy as NP3 ImportMatplotlib.pyplot as Plt4 5 6 #Construct Training Data7x = Np.arange (0., 10., 0.2)8m = Len (x)#number of training data points9x0 = Np.full (M, 1.0)                 TenInput_data = Np.vstack ([x0, X]). T#bias B as the first component of a weight vector OneTarget_data = 2 * x + 5 +Np.random.randn (m) A  -  - #two types of termination conditions theLoop_max = 10000#Maximum number of iterations (to prevent a dead loop) -Epsilon = 1e-3 -  - #Initialize weight value + np.random.seed (0) -W = NP.RANDOM.RANDN (2) + #w = Np.zeros (2) A  atAlpha = 0.001#Step Size (Note that the value is too long to cause oscillation, slow convergence rate) -diff =0. -Error = Np.zeros (2)  -Count = 0#Number of Cycles -finish = 0#Stop Flag - #-------------------------------------------stochastic gradient descent algorithm----------------------------------------------------------  in " " - While count < loop_max: to count + = 1 +  - # Traverse Training data set, update weights continuously the For I in range (m): * diff = Np.dot (w, input_data[i])-target_data[i] # Training set generation, calculating error values $ Panax Notoginseng # Use a random gradient descent algorithm to update weights using only one set of training data - w = W-alpha * diff * Input_data[i] the  + #------------------------------termination condition judgment----------------------------------------- A # If not terminated, continue to read the sample for processing, and if all the samples have been read, then the loop resumes reading the sample from the beginning to process.  the  + #----------------------------------termination condition judgment----------------------------------------- - Note: There are several iteration termination conditions, and the position of the judgment statement. Termination judgment can be placed after the weight vector is updated once, or can be placed after the update m times.  $ if Np.linalg.norm (W-error) < epsilon: # termination Condition: The absolute error of weight vectors calculated before and after two times is sufficiently small $ finish = 1 -  Break - Else: the error = W - print ' Loop count =%d '% count, ' \tw:[%f,%f] '% (w[0], w[1])Wuyi " " the  -  Wu #-----------------------------------------------Gradient Descent method-----------------------------------------------------------  -  whileCount <Loop_max: AboutCount + = 1 $  -     #The standard gradient drop is a summary error for all samples prior to the weight update, and the weighted value of the random gradient descent is updated by examining a training sample. -     #in the standard gradient descent, each step of the weight update sums the multiple samples and requires more computation -Sum_m = Np.zeros (2) A      forIinchRange (m): +dif = (Np.dot (w, input_data[i])-target_data[i]) *Input_data[i] theSum_m = Sum_m +dif -  $w = W-alpha * sum_m#Note the value of step alpha, which leads to oscillation the     #w = w-0.005 * sum_m # Alpha takes 0.005 to generate oscillation and requires an alpha to be smaller the      the     #determine if it is convergent the     ifNp.linalg.norm (W-error) <Epsilon: -finish = 1 in          Break the     Else: theError =W About Print 'Loop count =%d'% count,'\tw:[%f,%f]'% (W[0], w[1]) the  the  thePlt.plot (x, Target_data,'k +') +Plt.plot (x, w[1] * x + w[0],'R') -Plt.show ()

Reference: Http://www.tuicool.com/articles/MRbee2i

Gradient descent vs. random gradient descent (implementation of Python)