Example of implementing Lasso regression and Ridge regression algorithm with TensorFlow

This article mainly introduces the use of TensorFlow to achieve lasso regression and ridge regression algorithm example, has a certain reference value, now share to everyone, the need for friends can refer to

There are also some regularization methods that limit the effect of coefficients in the output of the regression algorithm, and the two most common regular methods are lasso regression and ridge regression.

The lasso regression and ridge regression algorithms are very similar to conventional linear regression algorithms, with the addition of regular items in the formula to limit the slope (or net slope). The main reason for this is to limit the effect of features on dependent variables by adding a loss function that relies on slope a.

For the Lasso regression algorithm, add an entry on the loss function: a given multiple of slope a. We use TensorFlow's logic operations, but without the gradient associated with these operations, we use a continuous estimate of the step function, also called the continuous step function, which jumps at the cutoff point. You'll see how to use the Lasso regression algorithm in a minute.

For the ridge regression algorithm, a L2 norm, the L2 of the slope coefficient, is added.

# LASSO and Ridge regression# LASSO regression and Ridge regression # # This function shows what to use TensorFlow to solve LASSO or # Ridge Regressio N for # y = Ax + b# # We'll use the iris data, specifically: # y = sepal Length # x = Petal width# Import required Lib Rariesimport Matplotlib.pyplot as Pltimport sysimport NumPy as Npimport tensorflow as Tffrom sklearn import Datasetsfrom t Ensorflow.python.framework import ops# Specify ' Ridge ' or ' LASSO ' regression_type = ' LASSO ' # clear out old Graphops.reset_d Efault_graph () # Create graphsess = tf. Session () # # # # Load iris data#### iris.data = [(sepal length, sepal width, petal length, petal Width)]iris = Datasets.load_ Iris () x_vals = Np.array ([x[3] for x in iris.data]) y_vals = Np.array ([y[0] for y in Iris.data]) # # # # # # # parameters#### De Clare Batch sizebatch_size = 50# Initialize placeholdersx_data = Tf.placeholder (Shape=[none, 1], Dtype=tf.float32) y_ target = Tf.placeholder (Shape=[none, 1], Dtype=tf.float32) # make results reproducibleseed = 13np.random.seed (Seed)Tf.set_random_seed (SEED) # Create variables for linear Regressiona = TF. Variable (Tf.random_normal (shape=[1,1])) b = tf. Variable (Tf.random_normal (shape=[1,1)) # Declare Model operationsmodel_output = Tf.add (Tf.matmul (X_data, A), B) # # # # # # # # # # Loss functions#### Select appropriate Loss function based on regression typeif Regression_type = = ' LASSO ': # Declare Lass  o Loss function # Adds loss functions, which are modified continuous step functions, and the cutoff point of the lasso regression is set to 0.9. # This means that the limiting slope coefficient does not exceed 0.9 # Lasso Loss = L2_loss + heavyside_step, # Where heavyside_step ~ 0 if A < constant, otherwise ~ 9 9 Lasso_param = tf.constant (0.9) Heavyside_step = TF.TRUEP (1., Tf.add (1., Tf.exp (Tf.multiply ( -50., Tf.subtract (A, Lasso_  param)))) Regularization_param = Tf.multiply (Heavyside_step, 99.) Loss = Tf.add (Tf.reduce_mean (Tf.square (Y_target-model_output)), regularization_param) elif Regression_type = = ' Ridge '  : # Declare The Ridge loss function # Ridge loss = L2_loss + L2 norm of slope ridge_param = tf.constant (1.) Ridge_loss = Tf.reduce_mean (Tf.square (A)) LOSS = Tf.expand_dims (Tf.add (Tf.reduce_mean (Tf.square (Y_target-model_output)), tf.multiply (Ridge_param, Ridge_loss) ), 0) else:print (' Invalid regression_type parameter value ', File=sys.stderr) # # # # optimizer#### Declare optimizermy_opt = Tf.train.GradientDescentOptimizer (0.001) Train_step = my_opt.minimize (loss) # # # # Run regression#### Initialize Variablesinit = Tf.global_variables_initializer () sess.run (init) # Training Looploss_vec = []for i in range: rand_in Dex = Np.random.choice (len (x_vals), size=batch_size) rand_x = Np.transpose ([X_vals[rand_index]]) rand_y = Np.transpose (  [Y_vals[rand_index]]) Sess.run (Train_step, feed_dict={x_data:rand_x, y_target:rand_y}) Temp_loss = Sess.run (loss, feed_dict={x_data:rand_x , y_target:rand_y}) Loss_vec.append (Temp_loss[0]) if (i+1)%300==0:print (' Step # ' + str (i+1) + ' A = ' + str (sess.ru  N (A)) + ' B = ' + str (sess.run (b))) print (' Loss = ' + str (temp_loss)) print (' \ n ') # # # Extract regression results#### Get the optimal CoefficiEnts[slope] = Sess.run (A) [y_intercept] = Sess.run (b) # Get best Fit Linebest_fit = []for i in X_vals:best_fit.append (slope *i+y_intercept) # # # # # # Plot results#### plot regression line against data pointsplt.plot (x_vals, y_vals, ' o ', label= ' data Poi NTS ') Plt.plot (x_vals, Best_fit, ' R ', label= ' best fit line ', linewidth=3) plt.legend (loc= ' upper left ') plt.title (' Sepal Length vs Pedal width ') plt.xlabel (' pedal width ') plt.ylabel (' Sepal Length ') plt.show () # Plot loss over Timeplt.plot (loss_ VEC, ' K ') plt.title (Regression_type + ' Loss per Generation ') Plt.xlabel (' Generation ') plt.ylabel (' Loss ') plt.show ()

Output Result:

Step #300 A = [[0.77170753]] b = [[1.82499862]]
Loss = [[10.26473045]]
Step #600 A = [[0.75908542]] b = [[3.2220633]]
Loss = [[3.06292033]]
Step #900 A = [[0.74843585]] b = [[3.9975822]]
Loss = [[1.23220456]]
Step #1200 A = [[0.73752165]] b = [[4.42974091]]
Loss = [[0.57872057]]
Step #1500 A = [[0.72942668]] b = [[4.67253113]]
Loss = [[0.40874988]]

Based on the standard linear regression estimation, a continuous step function is added to realize the lasso regression algorithm. Because of the gradient of the step function, we need to pay attention to the step size, because too large a step will cause eventual non-convergence.