pyspark linear regression

I. Summary Linear Regression Algorithms are a type of supervised learning algorithm used for Numerical Prediction of continuous functions. After preliminary modeling, the process determines the model parameters through the training set to obtain the final prediction function. Then, the predicted value can be obtained by inputting the independent variable.Ii. Basic Process 1. Preliminary modeling. Determine

(train_step,feed_dict={x_data:rand_x,y_data:rand_y}) Temp_loss=sess.run (loss,feed_dict={x_data:rand_x,y_data:rand_y})#Add a recordloss_rec.append (Temp_loss)#Print if(i+1)%25==0:Print('Step:%d a=%s b=%s'%(I,str (Sess.run (A)), str (Sess.run (b) )))Print('loss:%s'%str (temp_loss))#decimation Factor[slope]=Sess.run (A)Print(slope) [Intercept]=Sess.run (b) Best_fit=[] forIinchX_vals:best_fit.append (Slope*i+intercept)#x_vals shape (none,1)Plt.plot (X_vals,y_vals,'o', label='Data') Plt.plot (X_

voidPrint (javarddparseddata, Generalizedlinearmodel model) {Javapairrdd { DoublePrediction = Model.predict (Point.features ());//predicting training data with models return NewTuple2(Point.label (), prediction); }); Double MSE= Valuesandpreds.maptodouble ((tuple2//calculates the mean of the squared value of the difference between the predicted value and the actual valueSystem.out.println (Model.getclass (). GetName () + "training Mean squared Error =" +MSE);} Run result Linea

The key to multivariate linear regression is the self-variable filter. Back method is generally used. # Full variable regression of industrial power consumption lm.fullind Summary can print the P-value of each argument ("Pr (>|t|)") in the R language ） call:lm (formula = data[, ten] ~ data[, 3] + data[, 5] + data[, 6] + data[, 7] + data[, + + data[ , []]) resi

+1.999936002667351 -1.9999628815470636 +1.9999784712972968 A1.999987513352432 at1.9999927577444105 -1.999995799491758 -1.9999975637052196 -1.9999985869490273 -1.9999991804304358 -1.9999995246496527 in1.9999997242967986 -1.9999998400921433 to1.9999999072534431 +1.999999946206997 -1.9999999688000583 the1.9999999819040337 *1.9999999895043397 $1.999999993912517Panax Notoginseng1.9999999964692599 -1.9999999979521708 the1.999999998812259 +1.9999999993111102 A1.9999999996004438 the1.9999999997682574 +

price of a house is whether the area is more important or the room orientation is more important. We make x0 = 1, we can use vectors to represent In the above formula, once Theta is determined, then our straight line is determined, and we are able to forecast the house price. So the job we're going to do is to determine theta. The value of θ can have countless, how should we choose θ? 3. Model Establishment-least squares O

Supervised Learning Learn a function H: X → y H is called a hypothesis. 1. Linear Regression In this example, X is a two-dimensional vector, x1 represents living area, and x2 represents bedrooms. Functions/hypotheses H Set X0 = 1. Now, given a training set, how do we pick, or learn, the parameters θ? Now it is used to evaluate the θ parameter. One reasonable method seems to be to make h (x)

Mathematical model, if need to bank loans, according to the loan amount calculation, there are age, wages, assets and other factors, this translates into mathematics is:Now we are to fit a plane to make a diagram, but this fitting theoretical value and the results of the real difference, now we through the statistical error calculation method, we believe that each error is independent of the same distribution, and obey the mean value of 0 variance is the Gauss distribution of the square. Indepen

-alpha (alpha_i). *grad; End Plot (0: the, Jtheta (1: -),Char(Plotstyle (alpha_i)),'linewidth',2)%It is important to use the CHAR function to convert the packet () to the cell after the package () index.%so you can use the Char function or the {} index, so you don't have to convert. %a learning rate corresponding to the image drawn out later to draw the next learning rate corresponding to the image. onif(1= = Alpha (alpha_i))%The result of the experiment is that the alpha 1 o'clock is the best,

sample in the training set. Therefore, it is called batch gradient descent (batch gradient descent ), when the number of samples is small, it is acceptable, but when the number of samples is very large, this update will make the algorithm very inefficient. You can consider the following update method: In this way, each update step requires only one training sample to greatly accelerate the parameter update speed. Therefore, it is called stochastic gradient descent ), however, this update metho

); theta = Theta-alpha (alpha_i) .*grad; end plot (0:49, Jtheta ( 1:50), char (Plotstyle (alpha_i)), ' LineWidth ', 2)% be sure to convert by Char function hold on if (1 = = Alpha (alpha_i))% The experiment found that Alpha was1 o'clock the effect is best, then the theta value after the iteration is the desired value theta_grad_descent = theta endendlegend (' 0.01 ', ' 0.03 ', ' 0.1 ', ' 0.3 ', ' 1 ', ' 1.3 '); Xlabel (' Number of iterations ') ylabel (' cost function ')% below is the pred

}\left ({{x^{(i)}}} \right)-{y^i}} \right)}^2}} \]Target is \[\mathop {\min imize}\limits_{{\theta _0},{\theta _1}} j\left ({{\theta _0},{\theta _1}} \right) \]This cost function is also called the squared error cost function (squared error function)Summarize:Hypothesis: \[{h_\theta}\left (x \right) = {\theta _0} + \theta {x}\]Parameters: (θ0,θ1)Cost Functions: \[j\left ({{\theta _0},{\theta _1}} \right) = \frac{1}{{2m}}\sum\limits_{i = 1}^m {{\left} ({{H_\theta}\l EFT ({{x^{(i)}}} \right)-{y^i}

, 1)) A #Pre-valuation atPreY = w * x +b - - #loss value: The mean variance between the pre-estimate and the actual value -Loss = Tf.reduce_mean (Tf.square (PreY-y)) - #Optimizer: Gradient Descent method -Optimizer =Tf.train.GradientDescentOptimizer (learnrate) in #Training: Minimizing loss function -Trainer =optimizer.minimize (loss) to + With TF. Session () as Sess: - Sess.run (Tf.global_variables_initializer ()) the #print initial random coefficients * Print(

, meaning you have only 10 data, but there are 100 features, obviously, the data is not enough to cover all the features.You can delete some features (keep only data-related features) or use regularization.Exercises1.Don't know how to use both methods at the same time, are these two methods sequential related?Use dividing by the rangeRange = Max-min = 8836-4761 = 4075Vector/range after change to1.94381.27212.16831.1683For the above use mean normalizationAVG = 1.6382Range = 2.1683-1.1683 = 1X2 (4