spark machine learning example python

Citycluster[label[i]].append (Cityname[i]) #将每个簇的城市输出For I in range (len (citycluster)):Print ("expenses:%.2f"% expenses[i]) #将每个簇的平均花费输出Print (Citycluster[i])Click to run, you can come out results.Where the N_clusters class, the consumption level of similar cities gathered in a classExpense: The numerical plus of the central point of the cluster, that is, the average consumption levelImplementation process:1, establish the project, import Sklearn related packageImport NumPy as NPFrom Sklearn.cl

)]=1 else:print "The word:%s is not in my vocabulary!" %word return returnvecdef TRAINNBC (trainsamples,traincategory): Numtrainsamp=len (Trainsamples) NumWords=len (train Samples[0]) pabusive=sum (traincategory)/float (numtrainsamp) #y =1 or 0 feature Count P0num=np.ones (numwords) P1NUM=NP.O NES (numwords) #y =1 or 0 category count P0numtotal=numwords p1numtotal=numwords for I in Range (Numtrainsamp): if Traincategory[i]==1:p0num+=trainsamples[i] P0numtotal+=sum (Trainsamples[i]) E

=linearr.predict (X_train) #基于训练集得到的线性y值Plt.figure ()Plt.scatter (x_train,y_train,color= ' green ') #原始训练集数据散点图Plt.plot (x_train,y_train_pred,color= ' black ', linewidth=4) #线性回归的拟合线Plt.title (' Train ') #标题Plt.show ()Y_test_pred=linearr.predict (X_test)Plt.scatter (x_test,y_test,color= ' green ') #绘制测试集数据散点图Plt.plot (x_test,y_test_pred,color= ' black ', linewidth=4) #基于线性回归的预测线Plt.title (' Test ')Plt.show ()Print (' mse= ', Sm.mean_squared_error (y_test,y_test_pred)) #MSE值Print (' r2= ', Sm.r2_

Misc.forsale 0.91 0.70 0.79 257 the Rec.autos 0.89 0.89 0.89 238 - Rec.motorcycles 0.98 0.92 0.95 276 - Rec.sport.baseball 0.98 0.91 0.95 251 the Rec.sport.hockey 0.93 0.99 0.96 233 the Sci.crypt 0.86 0.98 0.91 238 the sci.electronics 0.85 0.88 0.86 249 the sci.med 0.92 0.94 0.93 245 - sci.space 0.89 0.96 0.92 221 the Soc.religion.christian 0.78 0.96 0.86 232 the talk.politics.guns 0.88 0.96 0.92 251 the talk.politics.mideast 0.90 0.98 0.94 23194 Talk.politics.misc 0.79 0.89 0.84 188 the Talk.r

:", X) - Print("Y:", Y) - innumiterations=100000 -alpha=0.0005 toTheta=np.ones (x.shape[1]) +Theta=graientdescent (x,y,theta,alpha,x.shape[0],numiterations) - Print(Theta)Operation Result:...... Too many output data to intercept only the next more than 10 linesIteration 99988/cost:3.930135Iteration 99989/cost:3.930135Iteration 99990/cost:3.930135Iteration 99991/cost:3.930135Iteration 99992/cost:3.930135Iteration 99993/cost:3.930135Iteration 99994/cost:3.930135Iteration 99995/cost:3.930135Iterat

This article mainly introduces the knowledge of Perceptron, uses the theory + code practice Way, and carries out the learning of perceptual device. This paper first introduces the Perceptron model, then introduces the Perceptron learning rules (Perceptron learning algorithm), finally through the Python code to achieve

The task of supervised learning in machine learning focuses on predicting the target/marker of an unknown sample based on existing empirical knowledge.According to the different types of target predictor variables, we divide the task of supervised learning into two categories: Classification

Verification code identification of Python classification model download Verification Code image processing two-valued original diagram declaration image class cutting picture annotation picture generating training set matrix CSV file validating training set training model identifying and calculating verification code endnotes Verification Code recognition of Python classification model Download Verificatio

(Ss_y.inverse_transform (y_test), Ss_y.inverse_transform (lr_y_predict)) $ Print("the mean square error of the linear is:", Lr_mse) -Lr_mae =Mean_absolute_error (Ss_y.inverse_transform (y_test), Ss_y.inverse_transform (lr_y_predict)) - Print("the average absolute error of the linear is:", Lr_mae) - A #evaluation of the SGD model +Sgdr_score =Sgdr.score (x_test, y_test) the Print("the default evaluation value for SGD is:", Sgdr_score) -sgdr_r_squared =R2_score (y_test, sgdr_y_predict) $ Print("

regression tree is:", Dtr.score (X_test, y_test)) - Print("the r_squared values for the flat regression tree are:", R2_score (Y_test, dtr_y_predict)) - Print("the mean square error of the regression tree is:", Mean_squared_error (Ss_y.inverse_transform (y_test), - Ss_y.inverse_transform (dtr_y_predict))) A Print("the average absolute error of the regression tree is:", Mean_absolute_error (Ss_y.inverse_transform (y_test), + Ss_y.inverse_transform (dtr_y_predict))) the - " " $ the default evalua

.score (X_train_poly2, Y_train))#0.9816421639597427Two-time linear regression model fitted curves:The fitting degree is better than 1 linear fitting.The following 4 linear regression models are performed:1 #four-time linear regression model fitting2Poly4 = Polynomialfeatures (degree=4)#4-time polynomial feature generator3X_train_poly4 =poly4.fit_transform (X_train)4 #Building Model Predictions5Regressor_poly4 =linearregression ()6 Regressor_poly4.fit (X_train_poly4, Y_train)7 #draw a graph of 2

) p (CI)/P (W)Calculate a specific document W belongs to C0 (insulting document) or C1 (non-insulting document), statistics the probability of each word in this document in two different categories, quantified by the Bayesian formula, that is, each word in a particular document in the p0v or p1v to find the corresponding word probability, Multiply these probabilities, i.e. P (W0|CI) p (W1|CI) p (w2|ci). P (WN|CI), multiplied by P (CI), the final result is two probability values, the probability

AboutDTC =Decisiontreeclassifier () $ #Training - Dtc.fit (X_train, Y_train) - #Predicting saved results -Y_predict =dtc.predict (x_test) A + " " the 4 Model Evaluation - " " $ Print("accuracy:", Dtc.score (X_test, y_test)) the Print("Other indicators: \ n", Classification_report (Y_predict, Y_test, target_names=['died','survived'])) the " " the accuracy: 0.7811550151975684 the Other indicators: - Precision recall F1-score support in the died 0.91 0.78 0.84 236 the survived 0.58 0.80 0.67 Abo

", Classification_report (Gbc_y_predict, Y_test, target_names=['died','survived']))103 104 " " the Single decision tree accuracy: 0.7811550151975684106 Other indicators:107 Precision recall F1-score support108 109 died 0.91 0.78 0.84 236 the survived 0.58 0.80 0.67111 the avg/total 0.81 0.78 0.79 329113 the Random forest accuracy: 0.78419452887538 the Other indicators: the Precision recall F1-score support117 118 died 0.91 0.78 0.84 237119 survived 0.58 0.80 0.68 - 121 avg/total 0.82 0.78 0.79

-za-z]"," ", Sent.lower (). Strip ()). Split () in sentences.append (temp) - to returnsentences + - #The sentences in the long news are stripped out for training . thesentences = [] * forIinchx: $Sentence_list =news_to_sentences (i)Panax NotoginsengSentences + =sentence_list - the + #Configure the dimension of the word vector ANum_features = 300 the #the frequency of the words that are to be considered +Min_word_count = 20 - #number of CPU cores used in parallel computing $Num_workers =

#岭回归主要是弥补在数据中出现异常值时, improve the stability of linear model, that is, robustness robustImport Pandas as PDImport NumPy as NPImport Matplotlib.pyplot as PltFrom Sklearn import Linear_modelImport Sklearn.metrics as SM#直接拿最小二乘法数据Ridgerg=linear_model. Ridge (alpha=0.5,fit_intercept=true,max_iter=10000) #alpha nearer to 0, the more the ridge regression approached the linear regression.Ridgerg.fit (X_train,y_train) #训练模型Y_train_pred=ridgerg.predict (X_train) #模型y值Y_test_pred=ridgerg.predict (x_test) #模

Python Kmeans clustering is relatively simple, first requires the import NumPy, from the Sklearn.cluster import Kmeans module:Import NumPy as NP from Import KmeansThen read the TXT file, get the corresponding data and convert it to numpy array:X == open ('rktj4.txt') for in f: = Re.compile ('\s+') x.append ([Float (Regex.Split (v) [3]), float ( Regex.Split (v) [6= Np.array (X)Set the number of classes and cluster:N_clusters = 5= Kmeans (n_clust

training sample Ax = titanic[["Pclass"," Age","Sex"]] aty = titanic["survived"] - #The average complement of the acquired age space -x[" Age"].fillna (x[" Age"].mean (), inplace=True) - - #split training data and test data -X_train, X_test, y_train, y_test =train_test_split (x, in y, -test_size=0.25, toRandom_state=33) + #extracting dictionary features for vectorization -VEC =Dictvectorizer () theX_train = Vec.fit_transform (X_train.to_dict (orient="Record")) *X_test = Vec.transform (X_test.to

= Polynomialfeatures (degree=4)#4-time polynomial feature generator -X_train_poly4 =poly4.fit_transform (X_train) Wu #Building Model Predictions -Regressor_poly4 =linearregression () About Regressor_poly4.fit (X_train_poly4, Y_train) $X_test_poly4 =poly4.transform (x_test) - Print("four-time linear model prediction score:", Regressor_poly4.score (X_test_poly4, Y_test))#0.8095880795746723 - - #learning and predicting using L1 norm regularization line

next cycle , select () no longer indicates that the socket is ready to send data For S in writable:TryNext_msg=message_queues[s].get_nowait ()Except Queue.empty:Print >>sys.stderr, ', S.getpeername (), ' Queue empty 'Outputs.remove (s)ElsePrint >>sys.stderr, ' sending%s to%s '% (Next_msg,s.getpeername ())S.send (NEXT_MSG)# Finally, if a socket has an error, closeFor S in exceptional:Print >>sys.stderr, ' exception condition on ', S.getpeername ()Inputs.remove (s)If s in outputs:Outputs.remove (