Pyspark machine Learning (1)--random forest

Source: Internet
Author: User
Tags pyspark

This article mainly implements the stochastic forest algorithm in the Pyspark environment:

%pyspark from Pyspark.ml.linalg import Vectors to pyspark.ml.feature import stringindexer from Pyspark.ml.classificati On the import randomforestclassifier from pyspark.sql import Row #任务目标: Solve two classification problems through random forests and evaluate #1 of classification effects. Read data = Spark.sql ("" "Sele CT * from DataTable "" "#2. Construct Training DataSet = Data.na.fill (' 0 '). Rdd.map (list) (Traindata, testData) = Dataset.randomsplit ([0.7, 0.3]) #print (Traindata.take (1)) Trainingset = Traindata.map (lambda x:row (label=x[-1), Features=vectors.dense (x [:-1])).
TODF () Train_num = Trainingset.count () print ("Training sample: {}". Format (train_num)) print (Trainingset.show ()) #3. Training with random forests Stringindexer = Stringindexer (inputcol= "label", outputcol= "indexed") Si_model = Stringindexer.fit (trainingset) tf = Si_ Model.transform (Trainingset) tf.show () RF = Randomforestclassifier (numtrees=10, maxdepth=8, labelCol= "indexed", seed= Rfcmodel = Rf.fit (TF) #输出模型特征重要性, subtree weight print (model feature: {},. Format (rfcmodel.featureimportances)) Print (model feature: {}). Format (rfcmodel.numfeatures)) #4. Test TeStset = Testdata.map (lambda x:row (label=x[-1), Features=vectors.dense (x[:-1))). TODF () print (test sample number: {}). Format (

Testset.count ()) print (Testset.show ()) Si_model = Stringindexer.fit (testset) TEST_TF = Si_model.transform (TestSet) result = Rfcmodel.transform (TEST_TF) result.show () #5. Classification effect Evaluation Total_amount=result.count () Correct_amount = Result.filter (result.indexed==result.prediction). Count () Precision_rate = Correct_amount/total_amount print (" Prediction accuracy is: {} ". Format (precision_rate)) Positive_precision_amount = Result.filter (result.indexed = = 0). Filter ( Result.prediction = = 0). Count () Negative_precision_amount = result.filter (result.indexed = = 1). Filter ( Result.prediction = = 1). Count () Positive_false_amount = result.filter (result.indexed = = 0). filter (Result.prediction = = 1. Count () Negative_false_amount = result.filter (result.indexed = = 1). filter (Result.prediction = = 0). Count () print (" Positive sample Forecast exact number: {}, Negative sample forecast exact quantity: {} ". Format (positive_precision_amount,negative_precision_amount)) Positive_amount = ResUlt.filter (result.indexed = = 0). Count () Negative_amount = Result.filter (result.indexed = 1). Count () print (number of positive samples: {}, Negative sample number: {} ". Format (Positive_amount,negative_amount)) print (" Number of positive sample predictions error: {}, negative sample error exact quantity: {} ". Format (Positive_false_ Amount,negative_false_amount)) Recall_rate1 = Positive_precision_amount/positive_amount Recall_rate2 = negative_ Precision_amount/negative_amount print ("Positive sample recall rate is: {}, negative sample recall rate is: {}". Format (RECALL_RATE1,RECALL_RATE2))  

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.