KNN algorithm Hadoop implementation and kaggle digit recognition data test

Software version:
hadoop2.6,myeclipse10.0, Maven 3.3.2

Source: Https://github.com/fansy1990/knn.

1. KNN algorithm Thinking if a sample is in the K most similar in the feature space (that is, the nearest neighbor in the feature space) Most of the samples belong to a category, then the sample belongs to that category. In the KNN algorithm, the selected neighbors are the objects that have been correctly categorized. This method determines the category to which the sample is to be divided based on the category of the nearest one or several samples in the categorical decision-making. --from "Neighboring algorithms", Baidu Encyclopedia
2. KNN algorithm Mr Implementation: the implementation of the KNN algorithm Hadoop, the main thing is to design mapper and reducer, so that the data flow can be applied to the Hadoop Mr Framework, that is, <K1,V1>--> <K2,V2>, &LT;K2, List<v2>>--> <K3,V3>;
2.1.Mapper Design

Mapper has three functions setup (), map (), Cleanup (), where the setup () function runs at the very beginning, the cleanup function runs at the last time, the map function is processed once per record, and the input file is set to the training data set. In the following format:

label,x1,x2,..., xn
The first column is the sample category, the other column is the sample attribute;
1) Read the test data in the Setup () function, the test data format is as follows:
x1,x2,..., xn
Simultaneously initializes an array list<type,distance&gt with the same size as the test data, which stores the neighbor data for each test sample. The nearest neighbor data contains K sample categories and corresponding distances;

2) Next, each training sample is read in the map function, and the adjacent data of all test samples is updated for the training sample, that is, LIST<TYPE,LABEL> the first initialization of the type=[-1,-1,-1,...,-1],label=[ Maxdouble,maxdouble,..., maxdouble].

Update specific practices:
Current training sample Data Tr_i and category Tr_type, current test sample te_i, and corresponding te_i_<type[],distance[]>
A. Calculate the distance distance_tr_te of Tr_i and te_i;
B. Traversing the te_i_<type[],distance[]> distance[]
Find distance[] The largest max_distance and subscript index_max_distance;
If max_distance> Distance_tr_te Then
Replace
Distance[index_max_distance]=distance_tr_te;
Type[index_max_distance]=tr_type;
End
End
3) Finally, in the cleanup () function, the output of each record in the list<type,lable>, where the default test data is a file, so here you can directly use the list of the subscript to represent each data, that is, the final mapper output is < ID,&LT;TYPE[],DISTANCE[]&GT;&GT; each ID represents a test data,<type[],distance[]> represents the adjacent category of the current test data and the corresponding distance;
2.2.Reducer Design

Reducer only design the reduce function, the reduce function needs to do is for the same test sample ID, statistical summary of the adjacent categories and corresponding distance data, and need to re-organize the data, the minimum distance of the K category values, and then at the same time, the number of K categories, That is, the final test sample ID of the category value, output can be;

2.3. Combiner Design

Combiner, here can add a combiner operation, combiner operation is the map side of the reduce operation, you can first summarize the data once, and then sent to the reducer side, reduce network IO. Combiner need input/output format has been, that is, input is <id,<type[],distance[]>>, and the output needs to be such a format. Here you can reducer the summary of the code directly to copy it, at the same time need to note that there is no need to require the majority.

3. test data using digit recognition data from Kaggle; Data download: Https://www.kaggle.com/c/digit-recognizer/data.
In the test algorithm, the first interception of less data to test, test data 18 samples, training samples for the first time more than 50 samples, the second 8.9M (about 5k samples) samples;

The results are compared:

The above test data is obtained using the Training data interception category, the left side of the above two graphs is the result of the prediction, the right is the actual category. As you can see from Figure 1 (less training data), the prediction accuracy is very low, but after the increase in the amount of training data (Figure 2 of 8.9M) there is a good prediction (accuracy rate is 100%);

Using the above algorithm to test the Kaggle data (training set 70M, test set 40M), the data can be downloaded from the above URL;

Hadoop Cluster Scenario:

Local Cluster Situation:

Node101 (master node, namenode,datanode,resourcemanager,nodemanager,secondarynamenode,3.7g,2 Core)

NODE102 (Data/Compute node, datanode,nodemanager,1g,1 Core)

The above two machines are virtual machines, the machine is the Windows7 system 4 core 8G of memory;

Take:

The obtained forecast results, submitted to the Kaggle official website, can be predicted by the accuracy rate, as follows:

The accuracy rate is about 92%, at the same time you can see the ranking of 557, this is very later.

4. Summary 1. KNN algorithm due to the need to calculate all the training data set and test data set distance, so the algorithm is time-consuming (here you can see a total of 110M of data takes about 11 hours), there is no tuning the algorithm, you can refer to some foreign papers to modify this, such as "efficient Parallel KNN Joins for Large Data in MapReduce; 2. Different data for the problem is not the same, here Kaggle digital recognition, usually use random forest to do;

Share, grow, be happy

Down-to-earth, focus

Reprint Please specify blog address: http://blog.csdn.net/fansy1990

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

KNN algorithm Hadoop implementation and kaggle digit recognition data test