MapReduce--Friend Referral

MapReduce implements a friend recommendation:

Zhang San's friends have Harry, Little Red, Zhao Liu; The same Harry, Little Red, Zhao Liu common friend is Zhang San;

In Harry and Little red do not know the premise, can through Zhang San mutual understanding, to Harry recommended friends for Little Red,

To small red recommended friend is Harry, is Harry, small red, Zhao Liu mutually recommended relationship.

According to the analysis is to have the same friends between the characters as a referral relationship, but to exclude the original two are friends of the situation.

Calculate a person's friend referral relationship, the recommended relationship value is 1, and then calculate everyone's friend referral relationship, finally will recommend the relationship value added, to calculate the most worthy of a few friends recommended.

Simply put, two non-friends, the more people who have a common friend, the more the two people deserve to recommend each other.

Data:

Harry John Doe Xiao Ling Xiao    li    Harry    Chili Harry Zhang San    Zhao Liu    Xiao Hong Xiao Ling Harry Zhang San    Zhaolizhang    Xiao Ling    John Doe    Chila Xiao    li Xiao    Ling    Zhang San    John Doe Xiao    Hong    John Doe    Zhao Liu

Implementation code:

1 Importorg.apache.hadoop.conf.Configuration;2 ImportOrg.apache.hadoop.fs.FileSystem;3 ImportOrg.apache.hadoop.fs.Path;4 Importorg.apache.hadoop.io.IntWritable;5 Importorg.apache.hadoop.io.LongWritable;6 ImportOrg.apache.hadoop.io.Text;7 ImportOrg.apache.hadoop.mapreduce.Job;8 ImportOrg.apache.hadoop.mapreduce.Mapper;9 ImportOrg.apache.hadoop.mapreduce.Reducer;Ten ImportOrg.apache.hadoop.mapreduce.lib.input.FileInputFormat; One ImportOrg.apache.hadoop.mapreduce.lib.output.FileOutputFormat; A  - Importjava.io.IOException; -  the /** - * Recommended by friends: - * Calculate the recommended value for two non-friends, that is, the number of two non-friends of common friends -  * + * Created by Edward on 2016/7/12. -  */ +  Public classRunjob { A  at  -      Public Static voidMain (string[] args) { -  -System.setproperty ("Hadoop_user_name", "root"); -  -Configuration conf =NewConfiguration (); in  -Conf.set ("Fs.defaultfs", "hdfs://node1:8020"); to  +         Try { -FileSystem fs =filesystem.get (conf); the  *Job Job =job.getinstance (conf); $Job.setjarbyclass (runjob.class);Panax NotoginsengJob.setmapperclass (Mymapper.class); -Job.setreducerclass (Myreducer.class); the  +             //you need to specify the key and value for map out AJob.setoutputkeyclass (Text.class); theJob.setoutputvalueclass (intwritable.class); +  -Fileinputformat.addinputpath (Job,NewPath ("/test/friend/input")); $  $Path PATH =NewPath ("/test/friend/output"); -             if(fs.exists (path))//If the directory exists, delete the directory -             { theFs.delete (Path,true); -             }Wuyi fileoutputformat.setoutputpath (Job, path); the  -             Booleanb = Job.waitforcompletion (true); Wu             if(b) -             { AboutSystem.out.println ("OK"); $             } -  -}Catch(Exception e) { - e.printstacktrace (); A         } +  the     } -  $      Public Static classMymapperextendsmapper<longwritable, text, text, intwritable> { the  the @Override the         protected voidMap (longwritable key, Text value, context context)throwsIOException, interruptedexception { the  -string[] str = value.tostring (). Split ("\ T"); in  the              for(intI=1; i<str.length; i++) { the                 //A's friend is B AboutContext.write (NewText (Str[0] + ":" + str[i]),NewIntwritable (0)); the                 //B's friend is a theContext.write (NewText (Str[i] + ":" + str[0]),NewIntwritable (0)); the                  for(intj = i + 1; J < Str.length; J + +) { +                     //A's recommended friend is B -Context.write (NewText (Str[i] + ":" + str[j]),NewIntwritable (1)); the                     //B's recommended friend is aBayiContext.write (NewText (Str[j] + ":" + str[i]),NewIntwritable (1)); the                 } the             } -         } -     } the  the      Public Static classMyreducerextendsReducer<text, Intwritable, Text, intwritable> { the  the @Override -         protected voidReduce (Text key, iterable<intwritable> values, context context)throwsIOException, interruptedexception { the  the             intsum = 0; the              for(intwritable i:values)94             { the                 if(I.get () = = 0) {//Two people are already good friends, excluded thesum = 0; the                     //Break ;98                     return; About                 } -Sum + =i.get ();101             }102Context.write (Key,Newintwritable (sum));103         }104     } the}

Calculation Result:

Xiao Lin: Xiao Ling2Xiao Li : Little Red1Xiao Li: Zhang San1Xiao Li: John Doe2Xiao Ling: Xiao li2Xiao Ling: Xiao Hong1Xiao Ling: John Doe3Xiao Hong: Xiao li1Xiao Hong: Xiao Ling1Little Red: Zhang San2Little Red: Harry1Zhang San: Xiao li1Zhang San: Xiao Hong2Zhang San: Harry2John Doe: Xiao li2John Doe: Xiao Ling3Harry: Xiao Hong1Harry: Zhang San2Harry: Zhao Liu3Zhao Liu: Harry3

A simple check of the results, a comparison of the graphs

Xiao Lin: Xiao Ling       2

Xiao Li Haling's common friend number is 2, respectively: Harry, Zhao Liu

MapReduce--Friend Referral