MapReduce implements QQ friend referral

We all know QQ user volume billion, each user has a lot of friends, therefore, the amount of data is very large, how to achieve QQ friends recommend it?
Here's an example:
A has QQ friend B
B has QQ friend C
Then A,c may be a friend.
When a is logged in, it will recommend C to a, and when C is logged in, it will recommend A to C.

Demo

input Data

map阶段key：主value：从key：从value：主将一条记录分别作为key，value进行输出。tom-->jasonjason-->tomtom-->lgdlgd-->tomreduce阶段将同一个key的values值进行两两组合。

Package friendsrecommended;Import Java. IO. IOException;Import Java. NET. URI;Import Java. NET. URISyntaxException;Import Java. Util. HashSet;Import Java. Util. Iterator;Import Java. Util. Set;import org. Apache. Hadoop. conf. Configuration;import org. Apache. Hadoop. FS. FileSystem;import org. Apache. Hadoop. FS. Path;import org. Apache. Hadoop. IO. Intwritable;import org. Apache. Hadoop. IO. Longwritable;import org. Apache. Hadoop. IO. Text;import org. Apache. Hadoop. MapReduce. Job;import org. Apache. Hadoop. MapReduce. Mapper;import org. Apache. Hadoop. MapReduce. Reducer;import org. Apache. Hadoop. MapReduce. Lib. Input. Fileinputformat;import org. Apache. Hadoop. MapReduce. Lib. Output. Fileoutputformat;public class Findfriends {private final static String Input_path ="Hdfs://liguodong:8020/liguodong";Private final static String Output_path ="hdfs://liguodong:8020/liguodong/qqfriendrecommended";public static void Main (string[] args) throws IOException, URISyntaxException, ClassNotFoundException, INTERRUPTEDEXCEP tion {Configuration conf = new configuration ();Final FileSystem FileSystem = FileSystem. Get(New URI (Input_path), conf);if (FileSystem. Exists(New Path (Output_path))) {FileSystem. Delete(New Path (Output_path), true);} Job Job = Job. getinstance(Conf,"QQ friend recommended"); Job. Setjarbyclass(Findfriends. Class);Fileinputformat. Addinputpath(Job, New Path (Input_path)); Job. Setmapperclass(Mymapper. Class);Job. Setreducerclass(Myreudcer. Class);Job. Setoutputkeyclass(Text. Class);Job. Setoutputvalueclass(Text. Class);Fileoutputformat. Setoutputpath(Job, New Path (Output_path));Submit Job System. Exit(Job. WaitForCompletion(true)?0:1);} public static class Mymapper extends mapper<longwritable,text,text,text>{@Override protected voi D map (longwritable K1, Text v1, Context context) throws IOException, interruptedexception {stri Ng line = V1. toString();string[] ss = Line. Split("\\s+");Context. Write(New Text (ss[0]), New Text (ss[1]));Context. Write(New Text (ss[1]), New Text (ss[0]));}} public static class Myreudcer extends Reducer<text, text, text, text>{@Override protected V OID reduce (Text K2, iterable<text> v2s,context Context) throws IOException, Interruptedexception {Set<String>Set= new Hashset<string> ();for (Text v2:v2s) {Set. Add(v2. toString());} if (Set. Size() >1) {for (Iterator i =Set. Iterator(); I.hasnext ();) {String qqname = (string) i. Next();for (Iterator j =Set. Iterator(); J.hasnext ();) {String otherqqname = (string) j. Next();if (!qqname. Equals(Otherqqname)) {Context. Write(new text (qqname), new text (Otherqqname));}                    }                }            }        }    }}

Output Result:

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

MapReduce to implement QQ friend referral