Singletontablejoin of the Hadoop program MapReduce

Source: Internet
Author: User

Requirements: single-table association issues. The relationship between the child and the parent in the document

Model: Child-parent.txt

Xiaoming Daxiong

Daxiong Alice

Daxiong Jack.

Output: Xiaoming Alice

Xiaoming Jack.

Analysis Design:

Mapper Part design:

1, <K1,K1>K1 representative: The number of rows of data location, V1 representative: A row of data.

2, left table: <K2,V2>K2 representative: The parent name, V2 representative: (1,child name), here 1: represents the left table flag.

3, right table: <K3,V3>K3 representative: Child name, V3 representative: (2,parent name), here 2: represents the right table flag.

Reduce part design:

4, <K4,V4>K4 representative: The same KEY,V4 representative:list<string>

5, seek Cartesian product <K5,V5>:K5 representative: Grandchild name, V5 representative: Grandparent name.

Program section:

Singletontablejoinmapper class

 PackageCom.cn.singletonTableJoin;Importjava.io.IOException;ImportJava.util.StringTokenizer;ImportOrg.apache.hadoop.io.Text;ImportOrg.apache.hadoop.mapreduce.Mapper; Public classSingletontablejoinmapperextendsMapper<object, text, text, text>{@Overrideprotected voidMap (Object key, text value, Mapper<object, text, text, text>. Context context)throwsIOException, interruptedexception {String childname=NewString (); String ParentName=NewString (); String Relationtype=NewString (); String[] Values=NewString[2]; inti = 0; StringTokenizer ITR=NewStringTokenizer (value.tostring ());  while(Itr.hasmoreelements ()) {Values[i]=Itr.nexttoken (); I++; }        if(Values[0].compareto ("child")! = 0) {ChildName= Values[0]; ParentName= Values[1]; Relationtype= "1"; Context.write (NewText (ParentName),NewText (relationtype+ "" +childname)); Relationtype= "2"; Context.write (NewText (ChildName),NewText (relationtype+ "" +parentname)); }    } }

Singletontablejoinreduce class:

 PackageCom.cn.singletonTableJoin;Importjava.io.IOException;Importjava.util.ArrayList;ImportJava.util.Iterator;Importjava.util.List;ImportOrg.apache.hadoop.io.Text;ImportOrg.apache.hadoop.mapreduce.Reducer; Public classSingletontablejoinreduceextendsReducer<text, text, text, text>{@Overrideprotected voidReduce (text key, iterable<text> values, Reducer<text, text, text, text>. Context context)throwsIOException, interruptedexception {List<String> grandchild =NewArraylist<string>(); List<String> grandparent =NewArraylist<string>(); Iterator<Text> ITR =Values.iterator ();  while(Itr.hasnext ()) {string[] record= Itr.next (). toString (). Split (""); if(0 = = Record[0].length ()) {Continue; }            if("1". Equals (record[0]) {Grandchild.add (record[1]); }Else if("2". Equals (record[0]) {Grandparent.add (record[1]); }        }        if(0! = grandchild.size () && 0! =grandparent.size ()) {             for(String grandchild:grandchild) { for(String grandparent:grandparent) {context.write (NewText (grandchild),NewText (grandparent)); }            }        }    }}

Singletontablejoin class

 PackageCom.cn.singletonTableJoin;Importorg.apache.hadoop.conf.Configuration;ImportOrg.apache.hadoop.fs.Path;ImportOrg.apache.hadoop.io.Text;ImportOrg.apache.hadoop.mapreduce.Job;ImportOrg.apache.hadoop.mapreduce.lib.input.FileInputFormat;ImportOrg.apache.hadoop.mapreduce.lib.output.FileOutputFormat;ImportOrg.apache.hadoop.util.GenericOptionsParser;/*** Single-Table Association *@authorRoot **/ Public classSingletontablejoin { Public Static voidMain (string[] args)throwsException {Configuration conf=NewConfiguration (); String[] Otherargs=Newgenericoptionsparser (conf, args). Getremainingargs (); if(Otherargs.length! = 2) {System.err.println ("Usage:singletontablejoin"); System.exit (2); }        //Create a jobJob Job =NewJob (conf, "Singletontablejoin"); Job.setjarbyclass (Singletontablejoin.class); //setting the input and output path of a fileFileinputformat.addinputpath (Job,NewPath (otherargs[0])); Fileoutputformat.setoutputpath (Job,NewPath (otherargs[1])); //set up mapper and reduce processing classesJob.setmapperclass (Singletontablejoinmapper.class); Job.setreducerclass (singletontablejoinreduce.class); //Setting the output key-value data typeJob.setoutputkeyclass (Text.class); Job.setoutputvalueclass (Text.class); //submit the job and wait for it to completeSystem.exit (Job.waitforcompletion (true) ? 0:1); }}

Take the summary as a habit.

Singletontablejoin of the Hadoop program MapReduce

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.