MapReduce Programming Example (1)-Statistical frequency of the program

Today began to MapReduce design patterns this book on the MapReduce example, I think this book on learning MapReduce programming very well, the book finished, basically can meet the mapreduce problems can also be dealt with. Let's start with the first piece. This procedure is to count a word frequency in the comment.xml. Go directly to the code.

Parses the XML file and stores it in the map.
package mrdp.utils;

Import Java.util.HashMap;
Import Java.util.Map;

public class Mrdputils {public
	
	static final string[] Redis_instances = {"P0", "P1", "P2", "P3",
			"P4", "P6"};
  //This helper function parses the "StackOverflow into" a Map for us.
	public static map<string, string> transformxmltomap (String xml) {
		map<string, string> Map = new HASHMAP&L T String, string> ();
		try {
			string[] tokens = Xml.trim (). substring (5, Xml.trim (). Length ()-3)
					. Split ("\");

			for (int i = 0; i < tokens.length-1 i = 2) {
				String key = Tokens[i].trim ();
				String val = tokens[i + 1];

				Map.put (key.substring (0, Key.length ()-1), Val);
			}
		catch (Stringindexoutofboundsexception e) {
			System.err.println (XML);
		}

		return map;
	}

Main program Package mrdp.ch1;
Import java.io.IOException;
Import Java.util.StringTokenizer;

Import Java.util.Map;

Import Mrdp.utils.MRDPUtils;
Import org.apache.hadoop.conf.Configuration;
Import Org.apache.hadoop.fs.Path;
Import org.apache.hadoop.io.IntWritable;
Import Org.apache.hadoop.io.Text;
Import Org.apache.hadoop.mapreduce.Job;
Import Org.apache.hadoop.mapreduce.Mapper;
Import Org.apache.hadoop.mapreduce.Reducer;
Import Org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
Import Org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
Import Org.apache.hadoop.util.GenericOptionsParser;

Import Org.apache.commons.lang.StringEscapeUtils; public class Commentwordcount {public static class Sowordcountmapper extends Mapper<object, text, text, Intwritab
		le> {Private final static intwritable one = new intwritable (1);

		Private Text Word = new text (); public void Map (Object key, Text value, Context context) throws IOException, interruptedexception {//Parse the Input string into a nice map map<string, string> parsed = Mrdputils.transformxmltomap (value. toString ());

			Grab the "Text" field, since that are what we are counting over String txt = parsed.get ("text");
			. Get would return NULL if the key was not there if (txt = null) {//skip this record return;
			}//Unescape the HTML because the so data is escaped.

			txt = stringescapeutils.unescapehtml (txt.tolowercase ()); Remove some annoying punctuation txt = txt.replaceall ("'", ""); Remove single quotes (e.g., can ' t) txt = txt.replaceall ("[^a-za-z]", ""); Replace the rest with a //Tokenize The string, then send the tokens away Stringtokeni
			Zer ITR = new StringTokenizer (TXT);
				while (Itr.hasmoretokens ()) {Word.set (Itr.nexttoken ());
			Context.write (Word, one); }} public static class Intsumreducer extends Reducer<text, intwritable, Text, intwritable> {private in TwritablE result = new intwritable (); public void reduce (Text key, iterable<intwritable> values, context context) throws IOException, Interruptedexcep
			tion {int sum = 0;
			for (intwritable val:values) {sum + = Val.get ();
			} result.set (sum);

		Context.write (key, result);
		} public static void Main (string[] args) throws Exception {Configuration conf = new Configuration ();
		string[] Otherargs = new Genericoptionsparser (conf, args). Getremainingargs ();
			if (otherargs.length!= 2) {System.err.println ("Usage:commentwordcount <in> <out>");
		System.exit (2);
		@SuppressWarnings ("deprecation") job Job = new Job (conf, "StackOverflow Comment Word Count");
		Job.setjarbyclass (Commentwordcount.class);
		Job.setmapperclass (Sowordcountmapper.class);
		Job.setcombinerclass (Intsumreducer.class);
		Job.setreducerclass (Intsumreducer.class);
		Job.setoutputkeyclass (Text.class);
		Job.setoutputvalueclass (Intwritable.class); Fileinputformat.addInputPath (Job, New Path (otherargs[0));
		Fileoutputformat.setoutputpath (Job, New Path (otherargs[1));
	System.exit (Job.waitforcompletion (true)? 0:1); }
}

This program does not have to explain too much, as long as Java and a little wordcount people know

My MapReduce program is debugged on Eclipse, and you need to fill in the parameters when you run this program, which is to fill in your own HDFs address in run configuration, as my parameters are:

Hdfs://localhost:8010/user/jpan/comments.xml  HDFS://LOCALHOST:8010/USER/JPAN/OUTPUT1

The link to the test data is in Http://pan.baidu.com/s/1c0xP6Dy, there are comment.xml files, and some other files we will use later.

MapReduce Design Patterns link to this book Http://pan.baidu.com/s/1jGt96Hg