hadoop mapreduce example

based on the recommended configuration of Horntonworks, a common memory allocation scheme for various components on Hadoop cluster is given. The right-most column of the scenario is a 8G VM allocation scheme that reserves 1-2g memory to the operating system, assigns 4G to Yarn/mapreduce, and of course includes hive, and the remaining 2-3g is reserved for hbase when it is necessary to use HBase.

= Mypath.getfilesystem (conf);if (Hdfs.isdirectory (MyPath)){Hdfs.delete (MyPath, true);}@SuppressWarnings ("deprecation")Job Job = new Job (conf, "gender");//Create a new taskJob.setjarbyclass (Gender.class);//Main classJob.setmapperclass (pcmapper.class);//mapperJob.setreducerclass (pcreducer.class);//reducerJob.setpartitionerclass (Myhashpartitioner.class);Job.setpartitionerclass (Pcpartitioner.class);//Set Partitioner classJob.setnumreducetasks (3);//reduce number set to 3Job.setmapoutputke

); } }}Datasort class PackageCom.cn.sort;Importorg.apache.hadoop.conf.Configuration;ImportOrg.apache.hadoop.fs.Path;Importorg.apache.hadoop.io.IntWritable;ImportOrg.apache.hadoop.mapreduce.Job;ImportOrg.apache.hadoop.mapreduce.lib.input.FileInputFormat;ImportOrg.apache.hadoop.mapreduce.lib.output.FileOutputFormat;ImportOrg.apache.hadoop.util.GenericOptionsParser;/*** Data Sorting *@authorRoot **/ Public classDatasort { Public Static voidMain (string[] args)throwsException {Configuration conf=

The 1.Text packet was wrongly guided.The import Com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider.Text;Change to import Org.apache.hadoop.io.Text;.2. The local compilation environment and the Java version in the production environment do not match. It is possible that the JDK does not match or the JRE does not match. It doesn't matter if they match.3.map and reduce are overloaded with mapper and reducer classes respectively. Cannot be a method of your own definitionThe

Hadoop Reading Notes series article:http://blog.csdn.net/caicongyang/article/category/21668551.partition GroupingPartition is the specified grouping algorithm, and the number of tasks to set reduce by setnumreducetasks2. Code Kpiapp.avaPackage Cmd;import Java.io.datainput;import java.io.dataoutput;import java.io.ioexception;import Java.net.URI; Import Org.apache.hadoop.conf.configuration;import Org.apache.hadoop.conf.configured;import Org.apache.hadoo

file size does not exceed 1.1 times times the Shard size, it will be divided into a shard, avoid opening two map, one of the running data is too small, wasting resources.Summary, the Shard process is about, first traverse the target file, filter some non-conforming files, and then add to the list, and then follow the file name to slice the Shard (the size of the previous calculation of the size of the formula, the end of a file may be merged, in fact, often write network programs know), and the

Hadoop Reading Notes series article:http://blog.csdn.net/caicongyang/article/category/21668551. Description:The two columns of data given are sorted first in ascending order of the first column, and the second column in ascending order when the first column is the sameData format:3332312221112. Code Sortapp.javaPackage Sort;import Java.io.datainput;import java.io.dataoutput;import java.io.ioexception;import Java.net.URI; Import Org.apache.hadoop.conf.

Hadoop Reading Notes series article:http://blog.csdn.net/caicongyang/article/category/21668551. Description:Finds the maximum value from the given file2. Code:Topapp.javaPackage Suanfa;import Java.io.ioexception;import Java.net.uri;import org.apache.hadoop.conf.configuration;import Org.apache.hadoop.fs.filesystem;import Org.apache.hadoop.fs.path;import Org.apache.hadoop.io.longwritable;import Org.apache.hadoop.io.nullwritable;import Org.apache.hadoop.

(args[1]));//Output path Job.setmapperclass (Scoremapper.class);//MapperJob.setreducerclass (Scorereducer.class);//Reducer Job.setmapoutputkeyclass (Text.class);//Mapper key Output typeJob.setmapoutputvalueclass (Scorewritable.class);//Mapper value Output type Job.setinputformatclass (Scoreinputformat.class);//Set Custom input formats Job.waitforcompletion (TRUE);return 0;} public static void Main (string[] args) throws Exception{string[] Args0 =// {"Hdfs://hadoopmaster:9000/score/score.txt","H

In the development of the Mr Program of Hadoop, it is often necessary to statistic some Map/reduce's running state information, which can be implemented by custom counter, which is done by the Code runtime check instead of the configuration information.1. Create a Counter enumeration class of your own.enum Process_counter { bad_records, bad_groups;}2, in need of statistics, such as map or reduce phase of the following operations.// added 1 // 1

longwritable for long. But there are some differences between text and string, which is a UTF-8 format of writable, and string in Java is a Unicode character. So the direct use of the value.tostring () method, the default character is UTF-8 encoded, so the original GBK encoded data using the text read into the direct use of the method will become garbled.The correct method is to convert the value of the input text type to a byte array (value.getbytes ()), using the string constructor string (by

Step OneIf not, do not set up the HBase development environment blog, see my next blog.HBase Development Environment Building (Eclipse\myeclipse + Maven)　　Step one, need to add. As follows:In the project name, right-click,Then, write Pom.xml, here not much to repeat. SeeHBase Development Environment Building (Eclipse\myeclipse + Maven)When you are done, write the code, right.Step two some steps after the HBase development environment is built (export exported jar package or Ant mode)Here, do not

= serverSocket. accept (); // Construct a data input stream to receive data DataInputStream in = new DataInputStream (soc. getInputStream ()); // Construct a data output stream to send data DataOutputStream out = new DataOutputStream (soc. getOutputStream ()); // Disconnect Soc. close () Client Process // Create a client Socket Socket soc = new Socket (serverHost, port ); // Construct a data input stream to receive data DataInputStream in = new DataInputStream (soc. ge

)); } } } }}Singletontablejoin class PackageCom.cn.singletonTableJoin;Importorg.apache.hadoop.conf.Configuration;ImportOrg.apache.hadoop.fs.Path;ImportOrg.apache.hadoop.io.Text;ImportOrg.apache.hadoop.mapreduce.Job;ImportOrg.apache.hadoop.mapreduce.lib.input.FileInputFormat;ImportOrg.apache.hadoop.mapreduce.lib.output.FileOutputFormat;ImportOrg.apache.hadoop.util.GenericOptionsParser;/*** Single-Table Association *@authorRoot **/ Public classSingletontablejoin { Public Stati

); //setting the input and output path of a fileFileinputformat.addinputpath (Job,NewPath (otherargs[0])); Fileoutputformat.setoutputpath (Job,NewPath (otherargs[1])); //set up mapper and reduce processing classesJob.setmapperclass (Averagemapper.class); Job.setreducerclass (averagereduce.class); //Setting the output key-value data typeJob.setoutputkeyclass (Text.class); Job.setoutputvalueclass (intwritable.class); //submit the job and wait for it to completeSystem.exit (Job.waitforcompletion (t

, [0, 20, 10, 25, 15])In the case of calling Combiner, the output data is now processed locally on each map (the maximum temperature of the current map is calculated) and then lost to reduce, as follows:Fir Map Combined:(1950, 20)Sec Map Combined:(1950, 25)At this point, reduce will use the following data as input, thereby reducing the amount of data transferred between map and reduce:(1950, [20, 25])4, the combiner processing data or map output data shuffle processing, so-called shuffle process

Hadoop MapReduce:MapReduce reads the data from disk every time it executes, and then puts the data on the disk after the calculation is complete.Spark Map Reduce:RDD is everything for dev:Basic Concepts:Graph RDD:Spark Runtime:ScheduleDepency Type:Scheduler Optimizations:Event Flow:Submit Job:New Job Instance:Job in Detail:Executor.launchtask:Standalone:Work Flow:Standalone Detail:Driver Application to Clustor:Worker Exception:Executor Exception:Maste

what circumstances. Scenario 1: If we know that two source data is divided into partition of the same size, and that each partition is sorted with a key value that fits as a join keyScenario 2: When you join large data, there is usually only one source data that is huge, and the other data can be reduced in order of magnitude. For example, a phone company's user data may have only thousands user data, but his transaction data may have 1 billion speci

I don't know why I don't really want to learn about mapreduce, but now I think this may take some time to study. Here I will record the wordcount code of the next mapreduce instance. 1, Pom. xml: 2、WordCountMapper：　　 Import org. Apache.

Prerequisite Preparation: 1.hadoop installation is operating normally. Hadoop installation Configuration Please refer to: Ubuntu under Hadoop 1.2.1 Configuration installation 2. The integrated development environment is normal. Integrated development environment Configuration Please refer to: Ubuntu building Hadoop Sou