mapreduce algorithm in hadoop

based on the recommended configuration of Horntonworks, a common memory allocation scheme for various components on Hadoop cluster is given. The right-most column of the scenario is a 8G VM allocation scheme that reserves 1-2g memory to the operating system, assigns 4G to Yarn/mapreduce, and of course includes hive, and the remaining 2-3g is reserved for hbase when it is necessary to use HBase.

HDFs = Mypath.getfilesystem (conf);//Get File systemif (Hdfs.isdirectory (MyPath)){//If this output path exists in the file system, delete theHdfs.delete (MyPath, true);} Job Wcjob = new Job (conf, "WC");//Build a Job object named TestanagramSet the jar package for the classes that are used by the entire jobWcjob.setjarbyclass (Wcrunner.class);Mapper and reducer classes used by this jobWcjob.setmapperclass (Wcmapper.class);Wcjob.setreducerclass (Wcreducer.class);Specify the output data kv type

Configuring MapReduceconfiguration>configuration>Plus this.configuration> property > name>Mapreduce.framework.namename> value>Yarnvalue> Property >configuration>And then configure it inside the yarn-site.xml.configuration> -- property > name>Yarn.resourcemanager.hostnamename> value>Hadoop1value> Property > property > name>Yarn.nodemanager.aux-servicesname> value>Mapreduce_shufflevalue> Property > property > name>Yarn.n

(Firstpartitioner.class);//partition functionJob.setsortcomparatorclass (Keycomparator.class);//This course does not have custom sortcomparator, but instead uses Intpair's own sortJob.setgroupingcomparatorclass (Groupingcomparator.class);//Group functionJob.setmapoutputkeyclass (Intpair.class);Job.setmapoutputvalueclass (Intwritable.class);Job.setoutputkeyclass (Text.class);Job.setoutputvalueclass (Intwritable.class);Job.setinputformatclass (Textinputformat.class);Job.setoutputformatclass (Text

Tags: hadoop Business Logic In fact, it is very simple to input two files, one as the basic data (student information file) and the other is the score information file.Student Information File: stores student data, including student ID and Student name Score data: stores scores of students, including student IDs, subjects, and scores. We will use M/R to associate data based on student IDs. The final result is student name, subject, and score. Analog d

processing classesJob.setmapperclass (Datamapper.class); Job.setreducerclass (datareduce.class); //Setting the output key-value data typeJob.setoutputkeyclass (Text.class); Job.setoutputvalueclass (Text.class); //submit the job and wait for it to completeSystem.exit (Job.waitforcompletion (true) ? 0:1); } }Add one point: When a file is sliced, it starts a mapper process according to the default 64M data block principle.Example: For example, Data.log has 20M, will start a mapper process, Data

(args[1]));//Output path Job.setmapperclass (Scoremapper.class);//MapperJob.setreducerclass (Scorereducer.class);//Reducer Job.setmapoutputkeyclass (Text.class);//Mapper key Output typeJob.setmapoutputvalueclass (Scorewritable.class);//Mapper value Output type Job.setinputformatclass (Scoreinputformat.class);//Set Custom input formats Job.waitforcompletion (TRUE);return 0;} public static void Main (string[] args) throws Exception{string[] Args0 =// {"Hdfs://hadoopmaster:9000/score/score.txt","H

In the development of the Mr Program of Hadoop, it is often necessary to statistic some Map/reduce's running state information, which can be implemented by custom counter, which is done by the Code runtime check instead of the configuration information.1. Create a Counter enumeration class of your own.enum Process_counter { bad_records, bad_groups;}2, in need of statistics, such as map or reduce phase of the following operations.// added 1 // 1

longwritable for long. But there are some differences between text and string, which is a UTF-8 format of writable, and string in Java is a Unicode character. So the direct use of the value.tostring () method, the default character is UTF-8 encoded, so the original GBK encoded data using the text read into the direct use of the method will become garbled.The correct method is to convert the value of the input text type to a byte array (value.getbytes ()), using the string constructor string (by

= Mypath.getfilesystem (conf);if (Hdfs.isdirectory (MyPath)){Hdfs.delete (MyPath, true);}@SuppressWarnings ("deprecation")Job Job = new Job (conf, "gender");//Create a new taskJob.setjarbyclass (Gender.class);//Main classJob.setmapperclass (pcmapper.class);//mapperJob.setreducerclass (pcreducer.class);//reducerJob.setpartitionerclass (Myhashpartitioner.class);Job.setpartitionerclass (Pcpartitioner.class);//Set Partitioner classJob.setnumreducetasks (3);//reduce number set to 3Job.setmapoutputke

); } }}Datasort class PackageCom.cn.sort;Importorg.apache.hadoop.conf.Configuration;ImportOrg.apache.hadoop.fs.Path;Importorg.apache.hadoop.io.IntWritable;ImportOrg.apache.hadoop.mapreduce.Job;ImportOrg.apache.hadoop.mapreduce.lib.input.FileInputFormat;ImportOrg.apache.hadoop.mapreduce.lib.output.FileOutputFormat;ImportOrg.apache.hadoop.util.GenericOptionsParser;/*** Data Sorting *@authorRoot **/ Public classDatasort { Public Static voidMain (string[] args)throwsException {Configuration conf=

The 1.Text packet was wrongly guided.The import Com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider.Text;Change to import Org.apache.hadoop.io.Text;.2. The local compilation environment and the Java version in the production environment do not match. It is possible that the JDK does not match or the JRE does not match. It doesn't matter if they match.3.map and reduce are overloaded with mapper and reducer classes respectively. Cannot be a method of your own definitionThe

file size does not exceed 1.1 times times the Shard size, it will be divided into a shard, avoid opening two map, one of the running data is too small, wasting resources.Summary, the Shard process is about, first traverse the target file, filter some non-conforming files, and then add to the list, and then follow the file name to slice the Shard (the size of the previous calculation of the size of the formula, the end of a file may be merged, in fact, often write network programs know), and the

Hadoop Reading Notes series article:http://blog.csdn.net/caicongyang/article/category/21668551. Description:The two columns of data given are sorted first in ascending order of the first column, and the second column in ascending order when the first column is the sameData format:3332312221112. Code Sortapp.javaPackage Sort;import Java.io.datainput;import java.io.dataoutput;import java.io.ioexception;import Java.net.URI; Import Org.apache.hadoop.conf.

Transferred from http://www.ptbird.cn/mapreduce-tempreture.html I. Description of requirements 1, data file description There are some data files stored in the HDFs in the form of text, as shown in the following example: In the middle of the date and time is a space, as a whole, indicating the detection of site monitoring time, followed by the detection of temperature, the middle through the tab \ t separated. 2, the demand calculation in 1949-1955 y

Step OneIf not, do not set up the HBase development environment blog, see my next blog.HBase Development Environment Building (Eclipse\myeclipse + Maven)　　Step one, need to add. As follows:In the project name, right-click,Then, write Pom.xml, here not much to repeat. SeeHBase Development Environment Building (Eclipse\myeclipse + Maven)When you are done, write the code, right.Step two some steps after the HBase development environment is built (export exported jar package or Ant mode)Here, do not

= serverSocket. accept (); // Construct a data input stream to receive data DataInputStream in = new DataInputStream (soc. getInputStream ()); // Construct a data output stream to send data DataOutputStream out = new DataOutputStream (soc. getOutputStream ()); // Disconnect Soc. close () Client Process // Create a client Socket Socket soc = new Socket (serverHost, port ); // Construct a data input stream to receive data DataInputStream in = new DataInputStream (soc. ge

)); } } } }}Singletontablejoin class PackageCom.cn.singletonTableJoin;Importorg.apache.hadoop.conf.Configuration;ImportOrg.apache.hadoop.fs.Path;ImportOrg.apache.hadoop.io.Text;ImportOrg.apache.hadoop.mapreduce.Job;ImportOrg.apache.hadoop.mapreduce.lib.input.FileInputFormat;ImportOrg.apache.hadoop.mapreduce.lib.output.FileOutputFormat;ImportOrg.apache.hadoop.util.GenericOptionsParser;/*** Single-Table Association *@authorRoot **/ Public classSingletontablejoin { Public Stati

); //setting the input and output path of a fileFileinputformat.addinputpath (Job,NewPath (otherargs[0])); Fileoutputformat.setoutputpath (Job,NewPath (otherargs[1])); //set up mapper and reduce processing classesJob.setmapperclass (Averagemapper.class); Job.setreducerclass (averagereduce.class); //Setting the output key-value data typeJob.setoutputkeyclass (Text.class); Job.setoutputvalueclass (intwritable.class); //submit the job and wait for it to completeSystem.exit (Job.waitforcompletion (t

, [0, 20, 10, 25, 15])In the case of calling Combiner, the output data is now processed locally on each map (the maximum temperature of the current map is calculated) and then lost to reduce, as follows:Fir Map Combined:(1950, 20)Sec Map Combined:(1950, 25)At this point, reduce will use the following data as input, thereby reducing the amount of data transferred between map and reduce:(1950, [20, 25])4, the combiner processing data or map output data shuffle processing, so-called shuffle process