Install and configure the hadoop plug-in myeclipse and eclipse in windows/Linux, and myeclipsehadoop

Install and configure the hadoop plug-in myeclipse and eclipse in windows/Linux, and myeclipsehadoop

I recently want to write a test program MaxMapperTemper on windows, and there is no server around, so I want to configure it on windows 7.

Succeeded. I want to take notes here to help you.

The installation and configuration steps are as follows:

Myeclipse 8.5

Hadoop-1.2.2-eclipse-plugin.jar

1. Install Hadoop development plug-in hadoop installation package contrib/directory has a plug-in hadoop-1.2.2-eclipse-plugin.jar, copy to myeclipse root directory/dropins directory. 2. Start myeclipse and Open Perspective: [Window]-> [Open Perspective]-> [Other... ]-> [Map/Reduce]-> [OK] 3. Open a View: [Window]-> [Show View]-> [Other... ]-> [MapReduce Tools]-> [Map/Reduce Locations]-> [OK] 4. Add Hadoop location: location name: localhost. map/Reduce Master in this box Host: Is the Cluster machine where jobtracker is located, here write localhostHort: Is the jobtracker port, write here is 9999 these two parameters are in the mapred-site.xml mapred. job. in the ip address and port DFS Master in tracker, Host: Is the Cluster machine where namenode is located, here write localhostPort: Is the namenode port, here write 8888 these two parameters is the fs in the core-site.xml. default. ip address and port (Use M/R master host) in name. If this check box is selected, the check box is the same as the host in the Map/Reduce Master box by default. If this check box is not selected, you can define the input by yourself. Here, jobtracker and namenode are on the same machine, so check it.) user name: the user name connecting to hadoop, because I installed hadoop with lsq users and didn't set up other users, lsq is used. Do not fill in the following. Click "finish". A record is added to the view. Restart myeclipse and re-edit the connection record just created. Now we edit the advance parameters tab. (restart and edit the advance parameters tab. Cause: when creating a new connection, some attributes of this advance paramters tab page will not be displayed, and cannot be set if it cannot be displayed. Therefore, you must restart eclipse and edit it before you can see it) here most of the attributes have been automatically filled in, in fact, is the core-defaulte.xml, hdfs-defaulte.xml, mapred-defaulte.xml inside some of the configuration properties shown. When installing hadoop, the configuration files of its site series are changed, so the same settings should be made here. The main concern is the following attributes: fs. defualt. name: mapred has been set on the General tab. job. tracker: dfs is also set on the General tab. replication: the default value here is 3, because I set it to 1 in the hdfs-site.xml, so it also needs to be set to 1 hadoop here. job. ugi: Enter lsq, Tardis, followed by the hadoop user connected to the comma, And the Tardis is written after the comma (this attribute does not know why I do not have it ...) then click finish, and the connection will be established (start the sshd service and start the hadoop process). Mark 5 on the connection. Create a Map/Reduce Project: [File]-> [New]-> [Project... ]-> [Map/Reduce]-> [Map/Reduce Project]-> [Project name: WordCount]-> [Configure Hadoop install directory... ]-> [Hadoop installation directory: D: \ cygwin \ home \ lsq \ hadoop-0.20.2]-> [Apply]-> [OK]-> [Next]-> [Allow output folders for source folders]-> [Finish]

6, new WordCount class: Add/write source code: D: \ cygwin \ home \ lsq \ hadoop-1.2.2/src/examples/org/apache/hadoop/examples/WordCount. javapackage org. apache. hadoop. examples; import java. io. IOException; import java. util. stringTokenizer; import org. apache. hadoop. conf. configuration; import org. apache. hadoop. fs. path; import org. apache. hadoop. io. intWritable; import org. apache. hadoop. io. text; import org. apache. hadoop. mapreduce. job; import Org. apache. hadoop. mapreduce. mapper; import org. apache. hadoop. mapreduce. reducer; import org. apache. hadoop. mapreduce. lib. input. fileInputFormat; import org. apache. hadoop. mapreduce. lib. output. fileOutputFormat; import org. apache. hadoop. util. genericOptionsParser; public class WordCount {public static class TokenizerMapper extends Mapper <Object, Text, Text, IntWritable> {private final static IntWritable one = New IntWritable (1); private Text word = new Text (); public void map (Object key, Text value, Context context) throws IOException, interruptedException {StringTokenizer itr = new StringTokenizer (value. toString (); while (itr. hasMoreTokens () {word. set (itr. nextToken (); context. write (word, one) ;}}public static class IntSumReducer extends CER <Text, IntWritable, Text, IntWritable> {private IntW Ritable result = new IntWritable (); public void reduce (Text key, Iterable <IntWritable> values, Context context) throws IOException, InterruptedException {int sum = 0; for (IntWritable val: values) {sum + = val. get ();} result. set (sum); context. write (key, result) ;}} public static void main (String [] args) throws Exception {Configuration conf = new Configuration (); String [] otherArgs = new Generi COptionsParser (conf, args). getRemainingArgs (); if (otherArgs. length! = 2) {System. err. println ("Usage: wordcount <in> <out>"); System. exit (2);} Job job = new Job (conf, "word count"); job. setJarByClass (WordCount. class); job. setMapperClass (TokenizerMapper. class); job. setCombinerClass (IntSumReducer. class); job. setReducerClass (IntSumReducer. class); job. setOutputKeyClass (Text. class); job. setOutputValueClass (IntWritable. class); FileInputFormat. addInputPath (job, new Path (OtherArgs [0]); FileOutputFormat. setOutputPath (job, new Path (otherArgs [1]); System. exit (job. waitForCompletion (true )? 0: 1);} 7. Upload the simulated data folder. To run the program, an Input Folder and an output folder are required. Output Folder, which is automatically generated after the program runs successfully. We need to input a folder for the program. (1) create a folder input in the current directory (such as the hadoop installation directory), and create two files file1 and file2 under the folder. The content of these two files is as follows: file1 Hello World Bye World file2 Hello Hadoop Goodbye Hadoop (2 ),. upload the folder input to the distributed file system. Run the following command in the Hadoop daemon terminal cd to the hadoop installation directory: bin/hadoop fs-put input in 8. Configure the running parameters: ① in the newly created Project WordCount, click WordCount. java, right-click --> Run As --> Run Configurations ② In the pop-up Run Configurations dialog box, click Java Application, right-click --> New, in this case, a new application named WordCount ③ is created to configure the running parameters, click Arguments, and enter "the Input Folder you want to pass to the Program and the folder you want the Program to save the computing result" in Program arguments ", example: (if the runtime times java. lang. outOfMemoryError: Java heap space configures VM arguments (under Program arguments)-Xms512m -Xmx1024m-XX: MaxPermSize = 256 m) 8. Click Run, Run the program, and Run the program. After a period of time, the running is completed, you can run the following command on the terminal to check whether the folder output: bin/hadoop fs-ls is generated: if bin/hadoop fs-cat output/* is displayed as follows, the first MapReduce program has been successfully run in myeclipse. Bye 1 Goodbye 1 Hadoop 2 Hello 2 World 2