Running the first Hadoop instance on Eclipse-WordCount (word counting program)

Demand

Calculates the frequency of each word in the file. The output results are ordered in alphabetical order by word. Each word and its frequency occupy one line, and there is a gap between the word and the frequency.

For example, enter a file with the following contents:

Hello World

Hello Hadoop

Hello MapReduce

corresponding to the input sample given above, the output sample is:

Hadoop 1

Hello 3

MapReduce 1

World 1

Programme development

For this case, the following mapreduce schemes can be designed:

1. Map stage Each node completes the input data to the word segmentation work

2. The shuffle phase completes the aggregation of the same word to the work of distributing to each reduce node (theshuffle phase is the default process for MapReduce )

3. Reduce phase is responsible for receiving all words and calculating their respective frequency

Summary

WordCount is a classic example of Hadoop, though simple but very representative.

To some extent, it also reflects the original intention of the design, the analysis of the log file.

Running the first Hadoop instance on Eclipse-WordCount (word counting program)