Mapreduce programming Series 12 integrate newlisp scripts with hadoop streaming technology

The environment in this article is different from that in hadoop 1.x. It is tested in hadoop 2.x. Functions are the same as those of the preceding log handler.

The first newlisp script acts as a Mapper. It reads text data in stdin, uses did as the key and value as 1, and then outputs the result to stdout.

The second newlisp script plays the role of reducer. It reads <key, values> In stdin, the key is DIC, and values is all values. After summing the value, it writes to stdout.

Finally, you can see the results under HDFS.

The advantage of Script Programming is that it is convenient to test. Now we first develop newlisp scripts to read files and follow the map logic for processing, and then hand them to the subsequent newlisp scripts to follow the reduce process.

The following is the map. LSP code:

#!/usr/bin/newlisp(while (read-line)  (set ‘value (parse (current-line) ","))  (println (string (value 2) "\t1")))(exit)

Test:

cat logs/sign_2014-05-10.0.csv | ./map.lsp

The results are good:

537025b84700aab27472b87f        1537023124700aab27472b82a        1537031a24700aab27472b982        1537023c84700aab27472b841        1537014e74700aab27472b48b        153702cac4700aab27472b928        1537049cd4700aab27472ba91        15370dd0b4700aab27472bde4        1

Split a row of records according to, put them in a list, and then take the third element, that is, the device ID. Then add \ t as the column separator, and then add 1.

In this way, the <key, value> in the form of did \ t1 \ n is converted to reduce. Note that the println function of newlisp code will automatically add \ n after the string.

The following code implements reduce. LSP:

(new Tree ‘my-table)(while (read-line)  (set ‘line-value (parse (current-line) "\t"))  (set ‘key (line-value 0))  (set ‘value (int (line-value 1)))  (set ‘v (my-table key))  (if v      (my-table key (+ v value))    (my-table key value)      ))(dolist (item (my-table)) (println (item 0) "\t" (item 1)))(exit)

Create a my-table to save <key, value>

Split each line of data output by map. LSP according to \ t to obtain the key and value.

Store the data in my-table and use the key for query. If there is a key, add 1 to the value. If there is no key, add it.

Finally, traverse the entire my-table and output data such as did \ tsum \ n.

The following command can be used to connect the map and reduce scripts for testing:

cat logs/sign_2014-05-10.0.csv | ./map.lsp | sort | ./reduce.lsp

When deploying a hadoop cluster, ensure that the newlisp binary program is deployed in the/usr/bin/directory of all nodes and has the execution permission. Because the newlisp program itself is very small, it is easy to deploy and use SCP directly.

Then run the hadoop command:

The hadoop jar hadoop-streaming-1.0.0.jar-files map. LSP reduce. LSP-Input/user/chenshu/share/logs-output/user/chenshu/share/output/lisp-mapper map. LSP-reducer reduce. LSP

Mapreduce programming Series 12 integrate newlisp scripts with hadoop streaming technology