Several small details about hadoop usage

I = m5m] ef
1. Chinese problemsKkrlf? RC
Chinese characters are parsed from the URL, but are still garbled characters printed in hadoop? We once thought that hadoop does not support Chinese characters. Later we checked the source code and found that hadoop only does not support outputting Chinese Characters in GBK format.

This is the code in textoutputformat. class. The default output of hadoop is inherited from fileoutputformat. One of the two subclasses of fileoutputformat is output based on binary streams, and the other is output based on text textoutputformat.

Public class textoutputformat <K, V> extends fileoutputformat <K, V> {7 (Ni _ | $ |
Protected static class linerecordwriter <K, V> & E {CQ # K
Implements recordwriter <K, V> {'7 lhwqn <
Private Static final string utf8 = "UTF-8"; // It is written to UTF-82 kp0 //
Private Static final byte [] newline; KTC ''xv
Static {: htz]
Try {0 _! ') +
Newline = "/N". getbytes (utf8); ry $ ZF ~ [
} Catch (unsupportedencodingexception UEE) {e (E_P #
Throw new illegalargumentexception ("can't find" + utf8 + "encoding"); lb (I ^
} Iy <| <* s2d
} N/-nbrvsf
... K-: WM 'C
Public linerecordwriter (dataoutputstream out, string keyvalueseparator) {11' ^ jmka
This. Out = out; D8) PS,
Try {'U </4 & W
This. keyvalueseparator = keyvalueseparator. getbytes (utf8);: 4gc 'B R
} Catch (unsupportedencodingexception UEE) {@ R. W + E =
Throw new illegalargumentexception ("can't find" + utf8 + "encoding"); J _ (dh2d
} Qu x1n
} AB} kt ($
... 0 [# Zn
Private void writeobject (Object O) throws ioexception {yt5 SY
If (O instanceof text) {rudj "OGO
Text to = (text) O; nspawi | 4
Out. Write (to. getbytes (), 0, To. getlength (); // you also need to modify Q & DM * here *! JQ
} Else {5 o't -'
Out. Write (O. tostring (). getbytes (utf8);/80 W? 9qj
} Rxi/& '+ H
} 9a6ij *#
... Qxquxf> :#
} | 3bcq (Zr/P
We can see that the default output of hadoop is written to UTF-8. Therefore, if the decode is correct, you can set character of Linux client to UTF-8 to see Chinese characters. Because hadoop outputs Chinese Characters in UTF-8 format. NF) B4 'nd
Most databases use GBK to define fields. What if we want hadoop to output Chinese Characters in GBK format to be compatible with databases? _. {I1*6y2
We can define a new class:. C5 )'
Public class gbkoutputformat <K, V> extends fileoutputformat <K, V> {STS nu +
Protected static class linerecordwriter <K, V> # q] ^ 9/; | 4n
Implements recordwriter <K, V> {noccotf
// Write it as GBK to F "UA 'erci
Private Static final string GBK = "GBK ";T2dv !} 7 P
Private Static final byte [] newline; rrqkw_9mb
Static {8y, IMJ/(V
Try {0! W $ CZ [
Newline = "/N". getbytes (GBK); Ms ~ + P'
} Catch (unsupportedencodingexception UEE) {} <b42
Throw new illegalargumentexception ("can't find" + GBK + "encoding"); X {u/| E {
} K D9 <&. y {
} Ldx/"FQ
... SJL &/),
Public linerecordwriter (dataoutputstream out, string keyvalueseparator) {P? O | n <46
This. Out = out; X-<L + WP
Try {0,] m.) WS
This. keyvalueseparator = keyvalueseparator. getbytes (GBK); JS 'J} W
} Catch (unsupportedencodingexception UEE) {_ '0c70
Throw new illegalargumentexception ("can't find" + GBK + "encoding"); VK> CF>
} P; U (2; 9 N
} J | au} z8m
... /(& Udg $
Private void writeobject (Object O) throws ioexception {? 0x = ASCP
If (O instanceof text) {$ 1ut '> %
// Text to = (text) O; k "/3/'t
// Out. Write (to. getbytes (), 0, To. getlength (); + A-z> T (
//} Else {@ H, 3 "2 W {EV
Out. Write (O. tostring (). getbytes (GBK); ikvfbtx: Y
} + 'F3 _ XD
} Isu4d
... El_il .:
} U [{va6
Then add conf1.setoutputformat (gbkoutputformat. Class); R ([W4 [~
That is, Chinese characters can be output in GBK format.

2. Comparison of compression and efficiency in the computing processHF // 2vl
We have previously introduced that compressing input files can improve some computing efficiency. Further explanation is provided. 2 gxkoch
Why does compression increase the computing speed? This is because mapreduce computing will distribute and copy data files to all datanode. compression can reduce the time spent on bandwidth. When the time is longer than the compression time, the computing speed will increase. (9of, 2] & E
In addition to compressing input files, hadoop can also compress map output and reduce output during computing. What is the effect of compression in this computation? ? UV | M
Test environment: hadoop cluster with 35 nodes, single machine with 2 CPU, 8 core, 8 GB memory, and RedHat 2.6.9, one for namenode and one for second namenode, namenode and second namenode are not datanode Yve <+ z/7
The input file size is 36 million GB without compression, and the records size is about. Mapreduce programs are divided into two jobs:; R] ~ 9aan
Job1: Map splits the record according to the user field as the key, and reduces as the outer join. In this way, the reduce output is 8.7 billion records, and the size is 540 gb q. bx NTA"
Job2: Map reads the 8.7 billion data records and outputs them. Reduce performs simple statistics. The last records is 0.25 billion records, with a size of 16 GB <~ Zpt & C] V
Computing Time: 54 min

Only the map in the second stage is compressed (the map output in the first stage is not large and there is no need for compression). Test Result: the computing time is 39 minutes.

The visible time is reduced by 15 minutes. Note the following parameters. U & W/NJ
When not compressed: O] DYS, V
Local bytes READ = 1923047905109: 3 [; 9 xchj
Local bytes written = 1685607947227 "j8') xxa (
Compression:/u> | ^ $4 #5
Local bytes READ = 770579526349 | RL/2j |
Local bytes written = 245469534966 PLR $ s
The number of local reads and writes is greatly reduced

As for the compression of reduce output, it is a pity that the test has basically not improved the speed. It may be because the output of the first job is mostly map on the local machine without network transmission. {Dyz | o <
Attachment: compress map output. You only need to add jobconf. setmapoutputcompressorclass (defaultcodec. Class)

3. Concerning the quantity setting of Reduce? & = JGK ^ EJ
The number of reduce instances is suitable. At present, it is considered that the reduce quantity is equal to half of the total cores of datanode in the cluster. For example, if there are 32 datanode in the cluster and each 8 cores, the reduce speed is set to 128 as fast. Because each machine has 8 cores, 4 for map, and 4 for reduce computing, it is suitable. U/(>
Minor test: For the same program J & [U $ p * k
Reduce num = 32, reduce time = 6 min 3e9udn2
Reduce num = 128, reduce time = 2 min m'vje | + T
Reduce num = 320, reduce time = 5 min

 

4. An error is thrown when a mapreduce instance runs normally.

Java. Io. ioexception: All datanodes XXX. XXX: XXX are bad. Aborting...

At org. Apache. hadoop. dfs. dfsclient $ dfsoutputstream. processdatanodeerror (dfsclient. Java: 2158)

At org. Apache. hadoop. dfs. dfsclient $ dfsoutputstream. Access $1400 (dfsclient. Java: 1735)

At org. Apache. hadoop. dfs. dfsclient $ dfsoutputstream $ datastreamer. Run (dfsclient. Java: 1889)

Java. Io. ioexception: cocould not get block locations. Aborting...

At org. Apache. hadoop. dfs. dfsclient $ dfsoutputstream. processdatanodeerror (dfsclient. Java: 2143)

At org. Apache. hadoop. dfs. dfsclient $ dfsoutputstream. Access $1400 (dfsclient. Java: 1735)

At org. Apache. hadoop. dfs. dfsclient $ dfsoutputstream $ datastreamer. Run (dfsclient. Java: 1889)

It was found that the cause of the problem was that too many files were opened on the Linux machine. Run the ulimit-N command to check that the default number of files opened in Linux is 1024. Modify/ECT/security/limit. conf and add hadoop soft 65535.

Run the program again (it is recommended that all datanode be modified) to solve the problem.

P.s: It is said that hadoop DFS cannot manage more than MB of files, which is to be verified.

5 after running for a period of time hadoop can not stop-all.sh problems, Display Error

No tasktracker to stop, no datanode to stop

The cause is that when hadoop stops, it is based on the mapred and DFS process numbers on datanode. The default process number is stored in/tmp. By default, Linux deletes files in this directory at intervals (generally one month or seven days. So delete the hadoop-hadoop-jobtracker.pid and hadoop-hadoop-namenode.pid two files, namenode will naturally not find the two processes on datanode.

The export hadoop_pid_dir in the configuration file can solve this problem.

There is also an article that mentions several hadoop/mapred optimization details.

{
Return checkurl (this)
} "Href =" http://thethethethethethe.spaces.live.com/blog/cns "target =" _ blank "> http://thethethethethethe.spaces.live.com/blog/cns! A001241972ea08ea! 228. Entry