"Reprint" Hadoop custom Recordreader

Transferred from: http://www.linuxidc.com/Linux/2012-04/57831.htm

The default linerecordreader is the key value when the map output is the offset of each line, the contents of each line as the value of the map, the default delimiter is carriage return and line wrapping.

Now to change the input <key,value> value of the map, the path (or filename) of the file corresponding to the key, value corresponds to the contents of the file (content).

Then we need to rewrite InputFormat and Recordreader, because Recordreader is called in InputFormat, of course, rewriting Recordreader is the point!

Here's a look at the rewrite of code InputFormat:

2. Public class Chdicinputformat extends fileinputformat<text,text>
4. implements jobconfigurable{
6. Private compressioncodecfactory compressioncodecs = null;
8. Public void Configure (jobconf conf) {
10. Compressioncodecs = new compressioncodecfactory (conf);
12. }
14. /**
16. * @brief issplitable file is not sliced, the file must be processed as a whole
18.      *
20. * @param FS
22. * @param file
24.      *
26. * @return False
28.      */
30. protected boolean issplitable (FileSystem FS, Path file) {
32. //Compressioncodec codec = compressioncodecs.getcode (file);
34. return false; //In file units, each unit as a split, even if the size of a single file exceeds 64M, that is, Hadoop a block size, also do not shard
36. }
40. Public recordreader<text,text> getrecordreader (Inputsplit genericsplit,
42. jobconf job, Reporter Reporter) throws ioexception{
44. Reporter.setstatus (Genericsplit.tostring ());
46. return new Chdicrecordreader (Job, (Filesplit) genericsplit);
48. }
52. }

Here's a look at Recordreader's rewrite:

2. Public class Chdicrecordreader implements recordreader<text,text> {
4. private static final log log = Logfactory.getlog (chdicrecordreader.   class. GetName ());
6. Private compressioncodecfactory compressioncodecs = null;
8. Private long start;
10. Private long pos;
12. Private long end;
14. private byte[] buffer;
16. private String keyName;
18. private fsdatainputstream Filein;
22. Public chdicrecordreader (Configuration job,filesplit split) throws ioexception{
24. Start = Split.getstart (); ///from which you can see that each file is as a split
26. End = Split.getlength () + start;
28. Final path Path = Split.getpath ();
30. KeyName = Path.tostring ();
32. Log.info ("filename in HDFs is:" + keyName);
34. Final FileSystem fs = Path.getfilesystem (Job);
36. Filein = Fs.open (path);
38. Filein.seek (start);
40. Buffer = new byte[(int) (End-start)];
42. This . pos = start;
46. }
50. Public Text CreateKey () {
52. return new Text ();
54. }
58. Public Text createvalue () {
60. return new Text ();
62. }
66. Public long getPos () throws ioexception{
68. return pos;
70. }
74. Public float getprogress () {
76. if (start = = end) {
78. return 0. 0f;
80. } Else {
82. return math.min (1. 0f, (Pos-start)/(float) (End-start));
84. }
86. }
90. Public boolean Next (text key, text value) throws ioexception{
92. While (pos < end) {
94. Key.set (KeyName);
96. Value.clear ();
98. Filein.readfully (Pos,buffer);
100. Value.set (buffer);
102. //Log.info ("---content:" + value.tostring ());
104. pos + = Buffer.length;
106. Log.info ("End is:" + end + "POS is:" + pos);
108. return true;
110. }
112. return false;
114. }
118. Public void close () throws ioexception{
120. if(filein! = null) {
122. Filein.close ();
124. }
128. }
132. }

This new read-in format is available by using the code above and then setting the InputFormat class in the main function.

"Reprint" Hadoop custom Recordreader