Key code points of Hadoopskipmode

If the Hadoop dataset contains data of different specifications or some data may cause program crash, you can set the Skip mode. The program code does not handle these exceptions and throws the exception to cause the program to crash, the hadoop framework re-executes the program and skips the record. But no code example is provided here.

If the Hadoop dataset contains data of different specifications or some data may cause program crash, you can set the Skip mode. The program code does not handle these exceptions and throws the exception to cause the program to crash, the hadoop framework re-executes the program and skips the record, see http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#Skipping+Bad+Records. However, no code example is provided here. I have tried several times and cannot work properly. After some troubleshooting, I finally confirmed the code mode that can work normally.

The following is a piece of code that can work:

Public class PwdSorter extends ConfiguredIMplements Tool {

......

Public int run (String [] args) throws Exception {

SkipBadRecords.SetMapperMaxSkipRecords(GetConf (), 5 );

JobConf jobConf = new JobConf (getConf (), getClass ());

......

}

Public static void main (String [] args) throws Exception {

Int res = ToolRunner.Run(New PwdSorter (), args );

System.Exit(Res );

}

}

The key point is SkipBadRecords.SetMapperMaxSkipRecordsBefore new JobConf. If the order of the two statements is reversed, the skip mode cannot work normally ., The default error data is recorded in the output/_ logs/skip directory. The storage format is sequence file.

In addition, if the new API, org. hadoop. mapreduce package, does not contain the SkipBadRecords class, mapred. SkipBadRecords can also be used to skip recording. (Fixed: I tried again later. The new API Skip mode failed and may still not work properly. It was recorded as incorrect .)

For more information about Hadoop, see Hadoop topic page http://www.linuxidc.com/topicnews.aspx? Tid = 13