Hadoop task execution optimization

Task execution optimization

1. Speculative execution:

If jobtracker finds a delayed task, it will start another same backup task, and then kill the other task after it is executed first, therefore, on the monitoring page, you can often see that jobs that have been normally executed will be killed.

 

2. speculative execution is enabled by default, but if it is a code problem, it cannot solve the problem, and it will make the cluster slower, by setting mapred.map.tasks.speculative.exe cution and mapred.cece.tasks.speculative.exe cution in the mapred-site.xml configuration file to enable or disable speculative execution for map or reduce tasks,

 

3. Reuse JVM.

This saves the time required to start a new JVM. set mapred in the mapred-site.xml configuration file. job. reuse. JVM. num. tasks sets the maximum number of tasks on a single JVM (1,> 1 or-1 indicates no limit );

 

4. Ignore Mode

After the task fails to read data twice, it will tell jobtracker the data location, and then restart the task and

 

Hadoop task execution optimization