Name |
Value |
Description |
Hadoop. Job. History. Location |
|
The path of the historical job file, which has no configuration parameters and does not need to be written in the configuration file. It is in the History folder of logs by default. |
Hadoop. Job. History. User. Location |
|
Location of historical User Files |
Io. Sort. Factor |
30 |
The number of files to be sorted when stream merging is processed. I think it is the number of files to be opened during sorting. |
Io. Sort. MB |
600 |
Memory Used for sorting, in MB. The default value is 1. I remember that it cannot exceed the mapred. Child. java. Opt setting; otherwise, oom |
Mapred. Job. Tracker |
Hadoopmaster: 9001 |
The configuration item connecting to the jobtrack server. The default value is "local", "map number 1", and "reduce number 1 ". |
Mapred. Job. tracker. http. Address |
0.0.0.0: 50030 |
Service Listening address of the tracker page of jobtracker |
Mapred. Job. tracker. handler. Count |
15 |
Jobtracker service thread count |
Mapred. task. tracker. Report. Address |
127.0.0.1: 0 |
Tasktracker listens to the server, which does not need to be configured and is not recommended by the official team. |
Mapred. Local. dir |
/Data1/HDFS/mapred/local, /Data2/HDFS/mapred/local, ... |
The folder used by mapred for local calculation. Multiple hard disks can be configured and separated by commas (,). |
Mapred. system. dir |
/Data1/HDFS/mapred/system, /Data2/HDFS/mapred/system, ... |
The folder used by mapred to store control files. Multiple hard disks can be configured and separated by commas. |
Mapred. Temp. dir |
/Data1/HDFS/mapred/temp, /Data2/HDFS/mapred/temp, ... |
The path of the Temporary Folder shared by mapred is described as above. |
Mapred. Local. dir. minspacestart |
1073741824 |
If the remaining space in the local operation folder is lower than this value, it is not calculated locally. Byte configuration. The default value is 0. |
Mapred. Local. dir. minspacekill |
1073741824 |
If the remaining space in the local computing folder is lower than this value, no new task is applied. The number of bytes. The default value is 0. |
Mapred. tasktracker. expiry. Interval |
60000 |
If TT does not send a heartbeat packet within this time period, it is deemed that TT has crashed. Unit: milliseconds |
Mapred. Map. Tasks |
2 |
By default, the number of maps used by each job indicates that if the DFS block size is set to 64 MB and a 60 MB file needs to be sorted, two map threads will be enabled, it does not work when jobtracker is set to local. |
Mapred. Reduce. Tasks |
1 |
Same as above |
Mapred. jobtracker. Restart. Recover |
True | false |
Enable task recovery upon restart. The default value is false. |
Mapred. jobtracker. taskscheduler |
Org. Apache. hadoop. mapred. Capacitytaskscheduler
Org. Apache. hadoop. mapred. Jobqueuetaskscheduler
Org. Apache. hadoop. mapred. Fairscheduler |
The important thing is to enable the task manager. If you do not set it, hadoop uses the FIFO scheduler by default. Other schedulers can use the fair and computing power scheduler. |
Mapred. Reduce. Parallel. Copies |
10 |
Number of parallel copies used by reduce in the shuffle stage. The default value is 5. |
Mapred. Child. java. opts |
-Xmx2048m -Djava. Library. Path = /Opt/hadoopgpl/native/ Linux-amd64-64 |
Memory size of the virtual machine used by each TT sub-process |
Tasktracker. http. threads |
50 |
Number of HTTP server threads used by TT to track task tasks |
Mapred. task. tracker. http. Address |
0.0.0.0: 50060 |
The http ip address and port that TT listens to by default. It can be left empty by default. If the port is set to 0, it is randomly used. |
Mapred. Output. Compress |
True | false |
The task results are compressed and output. The default value is false. False is recommended. |
Mapred. Output. Compression. Codec |
Org. Apache. hadoop. Io. Compress. defaultcodec |
The decoder used for output results. You can also use GZ, Bzip2, lzo, snappy, etc. |
Mapred. Compress. Map. Output |
True | false |
Whether the map output results are output in the compressed format before network switching. The default value is false. We recommend that you set this parameter to true to reduce the bandwidth usage and reduce the cost. |
Mapred. Map. Output. Compression. Codec |
Com. hadoop. compression. Lzo. lzocodec |
The codecs used for compressing the output in the map stage |
Map. Sort. Class |
Org. Apache. hadoop. util. Quicksort |
The algorithm used for map output sorting. The default value is "Quick Sort. |
Mapred. Hosts |
Conf/mhost. Allow |
List of TT servers allowed to connect to JT. All null values are allowed. |
Mapred. Hosts. Exclude |
Conf/mhost. Deny |
It is helpful to disable connection to the TT list of JT. |
Mapred. queue. Names |
ETL, Rush, default |
List of queue names used with the scheduler, separated by commas |
Mapred. tasktracker. Map. Tasks. Maximum |
12 |
Maximum number of map slots per server. |
Mapred. tasktracker. reduce. Tasks. Maximum |
6 |
Maximum number of reduce slots per server |