Hadoop configuration item organization (hdfs-site.xml)

 

Name	Value	Description
DFS. Default. Chunk. View. Size	32768	The size of each file displayed on the HTTP access page of namenode usually does not need to be set.
DFS. datanode. Du. Reserved	1073741824	The size of the space reserved by each disk, which must be set to be used mainly for non-HDFS files. The default value is not reserved, and the value is 0 bytes.
DFS. Name. dir	/Opt/data1/HDFS/Name, /Opt/data2/HDFS/Name, /Nfs/data/HDFS/Name	We recommend that you retain one copy of the metadata used by NN on NFS as the 1.0 ha solution. You can also use it on multiple hard disks of a server.
DFS. Web. ugi	Nobody, nobody	Users and groups used by Web Tracker page servers such as NN and JT
DFS. Permissions	True | false	Whether to enable the DFS permission. Generally, I set false to train others on the interface through development tools to avoid misoperation. If I set it to true, sometimes the data cannot be accessed because of the permission.
DFS. permissions. supergroup	Supergroup	The default HDFS super permission group is supergroup. the user who starts hadoop is usually superuser.
DFS. Data. dir	/Opt/data1/HDFS/data, /Opt/data2/HDFS/data, /Opt/data3/HDFS/data, ...	Real datanode data storage path. Multiple hard disks can be written and separated by commas (,).
DFS. datanode. Data. dir. perm	755	The path permission of the local folder used by datanode. The default value is 755.
DFS. Replication	3	The number of copies of HDFS data blocks. The default value is 3. Theoretically, more copies run faster, but more storage space is required. Rich people can call 5 or 6
DFS. Replication. Max	512	Sometimes, after a temporary DNS fault is restored, the data exceeds the default number of backups. The maximum number of copies is usually useless and does not need to be written into the configuration file.
DFS. Replication. Min	1	The minimum number of copies.
DFS. Block. Size	134217728	The size of each file block, which is 128 MB by default. This calculation requires 128*1024 ^ 2. I have met someone who writes 128000000 directly, which is very romantic.
DFS. df. Interval	60000	Disk usage statistics automatic refresh time, in milliseconds.
DFS. Client. Block. Write. retries	3	The maximum number of retries allowed for writing data blocks. No failure is captured before this number.
DFS. Heartbeat. Interval	3	Interval of the DN heartbeat detection. Seconds
DFS. namenode. handler. Count	10	Number of threads expanded after NN is started.
DFS. Balance. bandwidthpersec	1048576	The maximum bandwidth per second used for balance. bytes are used as the unit instead of bit.
DFS. Hosts	/Opt/hadoop/CONF/hosts. Allow	A host name list file, where the host is allowed to connect to Nn, must write an absolute path. If the file content is empty, all are considered acceptable.
DFS. Hosts. Exclude	/Opt/hadoop/CONF/hosts. Deny	The basic principle is the same as above, but the list of host names that prohibit access to NN is shown here. This is useful in removing the DN from the cluster.
DFS. Max. Objects	0	The maximum number of concurrent DFS objects. All files and directory blocks in HDFS are considered as an object. 0 indicates no restriction
DFS. Replication. Interval	3	Nn calculates the internal interval of the replication block, and usually does not need to be written into the configuration file. The default value is good.
DFS. Support. append	True | false	The new hadoop supports file append operations. This is to control whether file append is allowed, but the default value is false. The reason is that there are bugs in append operations.
DFS. datanode. Failed. volumes. tolerated	0	Maximum number of bad hard disks that can cause DN to crash. The default value 0 indicates that if one hard disk breaks down, the DN will shut down.
DFS. Secondary. http. Address	0.0.0.0: 50090	SNN tracker page listening address and port
DFS. datanode. Address	0.0.0.0: 50010	The service listening port of the DN. If the port is 0, it will randomly listen to the port and send a heartbeat notification to NN.
DFS. datanode. http. Address	0.0.0.0: 50075	The tracker page listening address and port of DN
DFS. datanode. IPC. Address	0.0.0.0: 50020	The IPC listening port of the DN. If the value is 0, the listening port is transmitted to the NN through the heartbeat at the random port.
DFS. datanode. handler. Count	3	Number of service threads started by DN
DFS. http. Address	0.0.0.0: 50070	Nn tracker page listening address and port
DFS. HTTPS. Enable	True | false	Whether the NN tracker is listening for HTTPS. The default value is false.
DFS. datanode. HTTPS. Address	0.0.0.0: 50475	HTTPS tracker page listening address and port of DN
DFS. HTTPS. Address	0.0.0.0: 50470	The listener address and port of the HTTPS tracker page of NN
DFS. datanode. Max. xcievers	2048	This parameter is equivalent to the maximum number of files opened in Linux. this parameter is not included in the document. When a dataxceiver error is reported, you need to increase the value. The default value is 256.

 

This is probably the main configuration. There are some HTTPS certfile and some internal time configurations. If they are not commonly used, they will not be written.

Hadoop configuration item organization (hdfs-site.xml)