A description of the parameter meaning of the Hadoop three configuration files

1 Getting the default configuration

Configure Hadoop, which is primarily configured with Core-site.xml,hdfs-site.xml, Mapred-site.xml three profiles, by default, these profiles are empty, so it is difficult to know which configuration of these profiles can take effect, and the configuration on the Internet may not be effective because of the different versions of Hadoop. There are two ways to browse more configurations:

1. Select the appropriate version of Hadoop, download decompression, search *.xml, find Core-default.xml,hdfs-default.xml,mapred-default.xml, these are the default configuration, you can refer to these configuration instructions and key, Configure the Hadoop cluster.

2. Browse Apache official website, three profile links are as follows:

Http://hadoop.apache.org/common/docs/current/core-default.html

Http://hadoop.apache.org/common/docs/current/hdfs-default.html

Http://hadoop.apache.org/common/docs/current/mapred-default.html

Here is the default profile for the current version of Hadoop, the other version number, to go to the official website to find another. The first method to find the default configuration is best because each attribute is described and can be used directly. In addition, Core-site.xml is a global configuration, and Hdfs-site.xml and mapred-site.xml are local configurations for HDFs and mapred respectively. 2 common port configurations 2.1 HDFs Port

Parameters	Describe	Default	Configuration file	Example value
Fs.default.name Namenode	Namenode RPC Interactive port	8020	Core-site.xml	hdfs://master:8020/
Dfs.http.address	Namenode Web Management port	50070	Hdfs-site.xml	0.0.0.0:50070
Dfs.datanode.address	Datanode Control Port	50010	Hdfs-site.xml	0.0.0.0:50010
Dfs.datanode.ipc.address	RPC server address and port for Datanode	50020	Hdfs-site.xml	0.0.0.0:50020
Dfs.datanode.http.address	HTTP servers and ports for Datanode	50075	Hdfs-site.xml	0.0.0.0:50075

2.2 Mr Port

Parameters	Describe	Default	Configuration file	Example value
Mapred.job.tracker	Job-tracker Interactive Port	8021	Mapred-site.xml	hdfs://master:8021/
Job	Tracker Web Management Ports	50030	Mapred-site.xml	0.0.0.0:50030
Mapred.task.tracker.http.address	HTTP port for Task-tracker	50060	Mapred-site.xml	0.0.0.0:50060

2.3 Other Ports

Parameters	Describe	Default	Configuration file	Example value
Dfs.secondary.http.address	Secondary Namenode Web Management port	50090	Hdfs-site.xml	0.0.0.0:50090

33 Default Configuration Reference file Description 3.1 core-default.html

Serial number	Name of parameter	Parameter values	Parameter description
1	Hadoop.tmp.dir	/tmp/hadoop-${user.name}	Temporary directory settings
2	hadoop.native.lib	true	Uses the local Hadoop library identity.
3	Hadoop.http.filter.initializers		HTTP Server Filter Chain settings
4	Hadoop.security.group.mapping	Org.apache.hadoop.security.ShellBasedUnixGroupsMapping	Class settings for a list of users within a group
5	Hadoop.security.authorization	False	Service-Side Certification Open
6	Hadoop.security.authentication	Simple	No Authentication or authentication settings
7	Hadoop.security.token.service.use_ip	True	Whether to open a switch that uses an IP address as a connection
8	Hadoop.logfile.size	10000000	Maximum log file is 10M
9	Hadoop.logfile.count	10	The number of log files is 10
10	Io.file.buffer.size	4096	The buffer for the stream file is 4K
11	Io.bytes.per.checksum	512	Check digit is 512 bytes
12	Io.skip.checksum.errors	False	If the checksum fails, it throws an exception or skips the identity. True to skip over.
13	Io.compression.codecs	Org.apache.hadoop.io.compress.DefaultCodec, Org.apache.hadoop.io.compress.GzipCodec, Org.apache.hadoop.io.compress.BZip2Codec, Org.apache.hadoop.io.compress.SnappyCodec	Compression and decompression mode setting
14	Io.serializations	Org.apache.hadoop.io.serializer.WritableSerialization	Class settings for sequencing and deserialization
15	Fs.default.name	file:///	The default file URI identification setting.
16	Fs.trash.interval	0	File discard identity set, 0 to disable this feature
17	Fs.file.impl	Org.apache.hadoop.fs.LocalFileSystem	Local File Action class settings
18	Fs.hdfs.impl	Org.apache.hadoop.hdfs.DistributedFileSystem	HDFs file Action class settings
19	Fs.s3.impl	Org.apache.hadoop.fs.s3.S3FileSystem	S3 file Action class settings
20	Fs.s3n.impl	Org.apache.hadoop.fs.s3native.NativeS3FileSystem	S3 file local Action class settings
21st	Fs.kfs.impl	Org.apache.hadoop.fs.kfs.KosmosFileSystem	KFS file action class settings.
22	Fs.hftp.impl	Org.apache.hadoop.hdfs.HftpFileSystem	HTTP mode action File settings
23	Fs.hsftp.impl	Org.apache.hadoop.hdfs.HsftpFileSystem	HTTPS mode operation file Settings
24	Fs.webhdfs.impl	Org.apache.hadoop.hdfs.web.WebHdfsFileSystem	Web-mode action file class settings
25	Fs.ftp.impl	Org.apache.hadoop.fs.ftp.FTPFileSystem	FTP File Action class settings
26	Fs.ramfs.impl	Org.apache.hadoop.fs.InMemoryFileSystem	Memory File Action class settings
27	Fs.har.impl	Org.apache.hadoop.fs.HarFileSystem	Compressed file operation class settings.
28	Fs.har.impl.disable.cache	True	Whether to cache identity settings for har files
29	Fs.checkpoint.dir	${hadoop.tmp.dir}/dfs/namesecondary	Store current record settings for backup name node
30	Fs.checkpoint.edits.dir	${fs.checkpoint.dir}	Backup name node log file storage current record settings
31	Fs.checkpoint.period	3600	Time interval settings for dynamic checking
32	Fs.checkpoint.size