Hadoop configuration item organization (core-site.xml)

Record the hadoop configuration and description. New configuration items are added and occasionally updated. By configuration file name

Take hadoop 1. x configuration as an Example

Core-site.xml

 

Name	Value	Description
FS. Default. Name	HDFS: // hadoopmaster: 9000	Define the URI and port of hadoopmaster
FS. Checkpoint. dir	/Opt/data/hadoop1/HDFS/namesecondary1	Defines the path for hadoop name backup. The official document says it is read and written to DFS. Name. dir.
FS. Checkpoint. Period	1800	Defines the backup interval of name backup, in seconds. It takes effect only for SNN. The default value is one hour.
FS. Checkpoint. Size	33554432	Backup interval by log size interval, effective only for SNN, default 64 m
Io. Compression. codecs	Org. Apache. hadoop. Io. Compress. defaultcodec, Com. hadoop. Compression. lzo. lzocodec, Com. hadoop. Compression. lzo. lzopcodec, Org. Apache. hadoop. Io. Compress. gzipcodec, Org. Apache. hadoop. Io. Compress. bzip2codec (For typographical adjustment, do not press ENTER for actual configuration)	The codecs used by hadoop. gzip and Bzip2 are built-in. The lzo must be installed with hadoopgpl or kevinweil, separated by commas (,), and snappy must also be installed separately.
Io. Compression. codec. lzo. Class	Com. hadoop. Compression. lzo. lzocodec	Compression encoder used by lzo
Topology. Script. file. Name	/Hadoop/bin/rackaware. py	Rack-aware script location
Topology. Script. Number. ARGs	1000	Rack-aware script-managed hosts, IP addresses
FS. Trash. Interval	10800	HDFS garbage bin settings, can be restored to accidental deletion, in minutes, 0 is disabled, add this item without restarting hadoop
Hadoop. http. Filter. initializers	Org. Apache. hadoop. Security. Authenticationfilterinitializer (For typographical adjustment, do not press ENTER for actual configuration)	Jobtracker and tasktracker are required. Users of HTTP access ports such as namenode and datanode must configure all nodes.
Hadoop. http. Authentication. Type	Simple | Kerberos | # authentication_handler_classname #	Verification method. The default value is simple. You can also define your own class. You need to configure all nodes.
Hadoop. http. Authentication. Token. Validity (For typographical adjustment, do not press ENTER for actual configuration)	36000	The validity period of the token. You must configure all nodes.
Hadoop. http. Authentication. Signature. Secret (For typographical adjustment, do not press ENTER for actual configuration)	Parameter unspecified by default	By default, private signatures are automatically generated when hadoop is started. You need to configure all nodes.
Hadoop. http. Authentication. Cookie. Domain	Domian. TLD	The domain name of the cookie used for HTTP verification. If the IP address is accessed, this item is invalid. You must configure a domain name for all nodes.
Hadoop. http. Authentication. Simple. Anonymous. Allowed (For typographical adjustment, do not press ENTER for actual configuration)	True | false	Dedicated for simple verification. Anonymous access is allowed by default. True
Hadoop. http. Authentication. Kerberos. Principal (For typographical adjustment, do not press ENTER for actual configuration)	HTTP/[email protected] $ localhost	Dedicated for Kerberos authentication. The authenticated entity must use HTTP as the K name
Hadoop. http. Authentication. Kerberos. keytab (For typographical adjustment, do not press ENTER for actual configuration)	/Home/xianglei/hadoop. keytab	Dedicated for Kerberos authentication, key file storage location
Hadoop. Security. Authorization	True | false	Hadoop service level authentication security verification, must be used with the hadoop-policy.xml, configured with dfsadmin, mradmin-refreshserviceacl refresh effective
Io. file. Buffer. Size	131072	Used as the buffer size for reading and writing serialized files
Hadoop. Security. Authentication	Simple | Kerberos	Permission verification for hadoop itself, non-HTTP access, simple or Kerberos
Hadoop. logfile. Size	1000000000	Set the log file size. If the size exceeds the limit, the new log is rolled.
Hadoop. logfile. Count	20	Maximum number of logs
Io. bytes. Per. checksum	1024	The number of bytes verified by each verification code. Do not exceed Io. file. Buffer. Size.
Io. Skip. checksum. Errors	True | false	When processing serialized files, skip the verification code error without throwing an exception. Default Value: false.
Io. serializations	Org. Apache. hadoop. Io. Serializer. writableserialization (Typographical needs. Do not press ENTER for actual configuration)	Serialized codecs
Io. seqfile. Compress. blocksize	1024000	The minimum block size, in bytes, of the serialized file.
Webinterface. Private. Actions	True | false	If this parameter is set to true, operations such as killing tasks and deleting files will be connected to the tracker page of JT and nn. The default value is false.

 

Based on the Apache manual and the actual configuration used in the company, the actual parameter size needs to be adjusted based on different hardware configurations. Currently, the parameter is based on namenode 96 GB memory and datanode 32 GB memory. The Implement of FS such as har, S3, and local is not used, so it is not written.

Hadoop configuration item organization (core-site.xml)