Hadoop, HBase, Hive, zookeeper default port description

Hadoop, HBase, Hive, zookeeper default port description

Component	Daemon	Port	Configuration	Description
Hdfs	DataNode	50010	Dfs.datanode.address	Datanode service port for data transfer
		50075	Dfs.datanode.http.address	Port for HTTP Service
		50475	Dfs.datanode.https.address	Ports for HTTPS services
		50020	Dfs.datanode.ipc.address	Ports for IPC Services
	NameNode	50070	Dfs.namenode.http-address	Port for HTTP Service
		50470	Dfs.namenode.https-address	Ports for HTTPS services
		8020	Fs.defaultfs	The RPC port that receives the client connection for obtaining file system metadata information.
	Journalnode	8485	Dfs.journalnode.rpc-address	RPC Service
		8480	Dfs.journalnode.http-address	HTTP Service
	Zkfc	8019	Dfs.ha.zkfc.port	ZooKeeper Failovercontroller, for nn HA
YARN	ResourceManager	8032	Yarn.resourcemanager.address	RM Applications Manager (ASM) port
		8030	Yarn.resourcemanager.scheduler.address	IPC Port for Scheduler components
		8031	Yarn.resourcemanager.resource-tracker.address	Ipc
		8033	Yarn.resourcemanager.admin.address	Ipc
		8088	Yarn.resourcemanager.webapp.address	HTTP Service Port
	NodeManager	8040	Yarn.nodemanager.localizer.address	Localizer IPC
		8042	Yarn.nodemanager.webapp.address	HTTP Service Port
		8041	Yarn.nodemanager.address	Port of container Manager in NM
	Jobhistory Server	10020	Mapreduce.jobhistory.address	Ipc
		19888	Mapreduce.jobhistory.webapp.address	HTTP Service Port
HBase	Master	60000	Hbase.master.port	Ipc
		60010	Hbase.master.info.port	HTTP Service Port
	Regionserver	60020	Hbase.regionserver.port	Ipc
		60030	Hbase.regionserver.info.port	HTTP Service Port
	Hquorumpeer	2181	Hbase.zookeeper.property.clientPort	Hbase-managed ZK mode, using a standalone zookeeper cluster will not enable the port.
		2888	Hbase.zookeeper.peerport	Hbase-managed ZK mode, using a standalone zookeeper cluster will not enable the port.
		3888	Hbase.zookeeper.leaderport	Hbase-managed ZK mode, using a standalone zookeeper cluster will not enable the port.
Hive	Metastore	9083	/etc/default/hive-metastore export port=<port> to update the default port
	Hiveserver	10000	/ETC/HIVE/CONF/HIVE-ENV.SH export hive_server2_thrift_port=<port> to update the default port
ZooKeeper	Server	2181	/etc/zookeeper/conf/zoo.cfg in Clientport=<port>	Ports that serve clients
		2888	/etc/zookeeper/conf/zoo.cfg in server.x=[hostname]:nnnnn[:nnnnn], blue part	The follower is used to connect to the leader and only listens on the leader on the port.
		3888	/etc/zookeeper/conf/zoo.cfg in server.x=[hostname]:nnnnn[:nnnnn], blue part	Used for the leader election. Required only if ELECTIONALG is 3 (default).

All port protocols are based on TCP.

For all Hadoop daemon that exist in the Web UI (HTTP service), there are URLs like:

/logs
List of log files for downloading and viewing

/loglevel
Allows you to set the log4j logging level, similar to Hadoop daemonlog

/stacks
stack trace for all threads, very helpful for debug

/jmx
Server-side metrics, output in JSON format.

/jmx?qry=hadoop:* will return all Hadoop-related metrics.
The/jmx?get=mxbeanname::attributename query specifies the value of the specified bean property, such as/jmx?get=hadoop:service=namenode,name=namenodeinfo:: Clusterid will return to Clusterid.
The processing class for this request: Org.apache.hadoop.jmx.JMXJsonServlet

Specific daemon also have specific URL path-specific information.

NameNode: http://:50070/

/dfshealth.jsp
HDFs Information page, which has links to view file systems

/dfsnodelist.jsp?whatnodes= (dead| LIVE)
Show dead or live status of Datanode

DataNode: http://:50075/

/blockscannerreport
Each datanode will specify interval validation block information

/fsck
Running the fsck command is not recommended for use when the cluster is busy.

In Hadoop 2.2.0, the yarn framework has a lot of default parameter values, and if you are in a situation where the machine resources are not sufficient, you need to modify these defaults to meet some of the task needs.
Both NodeManager and ResourceManager are configured in the Yarn-site.xml file, and when the MapReduce task is run, it is configured in Mapred-site.xml.
Here's a look at the relevant parameters and their default values:

Parameter name	Default Value	Process Name	configuration file	meaning Description
Yarn.nodemanager.resource.memory-mb	8192	NodeManager	Yarn-site.xml	Total amount of physical memory available from the physical host where the node resides
Yarn.nodemanager.resource.cpu-vcores	8	NodeManager	Yarn-site.xml	Total available virtual CPU resources for the physical host on which the node resides (CORE)
Yarn.nodemanager.vmem-pmem-ratio	2.1	NodeManager	Yarn-site.xml	Maximum amount of virtual memory that can be used with 1M of physical memory
Yarn.scheduler.minimum-allocation-mb	1024	ResourceManager	Yarn-site.xml	Minimum number of allocated memory resources for one application
Yarn.scheduler.maximum-allocation-mb	8192	ResourceManager	Yarn-site.xml	Maximum number of allocated memory resources for one application
Yarn.scheduler.minimum-allocation-vcores	1	ResourceManager	Yarn-site.xml	Minimum number of virtual CPU resources allocated for one application
Yarn.scheduler.maximum-allocation-vcores	8	ResourceManager	Yarn-site.xml	Maximum number of virtual CPU resources allocated for one application
Mapreduce.framework.name	Local	Mapreduce	Mapred-site.xml	Value one of the local, classic, or yarn, if not yarn, does not use yarn clusters for resource allocation
Mapreduce.map.memory.mb	1024	Mapreduce	Mapred-site.xml	The number of memory resources that can be requested by the map task for each MapReduce job
Mapreduce.map.cpu.vcores	1	Mapreduce	Mapred-site.xml	The number of virtual CPU resources that can be requested by the map task for each MapReduce job
Mapreduce.reduce.memory.mb	1024	Mapreduce	Mapred-site.xml	The number of memory resources that can be requested by the reduce task per mapreduce job
Yarn.nodemanager.resource.cpu-vcores	8	Mapreduce	Mapred-site.xml	The number of virtual CPU resources that can be requested by the reduce task for each mapreduce job