Hadoop, HBase, Hive, zookeeper default port description
Component
Daemon
Port
Configuration
Description
Hdfs
DataNode
50010
Dfs.datanode.address
Datanode service port for data transfer
50075
Dfs.datanode.http.address
Port for HTTP Service
50475
Dfs.datanode.https.address
Ports for HTTPS servic
In the last two days, we have set up a Linux big data processing environment for a project. The system is centos 6.3. I mainly configure JDK, install Tomcat, hadoop, hbase, and zookeeper software. I am also a newbie in hadoop. I have encountered many problems in configuring this environment and checked a lot of information, here is a summary for future review.
F
[:nnnnn], blue part
The follower is used to connect to the leader and only listens on the leader on the port.
3888
/etc/zookeeper/conf/zoo.cfg in server.x=[hostname]:nnnnn[:nnnnn], blue part
Used for the leader election. Required only if ELECTIONALG is 3 (default).
All port protocols are based on TCP.For all Hadoop daemon that exist in the Web UI (HTTP servi
Articles on the establishment of the Hadoop + Zookeeper + HBase platform on the Internet can be said to be overwhelming and numerous. Here we will publish a very basic document left behind when we configure this platform, for your reference, you are welcome to raise any questions.
Start with the most basic configuration of the Hadoop platform.
L first, configure
Introduction
Recently, with the need for scientific research, Hadoop clusters have been built from scratch, including separate zookeeper and HBase.
For Linux, Hadoop and other related basic knowledge is relatively small, so this series of sharing applies to a variety of small white, want to experience the Hadoop clust
The recent time to build up a bit hadoop-2.7.3 + hbase-1.3.1 + zookeeper-3.4.8 + hive-2.3.0 fully distributed platform environment, online query a lot of relevant information, installation success, deliberately recorded down for reference.
first, software preparation
VMware12, hadoop-2.7.3, hbase-1.3.1, zookeeper-3.
Environment Description
1, operating system CentOS 6.5
2, jdk-7u51-linux-x64.tar.gz
Hadoop-1.1.2.tar.gz
Hbase-0.94.7-security.tar.gz
zookeeper-3.4.5.tar.gz Setting the IP address
Set static IP
Perform
Vim/etc/sysconfig/network-scripts/ifcfg-eth0
device= "eth0"
bootproto= "static"
onboot= "yes"
Type= "Ethernet"
ipaddr= "192.168.40.137"
prefix= "gateway=" "192.168.40.2"
No Internet access after setting s
the Hadoop middleware series which opened the zookeeper this default port to occupy.Not sure, so it's not good to kill this port, so you can only modify the default port:Modify in hbase-site.xmlMake 2181 into 2182
Post-execution resultsInformation display
[Root@master bin]#/start-hbase.sh
starting master, logging To/home/hbase/logs/hbase-root-master-master.out
Java HotSpot (TM) 64-bit Server VM warning:
the Zookeeper directory Copy this path, and then go to config file to modify this, and the rest do not need to be modified After the configuration is complete, start zookeeper, and in the Zookeeper directory, execute the command: bin/zkserver.sh start View zookeeper status can be seen
Not much to say, directly on the dry goods! 1, start each machine zookeeper (bigdata-pro01.kfk.com, bigdata-pro02.kfk.com, bigdata-pro03.kfk.com)2, start the ZKFC (bigdata-pro01.kfk.com)[Email protected] hadoop-2.6.0]$ pwd/opt/modules/hadoop-2.6.0[Email protected] hadoop-2.6.0]$ sbin/
$./hbase shell2018-07-08 12:17:44,820 WARN [main] util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where APPLICABLESLF 4j:class path contains multiple slf4j bindings. Slf4j:found Binding in [jar:file:/users/chong/opt/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/ Staticloggerbinder.class]slf4j:found Binding in [jar:file:/users/chong/opt/hadoop
Note: My public key file is not under/root/hxsyl under/home/hxsyl/.ssh,Find/-name Id_rsaFind1.Run the command on each machine zkserver.sh start or run in the $zookeeper_home/bin directory./zkserver.sh start command. You can then use the command JPS to view zookeeper-initiated process Quorumpeermain.The zookeeper status can be viewed through the zkserver.sh status command. Normally there is only one leader i
Zookeeper is an open source distributed Coordination Service , created by Yahoo, and is an open-source implementation of Google Chubby . Distributed applications can implement functions such as data Publishing/subscriptions, load balancing, naming services, distributed coordination/notification, cluster Management, master elections, distributed locks, and distributed queues based on zookeeper. Brief Introdu
http://blog.csdn.net/zhaonanemail/article/details/6654558
Here are a few key relationships:
1. The results of the map and reduce operations appear to be written to hbase, but in fact HBase Hlog and storefile files in flush to disk operations, these two files are stored in HDFs datanode. HDFs is the permanent store.
What does 2.ZooKeeper have to do with Hadoop Core and hbase? What services are provided by
There are several main relationships:
1. the results produced after map and reduce operations seem to have been written to hbase. However, when the files in hlog and storefile of hbase are flush to disk, these two files are stored in the HDFS datanode, and HDFS is permanently stored.
2. What is the relationship between zookeeper and hadoop core and hbase? What services does
There was an article in detail about how to install Hadoop+hbase+zookeeper
The title of the article is: Hadoop+hbase+zookeeper distributed cluster construction perfect operation
Its website: http://blog.csdn.net/shatelang/article/details/7605939
This article is about hadoop1.0.0+hbase0.92.1+zookeeper3.3.4.
The installa
with no intermediate state.6, Sequential: For all servers, the same message is published in a consistent order.Basic principle650) this.width=650; "Src=" Http://s1.51cto.com/wyfs02/M00/85/4C/wKiom1efTNnA4ZCeAAX4DF7vo0w159.png-wh_500x0-wm_3 -wmp_4-s_1223101739.png "title=" Zookeeper2. PNG "alt=" Wkiom1eftnna4zceaax4df7vo0w159.png-wh_50 "/>Server many, there are master and slave points, but there is a leader, the other is follower,Each server, in memory, holds a piece of data that, when launched,
The distributed architecture is a centralized design.Master control machines connected to multiple processing nodesTherefore, it is critical to ensure the high availability of the master machine. Distributed locks are a good solution to this problem.Zookeeper is a Distributed Lock Management System for Highly Reliable metadata maintenance..
I. Application
1. Cluster Mode
In cluster mode, multiple zookeeper nodes are configured to start the
The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion;
products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the
content of the page makes you feel confusing, please write us an email, we will handle the problem
within 5 days after receiving your email.
If you find any instances of plagiarism from the community, please send an email to:
info-contact@alibabacloud.com
and provide relevant evidence. A staff member will contact you within 5 working days.