CDH version of the Hue installation configuration deployment and integrated Hadoop hbase hive MySQL and other authoritative guidance

Hue: Https://github.com/cloudera/hue

Hue Study document address : http://archive.cloudera.com/cdh5/cdh/5/hue-3.7.0-cdh5.3.6/manual.html

I'm currently using hue-3.7.0-cdh5.3.6.

Hue (Hue=hadoop User Experience)

Hue is an open-source Apache Hadoop UI system that evolved from Cloudera desktop and finally cloudera the company's contribution to the Apache Foundation's Hadoop community, which is based on the Python web framework Django implementation.

By using hue we can interact with the Hadoop cluster on the browser-side Web console to analyze processing data, such as manipulating data on HDFs, running the MapReduce Job, executing the SQL statements of hive, browsing the HBase database, and so on.

Hue Features:

Ability to support various versions of Hadoop
Hue default database: SQL Lite
File browser: Adding and pruning data
Hue download src package to compile once, two compile, in this is already compiled once

Hue deployment:

1. Download Dependency Package: Yum Source installation

sudo yum-y install ant asciidoc cyrus-sasl-devel cyrus-sasl-gssapi gcc gcc-c++ krb5-devel libtidy libxml2-devel libxslt-d Evel mvn MySQL mysql-devel openldap-devel python-devel sqlite-devel openssl-devel　

sudo yum install ant asciidoc cyrus-sasl-devel cyrus-sasl-gssapi cyrus-sasl-plain gcc gcc-c++ krb5-devel libffi-devel libx Ml2-devel libxslt-devel make MySQL mysql-devel openldap-devel python-devel sqlite-devel gmp-devel

2. Unzip the Hue tar package

　TAR-ZXVF hue-3.7.0-cdh5.3.6.tar.gz-c/specified directory

3, two times compilation

　　Enter the Hue directory: Execute make apps will appear with a build directory

Error: (CentOS 7 will be available)
Error:static declaration of ' x509_revoked_dup ' follows non-static declaration
Static x509_revoked * X509_revoked_dup (x509_revoked *orig) {
^
In file included from/usr/include/openssl/ssl.h:156:0,
From Openssl/crypto/x509.h:17,
From Openssl/crypto/crypto.h:30,
From Openssl/crypto/crl.c:3:
/usr/include/openssl/x509.h:751:15:note:previous declaration of ' X509_revoked_dup ' is here
X509_revoked *x509_revoked_dup (x509_revoked *rev);
^
Error:command ' gcc ' failed with exit status 1

To the following two delete:/usr/include/openssl/x509.h-"751, 752 lines
X509_revoked *x509_revoked_dup (x509_revoked *rev);
X509_req *x509_req_dup (X509_req *req);
# #必须删掉, Comment No

4. Enter to Hue-3.7.0-cdh5.3.6/desktop/conf

To configure the Hue.ini file:

Secret_key=jfe93j;2[290-eiw. keiwn2s3[' d;/.q[eiw^y#e=+iei* @Mn <qw5o
Http_host=hadoop01.xningge.com
http_port=8888
Time_zone=asia/shanghai

5. Start Hue

Two different ways

1-->CD build/env/bin---"./supervisor
2-->build/env/bin/supervisor

6. Browser Access hue

Host name +8888
Create a user name and password

component configurations for hue and Hadoop

1, the configuration of HDFs

　　Configuring in Hdfs-site.xml

<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>

　　Configuring in Core-site.xml

　　

<property>
<name>hadoop.proxyuser.hue.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hue.groups</name>
<value>*</value>
</property>

2. Restart the HDFS process

　　sbin/start-yarn.sh

3. Hue Configuration

　　[[Hdfs_clusters]]
# HA support by using HTTPFS

[[[[Default]]]
# Enter the filesystem URI
fs_defaultfs=hdfs://hadoop01.xningge.com:8020

# NameNode logical name.
# # Logical_name=

# use WEBHDFS/HTTPFS as the communication mechanism.
# Domain should be the NameNode or HTTPFS host.
# Default Port is 14000 for HTTPFS.
webhdfs_url=http://hadoop01.xningge.com:50070/webhdfs/v1

# This is the home of your Hadoop HDFS installation
hadoop_hdfs_home=/opt/modules/cdh/hadoop-2.5.0-cdh5.3.6

# Use this as the HDFS Hadoop launcher script
Hadoop_bin=/opt/modules/cdh/hadoop-2.5.0-cdh5.3.6/bin

# Change the If your HDFS cluster is kerberos-secured
# # Security_enabled=false

# Default Umask for file and directory creation, specified with an octal value.
# # UMASK=022

# Directory of the Hadoop configuration
Hadoop_conf_dir=/opt/modules/cdh/hadoop-2.5.0-cdh5.3.6/etc/hadoop

[[Yarn_clusters]]

[[[[Default]]]
# Enter the host on which running the ResourceManager
resourcemanager_host=hadoop01.xningge.com

# The port where the ResourceManager IPC listens on
resourcemanager_port=8032

# Whether to the submit jobs to this cluster
Submit_to=true

# Resource Manager Logical name (required for HA)
# # Logical_name=

# Change the If your YARN cluster is kerberos-secured
# # Security_enabled=false

# URL of the ResourceManager API
resourcemanager_api_url=http://hadoop01.xningge.com:8088

# URL of the ProxyServer API
proxy_api_url=http://hadoop01.xningge.com:8088

# URL of the Historyserver API
history_server_api_url=http://hadoop01.xningge.com:19888

Eg: This configuration is pseudo-distributed mode

4. Start Hue Service

　　Build/env/bin/supervisor

hue and Hive configuration

1. Hive Configuration

　　In Hive-site.xml configuration

<property>
<name>hive.server2.thrift.bind.host</name>
<value>hadoop01.xningge.com</value>
</property>
<property>
<name>hive.metastore.uris</name>
<value>hadoop01.xningge.com:9083</value>
</property>

2. Start Hive Service

　　Bin/hiveserver2 &

　　Bin/hive--service Metastore &

3. Hue Configuration

　　Modify the Hue.ini file

[Beeswax]

# Host where HiveServer2 is running.
# If Kerberos security is enabled, use fully-qualified domain name (FQDN).
hive_server_host=hadoop01.xningge.com

# Port where HiveServer2 Thrift server runs on.
hive_server_port=10000

# Hive configuration directory, where Hive-site.xml is located
hive_conf_dir=/opt/modules/cdh/hive-0.13.1-cdh5.3.6/conf

# Timeout in seconds-Thrift calls to Hive service
server_conn_timeout=120

# Choose Whether Hue uses the GetLog () Thrift call to retrieve Hive logs.
# If False, Hue would use the fetchresults () thrift call instead.
# # Use_get_log_api=true

# Set a LIMIT clause when browsing a partitioned table.
# A Positive value would be set as the LIMIT. If 0 or negative, do not set any limit.
# # BROWSE_PARTITIONED_TABLE_LIMIT=250

# A limit to the number of rows that can is downloaded from A query.
# A value of-1 means there'll be no limit.
# A maximum of 65,000 is applied to XLS downloads.
# # download_row_limit=1000000

# Hue would try to close the Hive query when the user leaves the editor page.
# This would free all the "query resources in HiveServer2", but also make its results inaccessible.
# # Close_queries=false

# Thrift version to use when communicating with HiveServer2
# # Thrift_version=5

hue and relational database configuration

[Librdbms]
# The RDBMS app can has any number of databases configured in the databases
# section. A database is known by it section name
# (IE sqlite, MySQL, Psql, and Oracle in the list below).

[[Databases]]
# SQLite configuration.
[[[SQLite]]//Note that you must uncomment it here
# Name to show in the UI.
Nice_name=sqlite

# for SQLite, name defines the path to the database.
name=/opt/modules/hue-3.7.0-cdh5.3.6/desktop/desktop.db

# Database backend to use.
Engine=sqlite

# Database options to send to the server when connecting.
# https://docs.djangoproject.com/en/1.4/ref/databases/
# # options={}

# MySQL, Oracle, or PostgreSQL configuration.

　　# #注意: The data here cannot be changed, the default is the database of Hue

[[[MySQL]]//Note that you must uncomment this
# Name to show in the UI.
nice_name= "My SQL DB"

# for MySQL and PostgreSQL, name is the name of the database.
# for Oracle, Name is instance of the Oracle server. For Express Edition
# This was ' XE ' by default.
name=sqoop//This name is the database table name

# Database backend to use. This can is:
# 1. Mysql
# 2. PostgreSQL
# 3. Oracle
Engine=mysql

# IP or hostname of the database to connect to.
host=hadoop01.xningge.com

# Port The database server is listening to. Defaults is:
# 1. mysql:3306
# 2. postgresql:5432
# 3. Oracle Express edition:1521
port=3306

# Username to authenticate with when connecting to the database.
User=xningge

# Password matching the username to authenticate and when
# Connecting to the database.
password=???

# Database options to send to the server when connecting.
# https://docs.djangoproject.com/en/1.4/ref/databases/
# # options={}

Hue and Zookeeper Configuration

Just modify the Hue.ini file
　　host_ports=hadoop01.xningge.com:2181
Start Zookeeper:

hue and Oozie configuration

Modified: Hue.ini File
[Liboozie]
Oozie_url=http://hadoop01.xningge.com:11000/oozie

If not out of:
　　Modified: Oozie-site.xml
　　<property>
<name>oozie.service.WorkflowAppService.system.libpath</name>
<value>/user/oozie/share/lib</value>
</property>

Re-create the Sharelib library under the Oozie directory:
　　bin/oozie-setup.sh sharelib Create-fs Hdfs://hadoop01.xningge.com:8020-locallib Oozie-sharelib-4.0.0-cdh5.3.6-yarn.tar.gz
Start Oozie:bin/oozied.sh start

hue vs. HBase configuration

To Modify the Hue.ini file:
hbase_clusters= (cluster|hadoop01.xningge.com:9090)
hbase_conf_dir=/opt/cdh_5.3.6/hbase-0.98.6-cdh5.3.6/conf
Modify Hbase-site.xml To add the following configuration:
<property>
<name>hbase.regionserver.thrift.http</name>
<value>true</value>
</property>
<property>
<name>hbase.thrift.support.proxyuser</name>
<value>true</value>
</property>
Start HBase:
bin/start-hbase.sh
bin/hbase-daemon.sh Start Thrift

# #hbase完全分布式
hbase_clusters= (cluster1|hostname:9090,cluster2|hostname:9090,cluster3|hostname:9090)

CDH version of the Hue installation configuration deployment and integrated Hadoop hbase hive MySQL and other authoritative guidelines