Build a Hadoop Client-that is, access Hadoop from hosts outside the Cluster

Build a Hadoop Client-that is, access Hadoop from hosts outside the Cluster

Build a Hadoop Client-that is, access Hadoop from hosts outside the Cluster

1. Add host ing (the same as namenode ing ):

Add the last line

[Root @ localhost ~] # Su-root

[Root @ localhost ~] # Vi/etc/hosts
127.0.0.1 localhost. localdomain localhost4 localhost4.localdomain4
: 1 localhost. localdomain localhost6 localhost6.localdomain6
192.168.48.129 hadoop-master
[Root @ localhost ~] #

2. Create a hadoop user

Create a hadoop User Group

Create a user, useradd-d/usr/hadoop-g hadoop-m hadoop (create a user, hadoop, specify the user's main directory, usr/hadoop, and hadoop Group)

Set the hadoop password for passwd hadoop (set the password to hadoop here)

[Root @ localhost ~] # Groupadd hadoop
[Root @ localhost ~] # Useradd-d/usr/hadoop-g hadoop-m hadoop
[Root @ localhost ~] # Passwd hadoop

3. Configure the jdk Environment

This installation is the hadoop-2.7.5, JDK 7 or above. Skip this step if it is already installed.

For JDK installation, refer to: Or CentOS7.2 to install JDK 1.7.

Or directly copying JDK files on the master node is more conducive to maintaining version consistency.

[Root @ localhost java] # su-root
[Root @ localhost java] # mkdir-p/usr/java
[Root @ localhost java] # scp-r hadoop @ hadoop-master:/usr/java/jdk1.7.0 _ 79/usr/java
[Root @ localhost java] # ll
Total 12
Drwxr-xr-x. 8 root 4096 Feb 13 0:34 default
Drwxr-xr-x. 8 root 4096 Feb 13 0:34 jdk1.7.0 _ 79
Drwxr-xr-x. 8 root 4096 Feb 13 0:34 latest

Set Java and hadoop Environment Variables

Ensure that/usr/java/jdk1.7.0.79 exists

Su-root

Vi/etc/profile

Ensure that/usr/java/jdk1.7.0.79 exists

Unset I
Unset-f pathmunge
JAVA_HOME =/usr/java/jdk1.7.0 _ 79
CLASSPATH =.: $ JAVA_HOME/lib/dt. jar: $ JAVA_HOME/lib/tools. jar
PATH =/usr/hadoop/hadoop-2.7.5/bin: $ JAVA_HOME/bin: $ PATH

Settings take effect (important)

[Root @ localhost ~] # Source/etc/profile
[Root @ localhost ~] #

After JDK is installed, confirm:

[Hadoop @ localhost ~] $ Java-version
Java version "1.7.0 _ 79"
Java (TM) SE Runtime Environment (build 1.7.0 _ 79-b15)
Java HotSpot (TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
[Hadoop @ localhost ~] $

4. Set hadoop Environment Variables

Copy the configured hadoop directory on namenode to the current host.

[Root @ localhost ~] # Su-hadoop
Last login: Sat Feb 24 14:04:55 CST 2018 on pts/1
[Hadoop @ localhost ~] $ Pwd
/Usr/hadoop
[Hadoop @ localhost ~] $ Scp-r hadoop @ hadoop-master:/usr/hadoop/hadoop-2.7.5.
The authenticity of host 'hadoop-master (192.168.48.129) 'can't be established.
ECDSA key fingerprint is 1e: cd: d1: 3d: b0: 5b: 62: 45: a3: 63: df: c7: 7a: 0f: b8: 7c.
Are you sure you want to continue connecting (yes/no )? Yes
Warning: Permanently added 'hadoop-master, 192.168.48.129 '(ECDSA) to the list of known hosts.
Hadoop @ hadoop-master's password:

[Hadoop @ localhost ~] $ Ll
Total 0
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Desktop
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Documents
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Downloads
Drwxr-xr-x 10 hadoop 150 Feb 24 hadoop-2.7.5
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Music
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Pictures
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Public
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Templates
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Videos
[Hadoop @ localhost ~] $

At this point, the Hadoop Client installation is complete and can be used later.

The result of executing the hadoop command is as follows,

[Hadoop @ localhost ~] $ Hadoop
Usage: hadoop [-- config confdir] [COMMAND | CLASSNAME]
CLASSNAME run the class named CLASSNAME
Or
Where COMMAND is one:
Fs run a generic filesystem user client
Version print the version
Jar <jar> run a jar file
Note: please use "yarn jar" to launch
YARN applications, not this command.
Checknative [-a |-h] check native hadoop and compression libraries availability
Distcp <srcurl> <desturl> copy file or directories recursively
Archive-archiveName NAME-p <parent path> <src> * <dest> create a hadoop archive
Classpath prints the class path needed to get
Credential interact with credential providers
Hadoop jar and the required libraries
Daemonlog get/set the log level for each daemon
Trace view and modify Hadoop tracing settings

Most commands print help when invoked w/o parameters.
[Hadoop @ localhost ~] $

5. Use hadoop

Create a local file

[Hadoop @ localhost ~] $ Hdfs dfs-ls
Found 1 items
Drwxr-xr-x-hadoop supergroup 0 2018-02-22 23:41 output
[Hadoop @ localhost ~] $ Vi my-local.txt
Hello boy!
Yehyeh

Upload local files to the Cluster

[Hadoop @ localhost ~] $ Hdfs dfs-mkdir upload
[Hadoop @ localhost ~] $ Hdfs dfs-ls upload
[Hadoop @ localhost ~] $ Hdfs dfs-ls
Found 2 items
Drwxr-xr-x-hadoop supergroup 0 2018-02-22 23:41 output
Drwxr-xr-x-hadoop supergroup 0 2018-02-23 22:38 upload
[Hadoop @ localhost ~] $ Hdfs dfs-ls upload
[Hadoop @ localhost ~] $ Hdfs dfs-put my-local.txt upload
[Hadoop @ localhost ~] $ Hdfs dfs-ls upload
Found 1 items
-Rw-r -- 3 hadoop supergroup 18 2018-02-23 upload/my-local.txt
[Hadoop @ localhost ~] $ Hdfs dfs-cat upload/my-local.txt
Hello boy!
Yehyeh
[Hadoop @ localhost ~] $

Ps: note that the local java version and the master copy of the file/etc/hadoop-env.sh configuration JAVA_HOME to be consistent have not been verified, this article is consistent.

Hadoop2.3-HA high-availability cluster environment construction

Hadoop project-Cloudera 5.10.1 (CDH) installation and deployment based on CentOS7

Detailed explanation of Hadoop2.7.2 cluster construction (high availability)

Use Ambari to deploy a Hadoop cluster (build an intranet HDP source)

Hadoop cluster installation in Ubuntu 14.04

Install Hadoop 2.7.2 In CentOS 6.7

Build a distributed Hadoop-2.7.3 cluster on Ubuntu 16.04

Build a Hadoop 2.6.4 distributed Cluster Environment in CentOS 7

Hadoop2.7.3 + Spark2.1.0 fully distributed cluster Construction Process