Build a Hadoop Client-that is, access Hadoop from hosts outside the Cluster
Build a Hadoop Client-that is, access Hadoop from hosts outside the Cluster
1. Add host ing (the same as namenode ing ):
Add the last line
[Root @ localhost ~] # Su-root
[Root @ localhost ~] # Vi/etc/hosts
127.0.0.1 localhost. localdomain localhost4 localhost4.localdomain4
: 1 localhost. localdomain localhost6 localhost6.localdomain6
192.168.48.129 hadoop-master
[Root @ localhost ~] #
2. Create a hadoop user
Create a hadoop User Group
Create a user, useradd-d/usr/hadoop-g hadoop-m hadoop (create a user, hadoop, specify the user's main directory, usr/hadoop, and hadoop Group)
Set the hadoop password for passwd hadoop (set the password to hadoop here)
[Root @ localhost ~] # Groupadd hadoop
[Root @ localhost ~] # Useradd-d/usr/hadoop-g hadoop-m hadoop
[Root @ localhost ~] # Passwd hadoop
3. Configure the jdk Environment
This installation is the hadoop-2.7.5, JDK 7 or above. Skip this step if it is already installed.
For JDK installation, refer to: Or CentOS7.2 to install JDK 1.7.
Or directly copying JDK files on the master node is more conducive to maintaining version consistency.
[Root @ localhost java] # su-root
[Root @ localhost java] # mkdir-p/usr/java
[Root @ localhost java] # scp-r hadoop @ hadoop-master:/usr/java/jdk1.7.0 _ 79/usr/java
[Root @ localhost java] # ll
Total 12
Drwxr-xr-x. 8 root 4096 Feb 13 0:34 default
Drwxr-xr-x. 8 root 4096 Feb 13 0:34 jdk1.7.0 _ 79
Drwxr-xr-x. 8 root 4096 Feb 13 0:34 latest
Set Java and hadoop Environment Variables
Ensure that/usr/java/jdk1.7.0.79 exists
Su-root
Vi/etc/profile
Ensure that/usr/java/jdk1.7.0.79 exists
Unset I
Unset-f pathmunge
JAVA_HOME =/usr/java/jdk1.7.0 _ 79
CLASSPATH =.: $ JAVA_HOME/lib/dt. jar: $ JAVA_HOME/lib/tools. jar
PATH =/usr/hadoop/hadoop-2.7.5/bin: $ JAVA_HOME/bin: $ PATH
Settings take effect (important)
[Root @ localhost ~] # Source/etc/profile
[Root @ localhost ~] #
After JDK is installed, confirm:
[Hadoop @ localhost ~] $ Java-version
Java version "1.7.0 _ 79"
Java (TM) SE Runtime Environment (build 1.7.0 _ 79-b15)
Java HotSpot (TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
[Hadoop @ localhost ~] $
4. Set hadoop Environment Variables
Copy the configured hadoop directory on namenode to the current host.
[Root @ localhost ~] # Su-hadoop
Last login: Sat Feb 24 14:04:55 CST 2018 on pts/1
[Hadoop @ localhost ~] $ Pwd
/Usr/hadoop
[Hadoop @ localhost ~] $ Scp-r hadoop @ hadoop-master:/usr/hadoop/hadoop-2.7.5.
The authenticity of host 'hadoop-master (192.168.48.129) 'can't be established.
ECDSA key fingerprint is 1e: cd: d1: 3d: b0: 5b: 62: 45: a3: 63: df: c7: 7a: 0f: b8: 7c.
Are you sure you want to continue connecting (yes/no )? Yes
Warning: Permanently added 'hadoop-master, 192.168.48.129 '(ECDSA) to the list of known hosts.
Hadoop @ hadoop-master's password:
[Hadoop @ localhost ~] $ Ll
Total 0
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Desktop
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Documents
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Downloads
Drwxr-xr-x 10 hadoop 150 Feb 24 hadoop-2.7.5
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Music
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Pictures
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Public
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Templates
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Videos
[Hadoop @ localhost ~] $
At this point, the Hadoop Client installation is complete and can be used later.
The result of executing the hadoop command is as follows,
[Hadoop @ localhost ~] $ Hadoop
Usage: hadoop [-- config confdir] [COMMAND | CLASSNAME]
CLASSNAME run the class named CLASSNAME
Or
Where COMMAND is one:
Fs run a generic filesystem user client
Version print the version
Jar <jar> run a jar file
Note: please use "yarn jar" to launch
YARN applications, not this command.
Checknative [-a |-h] check native hadoop and compression libraries availability
Distcp <srcurl> <desturl> copy file or directories recursively
Archive-archiveName NAME-p <parent path> <src> * <dest> create a hadoop archive
Classpath prints the class path needed to get
Credential interact with credential providers
Hadoop jar and the required libraries
Daemonlog get/set the log level for each daemon
Trace view and modify Hadoop tracing settings
Most commands print help when invoked w/o parameters.
[Hadoop @ localhost ~] $
5. Use hadoop
Create a local file
[Hadoop @ localhost ~] $ Hdfs dfs-ls
Found 1 items
Drwxr-xr-x-hadoop supergroup 0 2018-02-22 23:41 output
[Hadoop @ localhost ~] $ Vi my-local.txt
Hello boy!
Yehyeh
Upload local files to the Cluster
[Hadoop @ localhost ~] $ Hdfs dfs-mkdir upload
[Hadoop @ localhost ~] $ Hdfs dfs-ls upload
[Hadoop @ localhost ~] $ Hdfs dfs-ls
Found 2 items
Drwxr-xr-x-hadoop supergroup 0 2018-02-22 23:41 output
Drwxr-xr-x-hadoop supergroup 0 2018-02-23 22:38 upload
[Hadoop @ localhost ~] $ Hdfs dfs-ls upload
[Hadoop @ localhost ~] $ Hdfs dfs-put my-local.txt upload
[Hadoop @ localhost ~] $ Hdfs dfs-ls upload
Found 1 items
-Rw-r -- 3 hadoop supergroup 18 2018-02-23 upload/my-local.txt
[Hadoop @ localhost ~] $ Hdfs dfs-cat upload/my-local.txt
Hello boy!
Yehyeh
[Hadoop @ localhost ~] $
Ps: note that the local java version and the master copy of the file/etc/hadoop-env.sh configuration JAVA_HOME to be consistent have not been verified, this article is consistent.
Hadoop2.3-HA high-availability cluster environment construction
Hadoop project-Cloudera 5.10.1 (CDH) installation and deployment based on CentOS7
Detailed explanation of Hadoop2.7.2 cluster construction (high availability)
Use Ambari to deploy a Hadoop cluster (build an intranet HDP source)
Hadoop cluster installation in Ubuntu 14.04
Install Hadoop 2.7.2 In CentOS 6.7
Build a distributed Hadoop-2.7.3 cluster on Ubuntu 16.04
Build a Hadoop 2.6.4 distributed Cluster Environment in CentOS 7
Hadoop2.7.3 + Spark2.1.0 fully distributed cluster Construction Process