Build a Hadoop Client-that is, access Hadoop from hosts outside the Cluster

Source: Internet
Author: User
Tags hdfs dfs

Build a Hadoop Client-that is, access Hadoop from hosts outside the Cluster

Build a Hadoop Client-that is, access Hadoop from hosts outside the Cluster

1. Add host ing (the same as namenode ing ):

Add the last line

[Root @ localhost ~] # Su-root

[Root @ localhost ~] # Vi/etc/hosts
127.0.0.1 localhost. localdomain localhost4 localhost4.localdomain4
: 1 localhost. localdomain localhost6 localhost6.localdomain6
192.168.48.129 hadoop-master
[Root @ localhost ~] #


2. Create a hadoop user

Create a hadoop User Group

Create a user, useradd-d/usr/hadoop-g hadoop-m hadoop (create a user, hadoop, specify the user's main directory, usr/hadoop, and hadoop Group)

Set the hadoop password for passwd hadoop (set the password to hadoop here)

[Root @ localhost ~] # Groupadd hadoop
[Root @ localhost ~] # Useradd-d/usr/hadoop-g hadoop-m hadoop
[Root @ localhost ~] # Passwd hadoop

3. Configure the jdk Environment

This installation is the hadoop-2.7.5, JDK 7 or above. Skip this step if it is already installed.

For JDK installation, refer to: Or CentOS7.2 to install JDK 1.7.

Or directly copying JDK files on the master node is more conducive to maintaining version consistency.

[Root @ localhost java] # su-root
[Root @ localhost java] # mkdir-p/usr/java
[Root @ localhost java] # scp-r hadoop @ hadoop-master:/usr/java/jdk1.7.0 _ 79/usr/java
[Root @ localhost java] # ll
Total 12
Drwxr-xr-x. 8 root 4096 Feb 13 0:34 default
Drwxr-xr-x. 8 root 4096 Feb 13 0:34 jdk1.7.0 _ 79
Drwxr-xr-x. 8 root 4096 Feb 13 0:34 latest

Set Java and hadoop Environment Variables

Ensure that/usr/java/jdk1.7.0.79 exists

Su-root

Vi/etc/profile

Ensure that/usr/java/jdk1.7.0.79 exists

Unset I
Unset-f pathmunge
JAVA_HOME =/usr/java/jdk1.7.0 _ 79
CLASSPATH =.: $ JAVA_HOME/lib/dt. jar: $ JAVA_HOME/lib/tools. jar
PATH =/usr/hadoop/hadoop-2.7.5/bin: $ JAVA_HOME/bin: $ PATH

Settings take effect (important)

[Root @ localhost ~] # Source/etc/profile
[Root @ localhost ~] #

After JDK is installed, confirm:

[Hadoop @ localhost ~] $ Java-version
Java version "1.7.0 _ 79"
Java (TM) SE Runtime Environment (build 1.7.0 _ 79-b15)
Java HotSpot (TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
[Hadoop @ localhost ~] $

4. Set hadoop Environment Variables

Copy the configured hadoop directory on namenode to the current host.

[Root @ localhost ~] # Su-hadoop
Last login: Sat Feb 24 14:04:55 CST 2018 on pts/1
[Hadoop @ localhost ~] $ Pwd
/Usr/hadoop
[Hadoop @ localhost ~] $ Scp-r hadoop @ hadoop-master:/usr/hadoop/hadoop-2.7.5.
The authenticity of host 'hadoop-master (192.168.48.129) 'can't be established.
ECDSA key fingerprint is 1e: cd: d1: 3d: b0: 5b: 62: 45: a3: 63: df: c7: 7a: 0f: b8: 7c.
Are you sure you want to continue connecting (yes/no )? Yes
Warning: Permanently added 'hadoop-master, 192.168.48.129 '(ECDSA) to the list of known hosts.
Hadoop @ hadoop-master's password:

[Hadoop @ localhost ~] $ Ll
Total 0
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Desktop
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Documents
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Downloads
Drwxr-xr-x 10 hadoop 150 Feb 24 hadoop-2.7.5
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Music
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Pictures
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Public
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Templates
Drwxr-xr-x 2 hadoop 6 Feb 24 :32 Videos
[Hadoop @ localhost ~] $

At this point, the Hadoop Client installation is complete and can be used later.

The result of executing the hadoop command is as follows,

[Hadoop @ localhost ~] $ Hadoop
Usage: hadoop [-- config confdir] [COMMAND | CLASSNAME]
CLASSNAME run the class named CLASSNAME
Or
Where COMMAND is one:
Fs run a generic filesystem user client
Version print the version
Jar <jar> run a jar file
Note: please use "yarn jar" to launch
YARN applications, not this command.
Checknative [-a |-h] check native hadoop and compression libraries availability
Distcp <srcurl> <desturl> copy file or directories recursively
Archive-archiveName NAME-p <parent path> <src> * <dest> create a hadoop archive
Classpath prints the class path needed to get
Credential interact with credential providers
Hadoop jar and the required libraries
Daemonlog get/set the log level for each daemon
Trace view and modify Hadoop tracing settings

Most commands print help when invoked w/o parameters.
[Hadoop @ localhost ~] $

5. Use hadoop

Create a local file

[Hadoop @ localhost ~] $ Hdfs dfs-ls
Found 1 items
Drwxr-xr-x-hadoop supergroup 0 2018-02-22 23:41 output
[Hadoop @ localhost ~] $ Vi my-local.txt
Hello boy!
Yehyeh

Upload local files to the Cluster

[Hadoop @ localhost ~] $ Hdfs dfs-mkdir upload
[Hadoop @ localhost ~] $ Hdfs dfs-ls upload
[Hadoop @ localhost ~] $ Hdfs dfs-ls
Found 2 items
Drwxr-xr-x-hadoop supergroup 0 2018-02-22 23:41 output
Drwxr-xr-x-hadoop supergroup 0 2018-02-23 22:38 upload
[Hadoop @ localhost ~] $ Hdfs dfs-ls upload
[Hadoop @ localhost ~] $ Hdfs dfs-put my-local.txt upload
[Hadoop @ localhost ~] $ Hdfs dfs-ls upload
Found 1 items
-Rw-r -- 3 hadoop supergroup 18 2018-02-23 upload/my-local.txt
[Hadoop @ localhost ~] $ Hdfs dfs-cat upload/my-local.txt
Hello boy!
Yehyeh
[Hadoop @ localhost ~] $

Ps: note that the local java version and the master copy of the file/etc/hadoop-env.sh configuration JAVA_HOME to be consistent have not been verified, this article is consistent.

Hadoop2.3-HA high-availability cluster environment construction

Hadoop project-Cloudera 5.10.1 (CDH) installation and deployment based on CentOS7

Detailed explanation of Hadoop2.7.2 cluster construction (high availability)

Use Ambari to deploy a Hadoop cluster (build an intranet HDP source)

Hadoop cluster installation in Ubuntu 14.04

Install Hadoop 2.7.2 In CentOS 6.7

Build a distributed Hadoop-2.7.3 cluster on Ubuntu 16.04

Build a Hadoop 2.6.4 distributed Cluster Environment in CentOS 7

Hadoop2.7.3 + Spark2.1.0 fully distributed cluster Construction Process

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.