Hadoop cluster Construction

Hadoop cluster Construction

I. Purpose

This article describes how to install, configure, and manage Hadoop clusters with practical significance. The scale of a Hadoop cluster can be from a small cluster with several nodes to a large cluster with thousands of nodes. If you want to install Hadoop on a single machine, you can find the relevant details here.

Ii. Prerequisites

1. Make sure that all necessary software is installed on each node in your cluster.

2. Obtain the Hadoop software package.

Iii. Installation

To install a Hadoop cluster, you usually need to decompress the software to all machines in the cluster.

Generally, one machine in the cluster is specified as NameNode, and another machine is specified as JobTracker. These machines are masters. The remaining machines are used as DataNode and TaskTracker. These machines are slaves.

We use HADOOP_HOME to refer to the root path of the installation. Generally, the HADOOP_HOME path of all machines in the cluster is the same.

-------------------------------------- Split line --------------------------------------

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition

Build a Hadoop environment (using virtual machines to build two Ubuntu systems in a Winodws environment)

-------------------------------------- Split line --------------------------------------

Iv. Configuration

The following describes how to configure a Hadoop cluster.

1. Configuration File

Configure Hadoop using two important configuration files in the conf/directory:

1. hadoop-default.xml-default configuration for read-only.

2. hadoop-site.xml-cluster-specific configuration.

To learn more