Using virtual machine VirtualBox to configure the Hadoop development environment summary

The Mentor project will need to process large-scale sensing data in the future, so use Hadoop to study how to handle massive amounts of data. The installation of Hadoop was completed today, and a lot of problems were encountered during the period, which was summarized for later use.

The main reference is Hadoop configuration installation 1 and Hadoop configuration installation 2.

<TIP1> Mistake Campus network environment as LAN environment

First, two virtual machines (slaves) are configured on the local computer (master), and the network settings are automatically dhcp. So think of this "three computers", through the campus network switches constitute a local area network. One consequence of this is that although both Namenode and Datanode can be started, Namenode cannot monitor the running state of the Datanode. (=.= computer network did not learn well.) )

<TIP2> set SSH login without password

Master and slave, each other should be set without password login, otherwise it will also cause namenode can not monitor to Datanode.

<TIP3> Formatting Issues

After formatting, if you need to format again, you need to clear the newly generated Data,temp folder contents, and then format, otherwise it will also cause namenode can not be monitored to datanode.

<TIP4> configuration File Settings

Hadoop2.5.1, the configuration file settings are slightly different from hadoop1.x.x, the main thing is that the yarn-site.xml is set up correctly, otherwise the mapreduce operation cannot be performed.