[Linux] [Hadoop] Run hadoop and linuxhadoop

[Linux] [Hadoop] Run hadoop and linuxhadoop

The preceding installation process is to be supplemented. After hadoop installation is complete, run the relevant commands to run hadoop.

 

Run the following command to start all services:

hadoop@ubuntu:/usr/local/gz/hadoop-2.4.1$ ./sbin/start-all.sh

Of course there will be a lot of startup files under the directory hadoop-2.4.1/sbin:

#! /Usr/bin/env bash # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. see the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except T in compliance with # the License. you may obtain a copy of the License at # http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "as is" BASIS, # without warranties or conditions of any kind, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Start all hadoop daemons. run this on master node. echo "This script is Deprecated. instead use start-dfs.sh and start-yarn.sh"# Here the script has been discarded and we need to start it with start-dfs.sh and start-yarn.sh

Bin = 'dirname "$ {BASH_SOURCE-$0}" 'bin = 'CD "$ bin"; pwd' DEFAULT _ LIBEXEC_DIR = "$ bin "/.. /libexecHADOOP_LIBEXEC_DIR =$ {HADOOP_LIBEXEC_DIR:-$ DEFAULT_LIBEXEC_DIR }. $ HADOOP_LIBEXEC_DIR/hadoop-config.sh# Execute the relevant configuration file in the hadoop/libexec/hadoop-config.sh, the configuration file is all configuration-related paths, CLASSPATH, export-related

# The actual execution is the following two, that is, to execute the start-dfs.sh and the start-yarn.sh two scripts, and then they are still themselves to execute these two Commands

# start hdfs daemons if hdfs is presentif [ -f "${HADOOP_HDFS_HOME}"/sbin/start-dfs.sh ]; then  "${HADOOP_HDFS_HOME}"/sbin/start-dfs.sh --config $HADOOP_CONF_DIRfi# start yarn daemons if yarn is presentif [ -f "${HADOOP_YARN_HOME}"/sbin/start-yarn.sh ]; then  "${HADOOP_YARN_HOME}"/sbin/start-yarn.sh --config $HADOOP_CONF_DIRfi

After the execution is complete, call jps to check whether all services have been started:

2014-07-21 22:05:21, 064 INFO org. apache. hadoop. hdfs. server. common. storage: Lock on/usr/local/gz/hadoop-2.4.1/dfs/data/in_use.lock acquired by nodename 3312 @ ubuntu2014-07-21 22:05:21, 075 FATAL org. apache. hadoop. hdfs. server. datanode. dataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1: 9000. exiting. java. io. IOException:Incompatible clusterIDs in/usr/local/gz/hadoop-2.4.1/dfs/data: namenode clusterID = CID-2cfdb22e-07b2-4ab8-965d-fdb27645bd62; datanode clusterID = ID-2cfdb22e-07b2-4ab8-965d-fdb27645bd62At org. apache. hadoop. hdfs. server. datanode. dataStorage. doTransition (DataStorage. java: 477) at org. apache. hadoop. hdfs. server. datanode. dataStorage. recoverTransitionRead (DataStorage. java: 226) at org. apache. hadoop. hdfs. server. datanode. dataStorage. recoverTransitionRead (DataStorage. java: 254) at org. apache. hadoop. hdfs. server. datanode. dataNode. initStorage (DataNode. java: 974) at org. apache. hadoop. hdfs. server. datanode. dataNode. initBlockPool (DataNode. java: 945) at org. apache. hadoop. hdfs. server. datanode. BPOfferService. verifyAndSetNamespaceInfo (BPOfferService. java: 278) at org. apache. hadoop. hdfs. server. datanode. BPServiceActor. connectToNNAndHandshake (BPServiceActor. java: 220) at org. apache. hadoop. hdfs. server. datanode. BPServiceActor. run (BPServiceActor. java: 816) at java. lang. thread. run (Thread. java: 722) 22:05:21, 084 WARN org. apache. hadoop. hdfs. server. datanode. dataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1: 90002014-07-21 22:05:21, 102 INFO org. apache. hadoop. hdfs. server. datanode. dataNode: Removed Block pool <registering> (Datanode Uuid unassigned) 2014-07-21 22:05:23, 103 WARN org. apache. hadoop. hdfs. server. datanode. dataNode: Exiting Datanode2014-07-21 22:05:23, 106 INFO org. apache. hadoop. util. exitUtil: Exiting with status 02014-07-21 22:05:23, 112 INFO org. apache. hadoop. hdfs. server. datanode. dataNode: SHUTDOWN_MSG: /*************************************** * ******************** SHUTDOWN_MSG: shutting down DataNode at ubuntu/127.0.1.1 ********************************* ***************************/

After searching the error log, it is found that the problem is caused by formatting namenode by using commands after Hadoop is started. The clusterID of datanode and namenode are inconsistent:

Find the datanode and namenode configured in the hadoop/etc/hadoop/hdfs-site.xml configuration file. /current/VERSION, file, compare the clusterID in the two files, inconsistent, change the clusterID under Datanode to this ID under namenode, and then restart.

Reference connection:

http://www.cnblogs.com/kinglau/p/3796274.html

How to Learn hadoop first?

What are your motivations for learning hadoop? It's just fun. I still want to engage in this aspect.
If it is the latter, it would be a joke not to learn about linux. hadoop, simply put, involves multiple hosts for one storage or database. You don't know how to configure various linux environments in linux? How to control his load balancing?
You even need to learn about the internet. Some errors are abstract and you need to determine the cause of the Error Based on some network basics.

Linux operating system: it is better to use the hadoop architecture for development.

The use of hadoop is not as good as that of the linux system, but hadoop is better in linux. In fact, linux is similar, and the kernel is the same. It is also similar in use. I use centos, cenos is recommended. Many companies now use centos