Hadoop cluster configuration, problem solving approach

Overview: Hadoop cluster, 1 sets of Namenode, a secondnamenode, a jobtracker and Taiwan Datanode, the specific installation method on the Internet there are too many, the following is just their own set up the experimental environment and the problem solution. 1, the configuration IP corresponding hostname/etc/hosts configuration namenode and Datanode, shape as follows:

192.168.1.1 Namenode

192.168.1.2 Secondnamenode

192.168.1.3 Jobtracker

192.168.1.4 Datanode1

192.168.1.5 Datanode2 192.168.1.6 datanode3 2, configuring Java and Hadoop environment variables (omitted here), where the Hadoop environment can be configured in Namenode with commands to other nodes 3, configure machine-aware To enable the Hadoop rack-aware feature, configure an option in the Hadoop-site.xml configuration file of the machine where the Namenode is located:

<property>

<name>topology.script.file.name</name>

<value>/opt/modules/hadoop/hadoop-1.0.4/conf/RackAware.py</value> </property>
4. Safe Mode start error The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe mode is turned off automatically in seconds.         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal (Fsnamesystem.java : 2055)         at Org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs (Fsnamesystem.java : 2029)         at Org.apache.hadoop.hdfs.server.namenode.NameNode.mkdirs (namenode.java:817)         at SUN.REFLECT.NATIVEMETHODACCESSORIMPL.INVOKE0 (Native Method)         Sun.reflect.NativeMethodAccessorImpl.invoke (nativemethodaccessorimpl.java:39)         at Sun.reflect.DelegatingMethodAccessorImpl.invoke (delegatingmethodaccessorimpl.java:25)         At Java.lang.reflect.Method.invoke (method.java:597)         at org.apache.hadoop.ipc.rpc$ Server.call (rpc.java:563)         at Org.apacHe.hadoop.ipc.server$handler$1.run (server.java:1388)         at org.apache.hadoop.ipc.server$ Handler$1.run (server.java:1384)         at java.security.AccessController.doPrivileged (Native Method)         at Javax.security.auth.Subject.doAs (subject.java:396)         at O Rg.apache.hadoop.security.UserGroupInformation.doAs (usergroupinformation.java:1121)         at Org.apache.hadoop.ipc.server$handler.run (server.java:1382)         at Org.apache.hadoop.ipc.Client.call (client.java:1070)         at org.apache.hadoop.ipc.rpc$ Invoker.invoke (rpc.java:225)         at Com.sun.proxy. $Proxy 1.mkdirs (Unknown Source) Workaround: Just type in the directory of Hadoop:
$bin/hadoop Dfsadmin-safemode Leave
That is, the security mode of turning off Hadoop, which solves the problem. 5, MySQL installation solution vi/usr/my.cnf add skip_grant_tables sevice mysql restart. You can view the security files as follows: Cat. mysql_secret file 6, Hive cannot run because hive environment variable (1) is not configured to configure the Hive environment variable export hive_home=/opt/modules/hive/hive-0.1 0.0/export path= $PATH: $HIVE _home/bin (2) name the original installation package file and set the environment variable

cp/opt/modules/hive/hive-0.9.0/conf/hive-env.sh.template/opt/modules/hive/hive-0.9.0/conf/hive-env.sh

Export hadoop_heapsize=64 (default 1024, virtual machine memory is small must get rid of) hadoop_home=/opt/modules/hadoop/hadoop-1.0.4/ Export hive_conf_dir=/opt/modules/hive/hive-0.9.0/conf