problem. Execute the hadoop-examples-1.2.1.jar program, in fact, is to compile the java program into a jar file, and then run directly, you can get the results. In fact, this is also a method for running java programs in the future. Compile, package, and upload the program and run it. In addition, eclipse connects to Hadoop and can be tested online. The two methods have their own advantages and are not des
Hadoop-1.x installation and configuration
1. Install JDK and SSH before installing Hadoop.
Hadoop is developed in Java. MapReduce and Hadoop compilation depend on JDK. Therefore, JDK1.6 or later must be installed first (JDK 1.6 is generally used in the actual production envi
Reprinted from: http://www.cnblogs.com/spark-china/p/3941878.html
Prepare a second, third machine running Ubuntu system in VMware;
Building the second to third machine running Ubuntu in VMware is exactly the same as building the first machine, again not repeating it.Different points from installing the first Ubuntu machine are:1th: We name the second to third Ubuntu machine for Slave1, Slave2, as shown in:There are three virtual machines in the created VMware:2nd: To simplify the
1 installationinstall hadoopInstall the 2.6.0, directory is/usr/local/cellar/hadoop, if you want to install another version, download the TAR package decompression. Address: http://mirrors.cnnic.cn/apache/hadoop/common/2 configurationConfigure the Hadoop executable path bin and sbin to environment variablesHADOOP_HOME=/usr/local/Cellar/
-------------Add a child element7. Assign the user read access to the directory----------------sudo chown-r uit:uit/home/uit/hadoop-0.20.2Permissions this really made my egg ache for a while. Always thought it was not well-equipped, the original authority did not fix, later found the right after the decompressionLimit what all have, is because did not add this sentence. Plus, it's all part of the current user.8. Change the environment variable--------
time you run Hadoop, you want to format the file system for Hadoop.In the Hadoop directory, enter:$ bin/hadoop Namenode-formatTo start the Hadoop service:$ bin/start-all.shIf there is no error, it means that the launch was successful.(3) Verify that Hadoop is installed succ
contentsHadoop Fs-tail/user/trunk/test.txt #查看 The last 1000 lines of the/user/trunk/test.txt fileHadoop fs-rm/user/trunk/test.txt #删除/user/trunk/test.txt fileHadoop fs-help ls #查看ls命令的帮助文档Two HDFS deployment The main steps are as follows:1. Configure the installation environment for Hadoop;2. Configure the configuration file for Hadoop;3. Start the HDFs ser
Hadoop example code:
1. creatinga configuration object: to be able to read from or write to HDFS, you need tocreate a configuration object and pass configuration parameter to it usinghadoop configuration files.
ImportOrg. Apache.
variable.Note: If the file you downloaded is in RPM format, you can install it by using the following command:RPM-IVH jdk-7u72-linux-x64.rpm4.5. environment variable settings Modify the. Profile file (this is recommended so that other programs can also use the JDK in a friendly way)# Vi/etc/profileLocate the export PATH USER LOGNAME MAIL HOSTNAME histsize INPUTRC in the file, and change to the following form:Export java_home=/opt/java/jdk1.7.0_72Export path= $PATH: $JAVA _home/bin: $JAVA _home/
Three hadoop modes:Local Mode: local simulation, without using a Distributed File SystemPseudo-distributed mode: five processes are started on one host.Fully Distributed mode: at least three nodes, JobTracker and NameNode are on the same host, secondaryNameNode is a host, DataNode and Tasktracker are a host.Test environment:
CentOS2.6.32-358. el6.x86 _ 64
Jdk-7u21-linux-x64.rpm
Hadoop-0.20.2-cdh3u6.tar.gz1.
Hadoop installation and configuration Manual
I. Preparation
Hadoop runtime environment:
SSH service running properly
JDK
If you have not installed it, you can install it yourself.
Ii. BASICS (single-node Hadoop)
Hadoop download
H
complete this setting. If you have completed the preceding settings, you can test the Hadoop command in the command line, for example:
If you can see the above results, congratulations, Hadoop installation is complete. Next we can configure pseudo-distribution (Hadoop can run a single node in pseudo-distribution mode
For example, we demonstrate how to install Hadoop2.6.0 in a single node cluster. The installation of SSH and JDK is described in the previous article and is not covered here.Installation steps:(1) Place the downloaded Hadoop installation package in the specified directory, such as the home directory of your current user. Execute the following command to unpack the installation package:Tar xzf
rsync
Then confirm that you can use SSH to log on to localhost without a password
Enter the SSH localhost command: SSH localhost
Ssh-keygen-t dsa-P "-f ~ /. Ssh/id_dsa
Cat ~ /. Ssh/id_dsa.pub> ~ /. Ssh/authorized_keys
Note:-P is followed by two "" signs, indicating that the password is set to null.
After completing the preceding configuration, decompress the package and configure hadoop.
1. decompress
Data deduplication:
Data deduplication only occurs once, so the key in the reduce stage is used as the input, but there is no requirement for values-in, that is, the input key is directly used as the output key, and leave the value empty. The procedure is similar to wordcount:
Tip: Input/Output path configuration.
Import Java. io. ioexception; import Org. apache. hadoop. conf.
installed, you can decompress the package directly here. I use the following directory structure, as shown in the following environment variables. After decompression, put the package in/usr/Java. You need to configure the environment variable, VIM/etc/profile.
Export java_home =/usr/Java/jdk1.7.0 _ 60
Export jre_home =/usr/Java/jdk1.7.0 _ 60
Export classpath =.: $ java_home/lib/dt. jar: $ java_home/lib/tools. jar: $ jre_home/lib
Export Path = $ path: $ java_home/bin: jre_home/bin
Then ESC, sav
Hadoop-2.X installation and configuration
We use a single-node cluster as an example to demonstrate how to install Hadoop2.6.0. The installation of ssh and jdk is described in the previous article.
Installation steps:
(1) Place the downloaded Hadoop installation package to the specified directory, for
is, the host name and IP settings for each node are mentioned in the Hadoop configuration.
If it is not a server configured in hosts, it can also be specified by Hbase.regionserver.dns.nameserver. (This is configured in Hbase-site.xml)
5, NTP
Time calibration for all nodes. (There is a lot of information on the Internet, omitted here)
6, Ulimit and Nproc (all nodes must be set)
The default size is 1024, wi
Hadoop remote Client installation configuration
Client system: ubuntu12.04
Client User name: Mjiang
Server username: Hadoop download Hadoop installation package, guaranteed and server version consistent (or the Hadoop installation package for direct copy server)To http://mi
. Generate public and private keys:
$ ssh-keygen-y-T Rsa-p ""
At this point, two files are generated under/home/hduser/.ssh: Id_rsa and Id_rsa.pub, the former private key and the public key.
5. Now we append the public key to the Authorized_keys$ cat ~/.ssh/id_rsa.pub>> ~/.ssh/authorized_keys
6. Login to SSH, confirm that you do not need to enter the password
SSH localhost
7. Log OutExit
If you log in again, you don't need a password.
Four, install Hadoop
The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion;
products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the
content of the page makes you feel confusing, please write us an email, we will handle the problem
within 5 days after receiving your email.
If you find any instances of plagiarism from the community, please send an email to:
info-contact@alibabacloud.com
and provide relevant evidence. A staff member will contact you within 5 working days.