Installing Hadoop on your Mac
Java under the 1,mac
Ll/usr/bin |grep ' Java '
Default:
/system/library/frameworks/javavm.framework/versions/current/commands/java
Specific installation path:
/library/java/javavirtualmachines/below,
Java_home is/library/java/javavirtualmachines/jdk1.7.0_75/contents/home.
SSH under the 2,mac
First Test ssh localhost,
If it does not succeed, see whether to start allow RemoteLogin, and then try again if it fails then continue to the following
Ssh-keygen-t Dsa-p "-F ~/.SSH/ID_DSA
Ssh-keygen indicates the creation of a secret key;
-T denotes the secret key type;-P is used to provide a secret word;-f Specifies the generated key file.
This command creates two files Id_dsa and id_dsa.pub in the "~/.ssh/" folder, which is a pair of SSH private and public keys.
Next, append the public key to the authorized key and enter:
Cat~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
Try again
SSH localhost
If this cannot be opened, the workaround is as follows:
Enter your Mac's system Preferences –> share –> check remote login and set allow access for all users.
3, download the image, mirror, select a tar.gz
http://mirror.bit.edu.cn/apache/hadoop/common/
4, setting environment variables
VI ~/.bash_profile or/etc/profile
Exportjava_home=/library/java/javavirtualmachines/jdk1.7.0_75/contents/home
Exportjre_home= $JAVA _home/jre
Exporthadoop_home=/users/xiaolong.zhao/ittools/hadoop
#是防止出现: Warning: Warning error $HADOOP _home is deprecated.
Exporthadoop_home_warn_suppress=1
Exportpath= $JAVA _home/bin: $JRE _home/bin: $ hadoop_home/sbin: $HADOOP _home/bin: $PATH
Note: The only difference between Linux and Mac installation is that there will be a default JDK path under the Mac, and the SSH configuration
5, Configuration hadoop-env.sh
Under the/users/xiaolong.zhao/ittools/hadoop/etc/hadoop directory, locate hadoop-env.sh, and open the edit to set up the following:
Export Java_home=${java_home} (remove comments)
Export hadoop_heapsize=2000 (Remove comments)
Export hadoop_opts= "-djava.security.krb5.realm=ox. AC. uk-djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk "(Remove comments)
6, 4 core file configurations
6.1. Configuration Core-site.xml--(DEFAULTFS) specifies the hostname and port of the Namenode
<?xmlversion= "1.0" encoding= "UTF-8"?> <configuration> <!-- fs.defaultFS - This is a URI that describes the Namenode node in the cluster (including protocol, host name, port number), and each machine in the cluster needs to know the Namenode address. datanode nodes are first registered on the Namenode so that their data can be used. The standalone client program interacts with Datanode through this URI to get a list of blocks of files. --> <property> <name >fs.defaultfs</name> <value>hdfs://localhost :8000</value> </property> <!-- hadoop.tmp.dir is the underlying configuration that Hadoop file system relies on, and many paths depend on it. if Namenode and Datanode are not configured in the Hdfs-site.xml, the default is placed in this path--> <property> <name>hadoop.tmp.dir</ name> <value>/users/time/documents/hadoop-2.7.3/tmp</value> <description>A base for othertemporary directories.</description> </property> </configuration>
6.2. Configuration Hdfs-site.xml--Specifies the number of default parameter replicas for HDFs
<?xmlversion= "1.0" encoding= "UTF-8"?> <configuration> <!-- dfs.replication -It determines the number of data backups of the file blocks in the system. For a real application, it should be set to 3. less than three backups may affect the reliability of the data--> < Property> <name>dfs.replication</name > <value>1</value> </property> <!--the physical storage of datanaode and Namenode, if not set, its value hadoop.tmp.dir--> <property> <name> dfs.namenode.name.dir</name> < value>/home/hdfs/name</value> </property> <!-- can not be set, Namenode web monitoring port, default is 0.0.0.0:50070--> <property> <name>dfs.namenode.http-address</name> <value>localhost:57000</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/home/hdfs/data</value> </property> </configuration>
6.3 Mapred-site.xml Configuration
<property> <name>mapreduce.framework.name</name> <value>yarn</value> </property>
6.4 Yarn-site.xml Configuration
<?xmlversion= "1.0"?> <configuration> <!-- resourcemanager --> <!-- client visit address --> <property> <name>yarn.resourcemanager.address</name> < value>localhost:18040</value> </property> <!-- Applicationmaster visit address--> <property> <name> Yarn.resourcemanager.scheduler.address</name> <value>localhost:18030</value> </property> <!-- nodemanager visit address --> <property> <name> yarn.resourcemanager.resource-tracker.address</name> <value> localhost:18025</ value> </property> <!-- weB visit address --> <property> <name> yarn.resourcemanager.webapp.address</name> <value>localhost:18088</value> </property> <!-- nodemanager support mapreduce service--> <property> <name> yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>
7, formatted Namenode
HDFs Namenode-format
8. Start Hadoop
8.1sbin/start-all.sh
8.2 Web Test
8.2.1 Namenode-web
http://localhost:50070
8.2.2resourcemanager-web
http://localhost:18088
8.2.3 Nodemanager-web
http://localhost:8042
This article is from the "Dream in the Cloud" blog, please be sure to keep this source http://computerdragon.blog.51cto.com/6235984/1939523
Install hadoop-2.8 process under Mac or Linux