Installing a single-machine Hadoop system (full version)--mac

Ah ah, when it comes to this practice or the installation of something from-psychological to physical exclusion!! Miss Wu said the right, the teachers have lowered the very low requirements for us, can not self-fall again weak Ah!!

So, in this rainy afternoon, no nap I took a pair of sleepy eyes sitting in front of the computer, driving away the boyfriend, soaked in tea, touch is already slightly shiny forehead (it is more and more like a woman Han paper, Laughter cry), struggle ah ah ah Ah!! %>_<%

1. Curriculum review.

1.1 Hadoop system running software system: Linux operating system (small Red Hat or Ubuntu), SSH (Secure shell is mainly used to remotely manage Hadoop nodes and secure shared access between Hadoop nodes), Java

The main point here is a little bit of ssh, which is a network protocol for encrypting logins between computers, such as a user logging on to a remote computer from the local computer via the SSH protocol. See [1] for details.

1.2 Hadoop system Installation: A single way (installed on a single machine with Linux system running Hadoop system), single-machine pseudo-distributed mode (on a single machine with Linux, pseudo-distributed way, Using different Java processes to simulate the distributed operation of Namenode, Datenode, Jobtracker and other types of nodes, cluster distribution mode (install and run Hadoop system in a real cluster environment, each node of the cluster can run Linux)

Here, because the Max OSX system is UNIX-based, you can install the Hadoop system directly for experimentation. PS: I am really a rookie ah, oh, no, even the rookie is not a%>_<%

2. Installing Hadoop

Let's move on to the whole process of installing Hadoop ...

Referring to the blogs of the Great Gods [2], the following are a few steps that Mac installs Hadoop:

1) Install Homebrew

2) SSH localhost

3) Installing Hadoop

2.1 Installing HOMEBREW[3]

The reason for installing homebrew here is that Mac installs software with homebrew is really neat and elegant, just a command line code. Of course, speaking of the command line about code, how painful and boring for our rookie, but using Homebrew-cask to install the software really only need a line of command!

1) Install Homebrew

Open terminal (shortcut key control+space or directly in the upper right corner of the computer click Magnifier Input terminal), copy and paste the following statement or find the latest version of the Homebrew command line from the official website:

" $ (curl-fssl https://raw.githubusercontent.com/Homebrew/install/master/install) "

The process will be asked to enter the password, etc., follow the prompts to do.

2) Install Homebrew-cask

Open the terminal and copy and paste the following command line:

Brew Tap phinze/homebrew-caskbrew Install brew-cask

The following is my display results, for reference only (the last cursor appears as it was originally, indicating that the installation was successful):

So, how to install and uninstall the software with Homebrew-cask, borrow a picture of the Great God [3], the column of Gray often clear:

Some common cask commands are listed below:

Brew Cask Search lists all the software that can be installed brew cask search drop find all and drop related applications brew cask Info Thunder View Thunderbolt Application information, this shipment installs the latest version of Thunderbolt Oh! Brew Cask Uninstall QQ uninstall QQ

Update Software:

Brew Cask Uninstall QQ && Brew Cask Install QQ #因为cask每次安装都是安装最新的版本

At this point, homebrew-cask installation and nonsense part of the end, open the next SSK settings.

2.2 Configuring SSH localhost

• Open System Preferences-share-telnet (off by default, "open remotely");
• To generate a key pair, execute the following command

  $ ssh-keygen-t RSA

• Execute the following command to generate the Id_rsa file in the. ssh folder in the current user directory and you can log in later without using a password:

• $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

• Use the following command to test whether you can log on without a password:

• $ ssh localhost

2.3 Installing Hadoop

(*^__^*) hehe ... Here is the use of the previously installed Homebrew-cask, here just enter a command line to complete the installation of Hadoop, is not fried chicken easy ~ Of course, if you do not install homebrew, It's OK to install Hadoop directly with other software installed on your Mac, just download the latest version of Hadoop from its official website and install it.

Here is the Homebrew-cask way, the command line is as follows:

$ Brew Install Hadoop

If you are prompted to say that Java is not installed, you do not have the JDK installed, follow its command prompt, install Java, and then install Hadoop.

After you install Hadoop, you need to configure it to install successfully [4]. (Another huge project, there are a variety of small problems on the road, the following one by one ways to come)

• Locate the file in the directory to find one of the lines /usr/local/Cellar/hadoop/2.6.0/libexec/etc/hadoop hadoop-env.sh : (There are wooden friends do not create how to enter the directory AH. is to open the Finder and find the top toolbar with a "go"--"Go to Folder"--Copy the path in.

• Export hadoop_opts="$HADOOP _opts djava.net.preferipv4stack=true"

  Replace it with the following:

• Export hadoop_opts="$HADOOP _opts-djava.net.preferipv4stack=true-djava.security.krb5.realm=- djava.security.krb5.kdc="

• /usr/local/Cellar/hadoop/2.6.0/libexec/etc/hadoopfound under the directorycore-site.xml，插入如下代码：

• <configuration>  <property>     <name>hadoop.tmp.dir</name>  <value>/usr/ local/cellar/hadoop/hdfs/tmp</value>    Base for other temporary directories.</ description>  </property>  <property>     <name>fs.  Default.name</name>                                          <value>hdfs://localhost:9000</value>                               </property>                                                        </configuration>
  

• Locate the Mapred-site.xml in the directory where /usr/local/Cellar/hadoop/2.6.0/libexec/etc/hadoop you added the following:

• <configuration>      <property>        <name>mapred.job.tracker</name>        <value> localhost:9010</value>      </property></configuration>

  Over here!! This is it. I can't believe I didn't find mapred-site.xml. This file, only a mapred-site-template.xml file, and then can not open, finally there is no way to copy from a friend of such a file, specifically as follows: (Of course, the file's friends directly follow the above changes on the line)

• <?xml version="1.0"? ><?xml-stylesheet type="text/xsl"href="configuration.xsl"?><!--Licensed under the Apache License, Version2.0(The"License"); If you don't use ThisFile exceptinchcompliance with the License. Obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0unless required by applicable law or agreed toinchwriting, software distributed under the License isDistributed on an" as is"BASIS, without warranties or CONDITIONS of any KIND, either express or implied. See the License forThe specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><!--Put Site-specific property overridesinch  ThisFile. --><configuration> <property> <name>mapred.job.tracker</name> <value>loc Alhost:9010</value> </property></configuration>

• /usr/local/Cellar/hadoop/2.6.0/libexec/etc/hadoopfound under the directoryhdfs-site.xml

• <configuration>   <property>     <name>dfs.replication</name>     <value>1 </value>    </property></configuration>

• Before running the daemon, you must format the newly installed HDFS and innovate the empty file system by creating a storage directory and initializing metadata to execute the following command:

• $ HDFs Namenode-format# generates a string similar to the following: ... #此书省略大部分STARTUP_MSG:   1.6. 0_65*************** *********************************************/**************************************************** Shutdown_msg:shutting down NameNode at andrew-liudemacbook-pro.local/192.168.1.100************************ ************************************/

• Start daemon: /usr/local/Cellar/hadoop/2.6.0/sbin under Directory, execute the following command

• $ start-dfs.sh  #启动HDFS $ stop-dfs.sh  #停止HDFS Of course not now stop it

  PS: Self-reflection, I am small white I am a small white%>_<% terminal command is not really know AH. Here are a few commonly used for yourself and the same little white brain%>_<%%>_<%)

• /etc/-i #显示当前目录下所拥有的子目录的访问权限cd Desktop/ #切换至当前目录下的Desktopcd. /path #切换到当前目录下得path目录cd: // #切换到根目录 # in general, LS and CD are cross-used, enter a CD with LS to view, and then to enter the following layer of subdirectories

  Here I met a super serious problem, no USR access rights, anyway, Google for a long time, the specific steps are also some confusion, probably is to change the permissions, here just say how to modify a folder access rights, later status quo can. Right-click the folder you want to change permissions--"Show Introduction-" Click on the lower right corner of the small lock-"Enter password-" Change permissions-"Do not forget to lock after the change. That's OK.

Oh, yes, there's one more question, that is, I entered $start-dfs.sh after the prompt to say that the command is invalid, if this happens, you can change the above command to

• ./start-dfa.sh

Show the following style description to start successfully ~

Finally, enter http://localhost:50070 in the browser to access the Hadoop page.

From three o'clock in the afternoon toss to now, side record, side installation, side look only to install it, tomorrow again WordCount program run. Go home to sleep a beauty sleep, tomorrow guaranteed to complete, O (∩_∩) o haha ~

Reference documents:

[1]. SSH principle and application (i): Telnet--http://www.ruanyifeng.com/blog/2011/12/ssh_remote_login.html

[2]. MAC OS X Yosemite installing Hadoop 2.6 Records--http://blog.csdn.net/yechaodechuntian/article/details/44312343

[3]. Simple and elegant Mac OS x Software Installation Experience-homebrew-cask--http://ksmx.me/homebrew-cask-cli-workflow-to-install-mac-applications/

[4]. Install HADOOP--HTTP://ANDREWLIU.TK/2015/03/05/%E5%9C%A8MAC-OSX-YOSEMITE%E4%B8%8A%E5%AE%89%E8%A3 on Mac OSX Yosemite %85hadoop/

Installing a single-machine Hadoop system (full version)--mac