Rice Internship-ubuntu 14.04 LTS installation Spark 1.6.0 (pseudo-distributed)

Software that needs to be downloaded:

1.hadoop-2.6.4.tar.gz Download URL: http://hadoop.apache.org/releases.html

2.scala-2.11.7.tgz Download URL: http://www.scala-lang.org/

3.spark-1.6.0-bin-hadoop2.6.tgz Download URL: http://spark.apache.org/

4.jdk-8u73-linux-x64.tar.gz Download URL: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

Root User's Open

In order to simplify the Linux system permissions problem, I am logged in as root user and use Ubuntu system, and Ubuntu system does not open the root user by default, we need to open the root user, I refer to the URL to enable the root user: http:/ /jingyan.baidu.com/article/27fa73268144f346f8271f83.html.

1. Open the terminal terminal (CTRL+ALT+T):

2. After you enter sudo gedit/usr/share/lightdm/lightdm.conf.d/50-ubuntu.conf, you may be prompted to enter a password, which will be shown when you enter the edit box. In the edit box, enter Greeter-show-manual-login=true save to close.

3. After closing, go back to the terminal window, enter: sudo passwd root return, after the return will ask you to enter the password two times, the successful password has been updated the word is successful.

4. Then after the shutdown restart, the login GUI, you can enter the root username and password login.

Installing the Java JDK

1. With the root user login, the CD to the JDK download storage place, the use of TAR-XF jdk-8u73-linux-x64.tar.gz decompression, decompression and use the clip command MV to put the JDK into the/usr/java directory.

2. Install the Vim text editor using Apt-get installation VIM command, CD to/etc directory, use VIM profile to modify the file to add Java environment variable, after opening the profile file, add the following text at the end:

Export Java_home=/usr/java/jdk1.8.0_73export path= $JAVA _home/bin: $PATHexport classpath=.: $JAVA _home/lib/dt.jar:$ Java_home/lib/tools.jar

When the add is complete, enter the source profile in terminal to make the environment variable effective.

3. Test whether Java is configured successfully, enter Java-version in terminal if the following message is successful.

Installing Hadoop

Installation of Hadoop main reference website pseudo-distributed installation tutorial, reference URL: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html

1. Install SSH and rsync with the following two commands:

  $ sudo apt-get install ssh  $ sudo apt-get install rsync

2.cd to hadoop-2.6.4.tar.gz download directory, the use of TAR-XF command decompression, the extracted folder using the MV command to cut into the directory/opt, for the Spark,scala are similar to this operation, no longer cumbersome.

3. Edit the file/etc/profile, add the environment variables of Hadoop, remember source profile

4. After adding the HADOOP environment variable, CD to directory/opt/hadoop-2.6.4/etc/hadoop/, modify the hadoop-env.sh file, define the following variables:

Export Java_home=/usr/java/latest

5. Pseudo-distributed also need to modify the Etc/hadoop/core-site.xml file to:

<configuration>    <property>        <name>fs.defaultFS</name>        <value>hdfs:// Localhost:9000</value>    </property></configuration>

Modify the Etc/hadoop/hdfs-site.xml file to:

<configuration>    <property>        <name>dfs.replication</name>        <value>1< /value>    </property></configuration>

6. Make SSH access unrestricted, you need to set the following, first enter SSH localhost check whether you can not need a password to complete ssh localhost, if you do not need to generate the following key:

  $ ssh-keygen-t dsa-p '-F ~/.SSH/ID_DSA  $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys  $ chmod 0600 ~ /.ssh/authorized_keys

7. When the above steps are complete, the pseudo-distribution of Hadoop will be completed, and then you can test if the installation is successful, and you can view the URL http://hadoop.apache.org/docs/current/hadoop-project-dist/ The execution part of the hadoop-common/singlecluster.html.

Install Scala

It's easier to install Scala, just place the extracted scala-2.11.7 folder in the/opt directory, and then modify the Etc/profile directory to add the environment variables that Scala needs.

1.vim etc/profile Add Environment variables

2. Use the command scala-version to check if the configuration is successful, and if the following information appears, it means success.

Install Spark

1. Write again tomorrow.

Rice Internship-ubuntu 14.04 LTS installation Spark 1.6.0 (pseudo-distributed)