Spark is built under Windows environment

Since Spark is written in Scala, Spark is definitely the original support for Scala, so here is a Scala-based introduction to the spark environment, consisting of four steps: JDK installation, Scala installation, spark installation, Download and configuration of Hadoop. In order to highlight the "from Scratch" characteristics (all the title is not selected for the sake of), so the following steps slightly more verbose, the old driver can not have to read, skip directly on the good.
　　

A Installation of JDK and setting of environment variable 1.1 JDK installation

　　The JDK(full name is JAVATM Platform standard Edition development Kit) installation, is the Java SE Downloads, generally enter the page, will default to display a version of the JDK, as shown in, The latest version is JDK 8, and the more specific address is the Java SE Development Kit 8 Downloads:

The two places marked in red are clickable, and you can see some more detailed information about the latest version when you click inside, as shown in:

First of all, there are two versions of 8u101 and 8u102, and the official instructions given by Java are:

"Java SE 8u101 includes important security fixes. Oracle strongly recommends that all Java SE 8 users upgrade to this release. Java SE 8u102 is a patch-set update, including all of 8u101 plus additional features (described in the release notes). ”

This means that Java recommends that all developers upgrade from previous versions to JDK 8u101, while JDK 8u102 has a number of features in addition to all of the 101 features. For the version of the choice, the choice is good, in fact, for ordinary developers, there is no big difference, I am using the JDK 8u101 version.

Select the 8u101 version, then choose your corresponding development platform, because my machine is 64 bits, so I choose Windows64 bit version here. Remember, before downloading, you must accept the above license agreement, circled in red.

In addition to downloading the latest version of the JDK, it can also be downloaded to the historical version of the JDK in Oracle Java Archive, but the official recommendation is only for testing purposes.

JDK installation under Windows is very simple, follow the normal software installation ideas to double-click the downloaded EXE file, and then set your own installation directory (installation directory in the setting of environment variables need to use).
　　

1.2 Setting of environment variables

Next, set the appropriate environment variables, set the method: Right-click on the desktop "computer"-"Properties"-"Advanced system Settings", and then select "Advanced"-"environment variable" in the System properties, and then find the "Path" variable in the system variable, and select "Edit" button to come out a dialog box, you can add in the previous step in the JDK directory installed under the Bin folder path name, I here the Bin folder path name is: F:\Program Files\java\jdk1.8.0_101\bin, So add this to the path name, notice the semicolon in English ";" To split. After this is set up, you can run the CMD command-line window open under any directory

Java-version

See if you can output the relevant Java version information, and if you can output it, the JDK installation step is all over.

The entire process is as shown (the system variable settings for subsequent software installations are this process):

1.3 Some digression

Here are two off-topic, you crossing do not care about the words can be skipped here, does not affect the subsequent installation steps.
In the software installation, I believe that you have encountered the environment variables and system variables, so here to grilled a headache PATH, CLASSPATH and java_home parameters such as the specific meaning.

1.3.1 Environment variables, system variables, and user variables

• Environment variables include system variables and user variables
• The settings for the system variables work for all users under the operating system;
• Settings for user variables only work for the current user

If you are not particularly familiar with these concepts, it is advisable to look at the following points before looking back at these three words.

1.3.2 PATH

That is the system variable set in the previous step, tell the operating system where to find the execution path of Java.exe, when you sudden the command Line window to the following command,

Java-version

The operating system first startled, what the hell does "Java" mean? But spit groove to spit groove, live or have to dry, so leisurely remember the gates dad said three words:

1. When you don't understand a command in the command Line window, you first go to the current directory you are looking for, do you have the. exe program for this command? If so, use it to start the execution;
2. If not, do not give up, remember to go to those directories under the Path system variable to find out, if found, start and execute the command;
3. If the above two places are still not found, then you have to sprinkle a sweet, to report a mistake.

So we add the Bin folder under the JDK installation directory to the path system variable, and here we tell the operating system: if you can't find Java.exe in the current directory, go to the path system variable and find it Until we find Java.exe. So why set up the bin folder instead of the root directory of the JDK installation? The reason is that there is no java.exe in the root directory, only the bin folder can be ah hello ...

If you just run the Java command in the command-line window, you can also not set the system variables, but each time you run Java commands in a command-line window, you must take a long string of pathname to directly specify the location of the Java.exe, as shown below.

C:\Users\weizierxu>F:\Program Files\Java\jdk1.8.0_101\bin\java.exe -version‘F:\Program‘ 不是内部或外部命令，也不是可运行的程序或批处理文件。

Note: The reason for the error here is not that there is a problem with specifying the path name of the Java.exe directly, but that the path name with spaces cannot be resolved under the command line, so you need to use double quotes, as follows:

C:\Users\weizierxu>"F:\Program Files"\Java\jdk1.8.0_101\bin\java.exe -versionjava version "1.8.0_101"Java(TM) SE Runtime Environment (build 1.8.0_101-b13)Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

1.3.3 CLASSPATH

Classpath is when Java executes a compiled class file, tell Java which directories to find the class file, such as your program to use a jar package (the jar package is a compiled class file), then in the execution, Java needs to find this jar package, where to find it? From the directory specified by Classpath, search from left to right (the path names separated by semicolons) until you find the class file with the name you specified, and you will get an error if you can't find it. To do an experiment here, you can understand exactly what it means.
First, in the F:\Program Files\java directory, I wrote a program similar to Hello World with a notepad from windows, saved as a Testclasspath.java file (note that the suffix is changed to Java), as follows:

public class testclasspath{public static void  Main (string[] args) {System.out.println ( "Hello, this was a test on classpath!");}}  

Then I switch the current directory of CMD to the (via CD command) F:\Program Files\java directory, and then use the javac command to . Java The file is compiled as shown in:
 

As you can see, the javac command works correctly (no output indicates that it was compiled correctly), This is because the javac.exe that executes the command also exists in the bin directory under the JDK installation path, and this directory has been added to the path system variable, so cmd can recognize the command. This time you can see a testclasspath.class file in the F:\Program Files\java directory. But when I run this class file, I get an error. At this point, Classpath comes in handy, as is the method set for the Path system variable in the 1.2 section. Here in Classpath (if there is no classpath this option in the list of system variables, click New , and then add the path) to add the last ;. , The semicolon in English is separated from the previously existing path, followed by the dot . Represents the meaning of the current directory.
This time remember to start a new CMD window, and then use the CD command to switch to the Testclasspath.class directory, and then to execute, you can successfully get the results.

f:\program Files\Java> Java Testclasspathhello, this is a test on classpath!      

Therefore, unlike the Path variable, Java executes a class file, there is no default to find the file from the current directory, but only to CLASSPATH The specified directory to find the class file, if CLASSPATH The specified directory has this class file, then start execution, if not the error (here is to go to the current directory to find this class file, because the current path through . the way, has been added to the CLASSPATH System variable).
The above-mentioned method of specifying CLASSPATH System variables is written directly to the system variables, in order to avoid interference (such as multiple class files with the same name exist in multiple paths, these paths are added to the CLASSPATH system variables, Since the path of the CLASSPATH system variable is scanned from left to right when the class file is being searched, when the method is executed, the position is in the java testClassPath leftmost path of the CLASSPATH system variable. The corresponding class file, which is obviously not the result we want, so in these Ides such as Eclipse, there is no need to manually set the CLASSPATH system variable, but to set only the specific CLASSPATH of the current program system variable so that it does not affect the operation of other programs.

1.3.4 Java_home

Java_home is not the required parameters of JAVA itself, but other third-party tools need this parameter to configure their own parameters, it is not only the meaning of the software to tell those, my JDK installed in this directory, if you want to use my JAVA program, Just come to my directory and find out, and Java_home is the name of the JDK's installation path. For example, if my JDK is installed in a F:\Program Files\Java\jdk1.8.0_101 directory (note that the bin directory in this directory is the value to be added in the Path system variable in the 1.3.2 section), then the value to be added in java_home is F:\Program Files\Java\jdk1.8.0_101 , After encountering similar HOME system variables, it is the installation directory of the software.

Two. Scala's installation

First download from download PREVIOUS versions to the corresponding version, it is important to note that each version of Spark needs to correspond to the corresponding Scala version, such as the spark 1.6.2 I use here can only use the versions of Scala 2.10, The latest Spark 2.0 can only be used for each version of Scala 2.11, so when downloading, you need to be aware of the relationship between this Scala version and the spark version. I'm now using the Scala 2.10.6, which fits the various versions of Spark from 1.3.0 to spark 1.6.2. On the version page download PREVIOUS versions Select a version that suits your needs, it will go to the specific download page of that version, as shown in, remember to download the binary version of Scala, click the arrow in the figure, download it:

Once you have downloaded the Scala MSI file, you can double-click to perform the installation. Once the installation is successful, the Scala bin directory is added to the path system variable by default (if not, the bin directory path under the Scala installation directory is added to the system variable path in the same way as in the JDK installation steps), In order to verify that the installation was successful, open a new CMD window, enter it, scala and return it, if you can enter the Scala Interactive command environment, the installation is successful. As shown in the following:

If you cannot display version information and you cannot enter Scala's interactive command line, there are usually two possibilities:
-The path name of the Bin folder under the Scala installation directory is not added correctly in the path system variable, and is added as described in the JDK installation.
-Scala does not install correctly, repeat the above steps.

Three. Installation of Spark

The installation of Spark is very simple and goes directly to download Apache spark. There are two steps:

• Select the spark version that corresponds to the Hadoop version, as shown in;
• Then click the middle arrow spark-1.6.2-bin-hadoop2.6.tgz to wait for the download to finish.

Here is the version of pre-built, meaning that it has been compiled well, download to use directly, Spark also has source code can be downloaded, but you have to manually compile before you can use. After the download completes the file decompression (may need to extract two times), it is best to extract to a disk root directory, and renamed to Spark , simple and error-prone. It is also important to note that there are no spaces in the file directory pathname of Spark, and folder names like "program Files" are not allowed.

After the decompression basically almost can be run to the cmd command line. But this time every time you run Spark-shell (Spark's command-Line interactive window), you need cd to go to the installation directory of Spark, which is cumbersome, so you can add the spark's Bin directory to the system variable PATH . For example, my spark here in the bin directory path D:\Spark\bin , then the path name is added to the system variable path, the method and the JDK installation process environment variable settings consistent, after setting the system variables, in any directory of the cmd command line, directly execute spark-shellcommand, you can turn on the interactive command-line mode of Spark.

Four Hadoop download

After the system variable is set up, you can run Spark-shell in cmd in any current directory, but this time it is possible to encounter various errors, mainly because Spark is based on Hadoop, so it is also necessary to configure a Hadoop operating environment. The various historical versions of Hadoop can be seen in Hadoop releases, because the downloaded Spark is based on Hadoop 2.6 (the first step in the spark installation, which we chose Pre-built for Hadoop 2.6 ), I choose the 2.6.4 version here, select the appropriate version and click, go to the detailed download page, as shown, select the red flag in the image to download, here the SRC version is the source code, need to make changes to Hadoop or want to compile themselves can download the corresponding src file, I download here is the compiled version, that is, figure The ' hadoop-2.6.4.tar.gz ' file in the.

 
Download and extract to the specified directory, and then to the Environment variables section Hadoop_home for the extract directory of HADOOP, I am here F:\Program files\ Hadoop , and then set the bin directory under this directory to the PATH of the system variable, and here I am F:\Program files\hadoop\bin , If you have already added a hadoop_home system variable, you can also use %hadoop_home%\bin to specify the Bin folder path name. When these two system variables are set up, open a new cmd and enter the Spark-shell command directly.

Under normal circumstances, you can run successfully and enter the command-line environment of Spark, but for some users you might encounter a null pointer error. This time, the main reason is because there is no Winutils.exe file in the bin directory of Hadoop. The solution here is:

-Go to Https://github.com/steveloughran/winutils Select the Hadoop version number you installed, then go to the Bin directory, find the winutils.exe file, download the method is click on the winutils.exe file, After entering, there is a button in the upper right part of the page Download , click Download.
-After downloading winutils.exe , put this file into the bin directory of Hadoop, here I am F:\Program Files\hadoop\bin .
-Enter in an open cmd

F:\Program Files\hadoop\bin\winutils.exe chmod 777 /tmp/Hive 

This action is used to modify the permissions. Note that the previous F:\Program Files\hadoop\bin section should be replaced by the actual location of the bin directory where you installed it.

After these steps, then open a new CMD window again, and if normal, you should be able to run spark through direct input spark-shell .
The normal operating interface should look like the following:

As you can see, when the command is entered directly spark-shell , Spark starts and outputs some log information, most of which can be ignored, with two sentences to note:

as sc.SQL context available as sqlContext.

Spark contextAnd the SQL context difference is what, follow up again, now only need to remember, only see these two statements, only to show that spark really successful launch.

Five. Pyspark under Python

For spark under Python, similar to Scala's Spark-shell, there is also a pyspark, which is also an interactive command-line tool that can perform some simple debugging and testing on spark, similar to Spark-shell. For the need to install Python, it is recommended to use Python (x, y), it is the advantage of the collection of most of the toolkit, do not need to download it alone and can be directly import to use, and also save the cumbersome environment variable configuration, is Python (x, y)- Downloads, when the download is complete, double-click to run the installation. Because this tutorial mainly in Scala, about Python does not do too much to explain.

Six. Summary

At this point, the basic Spark local debugging environment is owned and is sufficient for the initial spark learning. However, this model is still not enough for actual spark development, and it needs to be aided by a more useful IDE to assist the development process. The next lecture will focus on Itellij idea and Maven's configuration process.

Spark is built under Windows environment