Build spark+hadoop development environment under Windows

Just make sure your computer is installed in the Java environment, and you're ready to start.

I. Preparatory work

1. Download the Hadoop2.7.1 version (write Spark and Hadoop are mostly used yarn, so Hadoop must be installed)

: HTTP://APACHE.FAYEA.COM/HADOOP/COMMON/HADOOP-2.7.1/

Download the hadoop-2.7.1.tar.gz and unzip it to a fixed directory as the Hadoop installation directory.

2. Download the support plugin

: Https://github.com/ShixiangWan/winutils, which has a lot of hadoop versions, here we take the Hadoop2.7.1 stable version as an example.

After downloading, the files in the bin of the Hadoop2.7.1 are overwritten and copied to the bin of the above installation directory.

3. Cover Hadoop.dll

Copy the Hadoop.dll under the plug-in bin directory to C:\Window\System32.

Two. Environment variable configuration

1. Add the environment variable hadoop_home, point to the Hadoop installation root directory;

2. Add%hadoop_home%\bin to the path variable;

3. Restart eclipse or IntelliJ to run the MAVEN project directly from Spark or Hadoop.

Resources:

http://blog.csdn.net/shenshendeai/article/details/57962167

Build spark+hadoop development environment under Windows