MAC Spark stand-alone deployment

Because the application needs to start learning data processing technology, online use of Spark, the same with the spark (spark performance is 100 times times the Hadoop, I also hearsay, no pro-test. ）

1.ssh Password-Free login configuration

Mac with SSH does not need to install, only need to create a secret and put in the secret file can be

生成秘要文件： ssh-keygen -t rsa
第一个输入提示是 生成文件名可以直接回车使用默认的文件名，如果默认文件名已经有文件存在会有提示是否覆盖，根据提示输入yes即可覆盖原有文件。紧接着提示输入密码和确认密码。
生成的文件默认在~/.ssh/目录中，然后将id_rsa.pub文件写入authorized_keys中 ：cat id_rsa.pub >> authorized_key
authorized_keys 是ssh登录是认证文件，如果存在key可以认为是信任用户直接登录。
文件夹下hosts（可能不存在）是一个缓存文件，记录登录的用户ip或用户名，如果用户key有变动可以删除此文件。
Mac中不允许root用户使用ssh登录（安全策略）默认最大连接数是10 ，默认不开启秘要登录。
开启秘要登录：PubkeyAuthentication yes 
最大连接数：MaxStartups 10
启动sshd服务：
sudo launchctl load-w/system/library/launchdaemons/ssh.plist
To stop the SSHD service:
sudo launchctl unload-w/system/library/launchdaemons/ssh.plist
To see if it starts:
sudo launchctl list | grep ssh

There may be file permissions issues, or IP blacklists, port occupancy, or other issues you can use shh-v localhost to view log analysis.

2. Configuring Java_home Environment variables
Use/usr/libexec/java_home-v in Mac to view JDK installation path add java_home environment variable in ~/.bash_profile and reload source ~/.bash_profile

3. Download the corresponding version of Hadoop (I am using 2.8.3 Other versions available for further information)
I am using the Hadoop program to compile the installation, can avoid the download version native load failure problems
Hadoop release File Download path http://hadoop.apache.org/releases.html

Install the Mevan and set the environment variables to test your own other tutorials,
Installing protobuf2.5.0
Installing Cmake:brew Install CMAK
Install OpenSSL with snappy:brew install OpenSSL, brew install snappy
Enter Hadoop-2.8.3-src folder after successful installation

1: Terminal execution

cmake/users/shihaolin/opt/software/hadoop-2.7.4-src/hadoop-tools/hadoop-pipes/src/-DJVM_ARCH_DATA_MODEL=64

After successful execution

2: Open the configuration Java environment variable place

3: Add the following two lines (depending on the OpenSSL installation version on your computer and the file address configuration)

Export OPENSSL_ROOT_DIR=/USR/LOCAL/CELLAR/OPENSSL/1.0.2L

Export Openssl_include_dir=/usr/local/cellar/openssl/1.0.2l/include

4: Remember to modify the environment variable and then make sure to execute the command immediately before executing

cmake/users/shihaolin/opt/software/hadoop-2.7.4-src/hadoop-tools/hadoop-pipes/src/-DJVM_ARCH_DATA_MODEL=6

And then execute
MVN Clean Package-pdist,native-dskiptests-dtar
After the compilation is successful, the compiled file can be placed in the desired location and the compiled file will be in the Dict Project target folder.
Compilation reference: 77655862
Then start configuring Hadoop
Reference https://www.cnblogs.com/bigdataZJ/p/hellospark.html

Permission problems may occur in the compilation, it is recommended to install with normal permissions.

MAC Spark stand-alone deployment