Step 4 modify the configuration file slaves as follows:
We set the slave nodes in the hadoop cluster to sparkworker1 and sparkworker2, and modify the slaves file content:
Step 5 modify the profile core-site.xml, as shown below:
Modify the content of the core-site.xml file:
The above is the minimal configuration of the core-site.xml file, the core-site.xml configuration can refer:
Http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
Step 6 modify the profile hdfs-site.xml, as shown below:
Modify the content of the hdfs-site.xml file:
The above is the minimum configuration of the hdfs-site.xml file, the hdfs-site.xml configuration course reference:
Http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
Step 7 modify the profile mapred-site.xml, as shown below:
Copy A mapred-site.xml.template named as a mapred-site.xml, open the mapred-site.xml
Modify the content of the mapred-site.xml file:
The above is mapred-site.xml minimal configuration, mapred-site.xml configuration can refer:
Http://hadoop.apache.org/docs/r2.2.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
Step 7 modify the profile yarn-site.xml, as shown below:
Modify the content of the yarn-site.xml:
The above content is the minimal configuration of the yarn-site.xml, the content of the yarn-site.xml file configuration can be referred:
Http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
[Spark Asia Pacific Research Institute Series] the path to spark practice-Chapter 1 building a spark cluster (step 5) (3)