Spark machine learning environment to build _spark

First, spark environment to build 1.1 download spark

Download Address: http://spark.apache.org/downloads.html

After the download is complete decompression can be.
Add Spark's running directory to environment variables:

#Spark Home
Export spark_home=/usr/local/cellar/spark-2.1.0-bin-hadoop2.7
export path= $PATH: $SPARK _home/ Bin

Here I am using a simple local stand-alone version, running the example of calculation pi to test:

Run-example Org.apache.spark.examples.SparkPi

If all goes well, you can see the following results:

.......
17/10/11 10:59:06 INFO dagscheduler:job 0 finished:reduce at sparkpi.scala:38, took 0.895042 s
Pi is roughly 3.14413 57206786034 .....

Ii. Download and install Scala

Download Address: http://www.scala-lang.org/download/

Unzip, add Scala directory to environment variables:

#Scala Home
Export scala_home=/usr/local/cellar/scala-2.12.0
export path= $PATH: $SCALA _home/bin

To view the Scala version information:

Scala-version
Scala Code Runner version 2.12.0--Copyright 2002-2016, LAMP/EPFL and Lightbend, Inc.

Third, the idea of the installation of Scala Plug-ins

Open the Idea,config to find plugins:

Search Scala:
Iv. creating the SBT project in idea

New project, select SCALA->SBT:

To configure the project name and path:

New Scala Class:

Kind Select object, note, do not select class here.

Write a Hello World:

Run (if the above file chooses class, there is no Scala file running to the command):

Results:
V. Spark MAVEN Project

Write the spark program in the MAVEN project and add the coordinates of the spark:

<dependency>
      <groupId>org.apache.spark</groupId>
      <artifactid>spark-core_2.10 </artifactId>
      <version>2.1.0</version>
 </dependency>

To support Scala in Aven Engineering, you need to configure the Scala SDK. In idea, select File-> Project Structure->global libraries and add the Scala SDK:

Be sure to note the Scala version here. Vi. using spark to analyze user's shopping records

The following data is the user buys the product the record, the data column divides with the comma, sequentially is the user name, the commodity name, the price, saves the following data to the file Userpurchasehistory.csv:

John,iphone cover,9.99
john,headphones,5.49
jack,iphone cover,9.99 jill,samsung
Galaxy cover,8.95
Bob,ipad cover,5.49
Jack,ipad cover,5.49

Create a new Scala file: Userpurchaseapp.scala, write code:

Package Com.sprakmllib import Org.apache.spark.
  {sparkconf, sparkcontext}/** * Created by Bee on 17/10/10. */Object Userpurchaseapp {def main (args:array[string]): unit = {val sc = new Sparkcontext ("local[1]", "a" Val user_data = Sc.textfile ("/users/bee/documents/spark/sparkmllib/userpurchasehistory.csv"). Map (Line =>
    Line.split (",")). Map (Purchaserecord => (Purchaserecord (0), Purchaserecord (1), Purchaserecord (2))//Number of purchases

    Val numpurchase = User_data.count (); println ("Purchase number:" +numpurchase)//different customer for purchase of merchandise val uniqueusers = user_data.map {case (user, product, price) => US Er}.distinct (). Count () println ("Different customers who buy the goods: +uniqueusers)//gross income val totalrevenue = user_data.map {case (US Er, product, price) => price.todouble}.sum () println ("Gross income:" +totalrevenue)//Statistics best-selling product Val Productsbypop Ularity=user_data.map{case (User,product,price) => (product,1)}. Reducebykey (_+_). Collect (). SORTBY (-_._2) Val mostpopular=productsbypopularity (0) println ("Statistics Best selling Product:" +mostpopular)}}
 

Run Result:

Number of purchases: 6
different customers buy Goods:  4
Total income:  45.400000000000006
Statistics best-selling products:  (IPad cover,2)