Spark Quick Start Guide

Respect for copyright. What is http://blog.csdn.net/macyang/article/details/7100523-Spark?
Spark is a MapReduce-like cluster computing framework designed to support
Low-latency iterative jobs and interactive use from an interpreter. It is
Written in Scala, a high-level language for the JVM, and exposes a clean
Language-integrated syntax that makes it easy to write parallel jobs.
Spark runs on top of the Mesos cluster manager.

-Spark?

Git clone git: // github.com/mesos/spark.git

-Spark compilation and running?

1) scala 2.9 + (add bin to PATH or set the SCALA_HOME environment variable)

2) spark supports local mode and cluster mode, and mesos does not need to be installed in local mode.

3) To Run spark on the cluster, install mesos

4) use spark's built-in sbt for compilation/packaging: sbt/sbt compile, sbt/sbt assembly

5) run the spark program using the run script provided by spark.

-Verify that the spark environment is OK?

Run the following command in the spark directory:

1) local single thread:./run spark. examples. SparkPi local

2) local multi-core:./run spark. examples. SparkPi local [2]

3) mesos local master:./run spark. examples. SparkPi master @ localhost: 5050

-What does Spark Programming Guide introduce?

1) put the Spark jar package (sbt/sbt assembly) into CLASSPATH

2) Spark Application can run on local or mesos

3) Spark provides two types of RDD: parallreceived Collections and Hadoop Datasets. RDD can
Fault-tolerant is supported to restore the partition loss caused by node crash.

4) the RDD supports two types of Operation: transformation and action.
Lazy operations can only trigger transformations when an action is actually called.

5) Spark provides two types of shared variables: Broadcast Variables and Accumulators.
Broadcast variables distributes a share variable to each machine, instead of every task by default;
However, accumulator only supports count and sum addition operations, and only the dirver program can obtain its value.

-How to write some spark applications?

Look at some spark examples, such as: http://www.spark-project.org/examples.html

Https://github.com/mesos/spark/tree/master/examples

-What should I do if I encounter a problem?

First of all, it is a problem that google encounters. If it still cannot be solved, you can go to spark google group to ask the author a question:
Http://groups.google.com/group/spark-users? Hl = en

-What should I do if I want to have a deep understanding of spark?

Reading spark theory paper: http://www.eecs.berkeley.edu/Pubs/TechRpts/2011/EECS-2011-82.pdf

Spark Source Code: https://github.com/mesos/spark