Spark1.0.0 pseudo distribution Installation Guide

? I. Instructions for downloading

Software preparation:

Spark-1.0.0-bin-hadoop1.tgz: spark1.0.0.

Scala-2.10.4.tgz download: Scala 2.10.4

Hadoop-1.2.1-bin.tar.gz: hadoop-1.2.1-bin.tar.gz

Jdk-7u60-linux-i586.tar.gz: Go to the official website to download on the line, this 1.7.x will do

Ii. Installation Steps

Hadoop-1.2.1 installation steps, see: http://my.oschina.net/dataRunner/blog/292584

1. decompress:

tar -zxvf scala-2.10.4.tgz mv  scala-2.10.4 scalatar -zxvf spark-1.0.0-bin-hadoop1.tgz mv spark-1.0.0-bin-hadoop1 spark

2. Configure environment variables:

Vim/etc/profile (Add the following content to the last line) Export hadoop_home_warn_suppress = 1 Export java_home =/home/big_data/jdkexport jre_home =$ {java_home}/jreexport class_path =.: $ {java_home}/lib: $ {jre_home}/libexport export =/home/big_data/export hive_home =/home/big_data/hiveexport scala_home =/home/big_data/scalaexport spark_home =/home/big_data/sparkexport Path =.: $ spark_home/bin: $ scala_home/bin: $ hive_home/bin: $ hadoop_home/bin: $ java_home/bin: $ path

3. Modify the spark spark-env.sh File

CD spark/confcp spark-env.sh.template spark-env.shvim spark-env.sh (just add the following to the last line) export java_home =/home/big_data/jdkexport scala_home =/home/big_data/scalaexport spark_master_ip = 192.168.80.100export spark_worker_memory = 200 Mexican export =/home/big_data/hadoop/Conf

Then the configuration is complete !!! (That's simple. Everyone knows this, but there are too few people sharing it)

3. Test Procedure

Hadoop-1.2.1 test steps, see: http://my.oschina.net/dataRunner/blog/292584

1. Verify Scala

[[email protected] ~]# scala -versionScala code runner version 2.10.4 -- Copyright 2002-2013, LAMP/EPFL[[email protected] ~]# [[email protected] big_data]# scalaWelcome to Scala version 2.10.4 (Java HotSpot(TM) Client VM, Java 1.7.0_60).Type in expressions to have them evaluated.Type :help for more information.scala> 1+1res0: Int = 2scala> :q

2. Verify spark (start hadoop-dfs.sh first)

[[Email protected] big_data] # cd spark [[email protected] spark] # cd sbin/start-all.sh (you can also start [[email protected] spark] $ sbin/start-master.sh either through HTTP: // master: 8080/See the corresponding interface [[email protected] spark] $ sbin/start-slaves.sh Park: // master: 7077 can be through http: // master: 8081/See the corresponding interface) [[email protected] spark] # JPs [[email protected] ~] # Jps4629 namenode (hadoop) 5007 master (SPARK) 6150 jps4832 secondarynamenode (hadoop) 5107 worker (SPARK) 4734 datanode (hadoop) can be accessed through http: // 192.168.80.100: 8080/See the corresponding interface [[email protected] big_data] # spark-shellspark assembly has been built with hive, including datanucleus jars on classpath14/07/20 21:41:04 info spark. securitymanager: Changing view ACLs to: root14/07/20 21:41:04 info spark. securit Ymanager: securitymanager: authentication disabled; UI ACLs disabled; users with view permissions: Set (Root) 14/07/20 21:41:04 info spark. httpserver: Starting HTTP server 14/07/20 21:41:05 info server. server: jetty-8.y.z-SNAPSHOT14/07/20 21:41:05 info server. abstractconnector: started [email protected]: 43343 welcome ______/__/__________//___\\/_\/_'/__/'_//___/. _/\ _, _/\ _ \ ve Rsion 1.0.0/_/using Scala version 2.10.4 (Java hotspot (TM) Client Vm, Java 1.7.0 _ 60 )... Scala> Val file = SC. textfile ("HDFS: // master: 9000/input") 14/07/20 21:51:05 info storage. memorystore: ensurefreespace (608) called with curmem = 31527, maxmem = 31138775014/07/20 21:51:05 info storage. memorystore: block broadcast_1 stored as values to memory (estimated size 608.0 B, free 296.9 MB) file: Org. apache. spark. RDD. RDD [Str Ing] = mappedrdd [5] At textfile at <Console>: 12 Scala> Val COUNT = file. flatmap (line => line. split ("")). map (WORD => (word, 1 )). reducebykey (_ + _) 14/07/20 21:51:14 info mapred. fileinputformat: total input paths to process: 1 count: Org. apache. spark. RDD. RDD [(string, INT)] = mappartitionsrdd [10] At performancebykey at <Console>: 14 Scala> count. collect () 14/07/20 21:51:48 info spark. sparkcontext: job finished: collec T at <Console>: 17, took 2.482381535 sres0: array [(string, INT)] = array (previously-registered, 1), (this, 3), (spark, 1), (it, 3), (original, 1), (than, 1), (its, 1), (previously, 1), (have, 2 ), (Upon, 1), (Order, 2), (whenever, 1), (It's, 1), (cocould, 3), (configuration, 1 ), (Master's, 1), (spark_daemon_java_opts, 1), (this, 2), (which, 2), (applications, 2), (register, 1 ), (doing, 1), (for, 3), (just, 2), (used, 1), (NY, 1), (go, 1), (equivalent, 1), (master, 4), (killing, 1), (time, 1), (availability ,, (stop-master.sh, 1), (process ., 1), (Future, 1), (node, 1), (The, 9), (workers, 1), (however, 1), (up, 2 ), (Details, 1), (not, 3), (recovered, 1), (process, 1), (enable, 3), (spark-env, 1 ), (enough, 1), (CAN, 4), (if, 3), (while, 2), (provided, 1), (BE, 5), (mode ., 1), (minute, 1), (when, 1), (all, 2), (written, 1), (store, 1), (enter, 1), (Then, 1), (as, 1), (officially, 1 )... scala> count. saveastextfile ("HDFS: // master: 9000/output") (Save the result to the/Output Folder on HDFS) Scala>: qstopping spark context. [[email protected] ~] # Hadoop FS-ls/found 3 itemsdrwxr-XR-X-root supergroup 0/home-RW-r -- 1 root supergroup 1722/inputdrwxr-XR-x -Root supergroup 0/output [[email protected] ~] # [Email protected] ~] # Hadoop FS-CAT/output/P *... (Mount, 1) (production-level, 1) (recovery )., 1) (workers/applications, 1) (perspective ., 1) (So, 2) (and, 1) (zookeeper, 2) (system, 1) (needs, 1) (property meaning, 1) (solution, 1) (seems, 1)

Now, the installation and testing are complete. This is the end of the Getting Started tutorial!

You can smile with excitement. Thanks, spark is so simple. (Pseudo distribution Oh, huh, for learning)

If you like this sharing spirit, please join us

Bytes ------------------------------------------------------------------------------------------------------------

Author: Jiang Zhonglian, one of the pioneers of data

QQ: 248087140

Motto:

You are awesome in your field,

And with a group of white ones,

Others will respect you from the heart.

-- Click to join us.