Spark Basic operations

Start start-dfs.sh start-yarn.sh
Change host name su root CD hostname localhost
See Backstage service JPS 29456 NameNode 29863 secondarynamenode 30220 ResourceManager 30718 JPS 29548 DataNode 30307 NodeManager
Spark-shell local Spark-shell--driver-memory 512M--executor-memory 512M driver and executor memory, default 1G
Submit the application to the Spark cluster Spark-submit--class class in the jar package path [--executor-memory 256M] jar Package For example: Spark-submit--master yarn--class sparks Treaming. Sparksteaming Sparkstreaming.jar
Hadoop-shell yarn Spark-shell--master yarn--driver-memory 128M--executor-memory 128M http://localhost:8088/proxy/ Application_1510121456507_0002/spark-submit--master yarn--class sparkstreaming. Sparksteaming Sparkstreaming.jar
Operations such as the Spark Shuffle map, Shuffle, and reduce group\reducebykey cause shuffle,shuffle because of the data transfer between different servers, so performance is low
HDF Operation Hadoop Fs-mkdir/sougou Hadoop fs-put sogou01.txt/sougou hadoop fs-ls/sougou Scala Transformations:map, filter , FlatMap, sample, Groupbykey, Reducebykey, Union, join, Cogroup, Mapvalues,sort,partionby
Actions:count, collect, reduce, lookup, save
Sparkcontext is the abstraction of driver in the program.
var rdd=sc.textfile ("File:///home/hadoop/derby.log") var Wordcount=rdd.flatmap (_.split (""). Map (x=> (x,1)). Reducebykey (_+_) wordcount.take map (x=> (x,1)): Narrow dependency Reducebykey: Wide Dependency
Spark SQL modifies Hadoop/...spark.../conf/hive_site.xml, and finally adds a property to copy this file to the hadoop/apache...hive...bin/conf/command line execution: Nohup Hive--service metastore>metastore.log 2>&1 & jps/home/hadoop/spark-1.5.1-bin-hadoop2.4/sbin/ Start-thriftserver.sh/home/hadoop/spark-1.5.1-bin-hadoop2.4/bin/beeline!connect jdbc:hive2://localhost:10000