spark sample programs in scala

Val sc=new Sparkcontext (CONF) //The statement above is equivalent to Val sc=new sparkcontext ("local", "Testrdd") val data=sc.textfile ("e:// Hello.txt ")//Read local file data.flatmap (_.split (" "))//underscore is a placeholder, FlatMap is a way to manipulate rows, splitting the data that is being read in . Map ((_,1))// Convert each item to Key-value, and the data is Key,value is 1 . Reducebykey (_+_)//combine items with the same key into one . Col

Java switch-case (pair value)Scala is not only for values, but also for types, collections (map,list metadata matching), Object,classScala uses a lot of pattern matching (Match case)Scala's pattern match, which differs from the Java switch case:1. Not only can match the value, can match type2. Can match the collection of arraysAn array of the same array, the same length, and an array beginning with an elementAutomatic variable assignment for arrays of

What is Scala?　　S-Cala is a language designed to achieve scaleable language. Officially, it is called the mixed language of object-oriented language and functional language.Scala can be seamlessly spliced with Java programs, because Scala files are compiled as. class files and run on the JVM.Spark was developed by Scala.Scala installation?　　Here's the process of

[Introduction to Apache spark Big Data Analysis (i) (http://www.csdn.net/article/2015-11-25/2826324) Spark Note 5:sparkcontext,sparkconf Spark reads HBase Scala's powerful collection data operations example Some RDD operations and transformations in spark # Create Textfilerdd val textfile = Sc.textfile ("readme.md") Te

("")))Val hamfeatures = ham.map (email + tf.transform (email.split ("")))Create labeledpoint datasets for positive (spam) and negative (ham) examples.Val positiveexamples = spamfeatures.map (features = Labeledpoint (1, features))Val negativeexamples = hamfeatures.map (features = Labeledpoint (0, features))Val Trainingdata = positiveexamples + + negativeexamplesTrainingdata.cache ()//cache data since Logistic Regression is an iterative algorithm.Create a Logistic Regression learner which uses th

file system, read the file from HDFs by default classification and function of spark operators value type transformation operator input partition and output partition one-to- one Map FlatMap mappartitions Glom input partition and output partition many-to-one type Union Cartesian input partition and output partition Many-to-many types GroupBy output partition as input partition subset type Filter distinct Subtract

WordCount written by Java"). Setmaster ("local")); /*** 2nd step: Create a Sparkcontext Object * Sparkcontext is the only entry for all the functions of the Spark program, whether in Scala, Java, Python *, R, etc. must have a sparkcontext (different Language specific class name is different, if Java is javasparkcontext) * Sparkcontext Core role: Initialize the Spark