spark scala tutorial

SparkstreamingcontextWe create Sparkstreamingcontext objects in a configuration-based manner:The third step is to create the spark streaming input data source:We configure the data source as local port 9999 (note that port requirements are not being used):Fourth step: As with the RDD programming, we program based on Dstream, because Dstream is the template that the RDD generates, and before spark streaming

Contents of this lesson:A thorough explanation of functional programming in 1:scalaScala functional programming in the 2:spark source code3: Cases and jobsFunctional Programming Begins:def fun1 (name:string) {println (name)}Assign a function name to a variable, then this variable is a function.Val Fun1_v = fun1_Visit Fun1_v ("Scala")Result: Scalaanonymous function: Parameter name = = point to function bodyV

was successful. This is followed by some basic grammar learning in Scala. val Variable declaration: declares a Val variable to hold the expression's computed result, which is immutable.Val result = 1 + 1 Subsequent constants are available for continuation, Eg:2 * result However, the Val constant declaration does not change its value, otherwise it returns an error. var variable declaration: DECLARE var variable, can change the refere

://master. Hadoop:8390/gradient_data/spam.data.txt ") Val Lines=Text.map { line=Line.split (" "). Map (_.todouble)} Val points= Lines.map (Parsepoint (_))//(Parsepoint (_)) looks the samevar w = Densevector.rand (Lines.first (). size-2) Val iterations= 100 for(I to iterations) {val Gradient= Points.map (P = = (1/(1 + exp (-P.Y * (w dot p.x)))-1) * P.Y *p.x). Reduce (_+_) W-=gradient} println ("Finish data loading, W num:" + w.length + "; W: "+W)}}Then on the M42n05 machine, the first use

Listen to Liaoliang's Spark 3000 disciple series lesson four Scala pattern matching and type parameters, summarized as follows:Pattern matching:def data (array:array[string]) {Array match{Case Array (a,b,c) = println (A+b+c)Case Array ("Spark", _*) =//matches an array with spark as the first elementCase _ = ...}}After-

First, spark frame previewMainly have Core, GraphX, MLlib, spark streaming, spark SQL and so on several parts.GRAPHX is a graph calculation and graph mining, in which the mainstream diagram calculation framework now has: Pregal, HAMA, giraph (these parts are in the form of hyper-step synchronization), and Graphlab and Spark