2nd. Scala object-oriented thorough mastery and spark source Sparkcontext,rdd reading Summary

Lesson 2nd: Scala's object-oriented mastery and spark source reading
Contents of this issue:
1 Scala's class, object in real combat
2 abstract classes and interfaces in Scala
3 comprehensive case and spark source code analysis

One: Define Class
Class hiscala{
private var name = "Spark"

Def sayname () {
println (name)
}
def getName = Name
}

In Scala, a variable is the same as a method in a class and can be assigned directly to a method.

There is a big difference between get and set in Scala and Get,set in Java:
In Scala, if private is defined in the program before a variable or method, Scala automatically generates a private get and set party for the variable or method
method, if no private is defined in the program before the variable or method, then Scala will automatically generate a public get and set method for the variable or method.

To create an object of a class
Val scal = new Hiscala
Scal.sayname ()
scal.sayname//If you do not pass the reference, it is equal to
Scal.getname

var name= "Spark"//If you do not add private, the default is to generate a public get and set method, externally adjustable

Private[this] Deep-seated access control over class objects

Constructors for classes
Note: All overloaded constructors must first call the default constructor, or call other overloaded constructors

Constructors for classes: overloading
def this (name:string)
{
This ()//Call the default constructor
This.name=name
}

def this (name:string,age:int) {
This (name)//Call other overloaded constructors
This.age=age
}

Associated Objects object:
In a Scala program, after defining a class, it is common to define an object of the same name, which is used to hold static members or methods, when we first tune
With this associated object, it executes the constructor of the object, except that the constructor does not accept arguments, and this constructor is only called once, and will not
In the call.
Definition: Object person{
println ("Scala")
Val salary=0.0
def getsalary=salary
}

The Apply method of the associated object: It is the factory method of the object of the current class, which controls the generation of objects.

Scala does not support multiple relays

Trait: A container for tool methods

Sparkcontext class Source parsing:

Sparkcontex is located in the project's source path \spark-master\core\src\main\scala\org\apache\spark\ Sparkcontext.scala, the source file contains the Sparkcontextclasss declaration and its associated object Sparkcontextobject

Class Sparkcontext extends the logging. Logging is a trait, which is a container for storing tool methods, trait encapsulation methods and fields. By mixing trait into classes, you can reuse them. A class that can inherit any number of trait, multiple trait inheritance is implemented with, and trait can be considered Java interface interface, very similar to interface. The only difference is that interface declares functions, but does not implement them, trait can implement functions.

The difference between trait and class is that trait does not have any class arguments, and the method that trait calls is dynamically bound.

There are several overloaded constructors in class Sparkcontext classes:
As follows:
def this (master:string, appname:string, conf:sparkconf) =
This (sparkcontext.updatedconf (conf, master, AppName))

@deprecated ("Passing in preferred locations have no effect at all, see SPARK-10921", "1.6.0")
def this (
Master:string,
Appname:string,
Sparkhome:string = NULL,
Jars:seq[string] = Nil,
Environment:map[string, String] = Map (),
Preferrednodelocationdata:map[string, Set[splitinfo]] = Map ()) =
{
This (sparkcontext.updatedconf (new sparkconf (), Master, AppName, Sparkhome, Jars, Environment))
if (preferrednodelocationdata.nonempty) {
Logwarning ("Passing in preferred locations have no effect at all, see SPARK-8949")
}
}

Val startTime = System.currenttimemillis ()
StartTime does not add Private,scala compile, starttime this variable is private, but it automatically generates its public get and set methods for external object invocation.

Private[spark] Val stopped:atomicboolean = new Atomicboolean (false)
Private[class_name] Specifies the class that can access the field, the level of access is stricter, and at compile time, the get and set methods are automatically generated, and the class_name must be the outer class of the currently defined class or class.

Private Def assertnotstopped (): Unit = {}
This method is a process because the return value is unit and no result is returned.

RDD Type source code Analysis:
Class Rdd It's an abstract class,

Private[spark] def conf = sc.conf
Private[class_name] Specifies the class that can access the field, the level of access is stricter, and at compile time, the get and set methods are automatically generated, and the class_name must be the outer class of the currently defined class or class.

The class Rdd class has a number of methods that add the final modifier, which means: This method or property, when the subclass inherits this class, is not writable, as follows:
Final Def dependencies:seq[dependency[_]] = {
Checkpointrdd.map (r = List (new onetoonedependency (R))). Getorelse {
if (Dependencies_ = = null) {
Dependencies_ = Getdependencies
}
Dependencies_
}
}

Final def partitions:array[partition] = {
Checkpointrdd.map (_.partitions). Getorelse {
if (Partitions_ = = null) {
Partitions_ = GetPartitions
}
Partitions_
}
}

Final def getnumpartitions:int = Partitions.length

2nd. Scala object-oriented thorough mastery and spark source Sparkcontext,rdd reading Summary