Big Data Spark mushroom cloud prequel lesson 15th: Scala type parameter programming combat and spark source appreciation (study notes)

The 15th lesson: Scala type parameter programming combat and spark source appreciation

This lesson:

1. Use of the Scala type system in the spark source code
2. Scala type System programming operation combat

use of the Scala type system in the spark source codeClassof[rdd[_]] This is also a type system

This means that the type B must be at least a type ordering Scala type System programming operation CombatAs the largest of the type system, it is possible to limit the type, the type system in Scala, he is also the image. e.g. we can build person this category, now can build a what type of people, such as the Yi Millionaire this type of human beings

2. The types and methods of Scala, functions can be generics, and the generic type and method can be seen everywhere in the spark source, specifying the type of the entity at the time of the fact, such as Spark's core, most basic, and most important, the definition of the type of RDD in the structure is generic, RDD Almost all of the methods are generic, and the reason for this is that the RDD derives a lot of sub-categories, and the sub-category is equipped with various source and business logically edits.
   
   
3. For the bounds of the boundary, it is divided into the upper and lower bounds:
   
   1. Upper boundary upperbound: The generic type must be a type or a subclass of a type, and the language is : A new phenomenon in this is the qualification of the type, or the small to yourself) < Identifier:upper bound-the highest acceptable class type > 
      "A is less than B ... A is under B ... A is a subtype of B. "
   2. Lower boundary lowerbound: The generic type must be a type or a type of parent, the language method >: A new phenomenon in this is the qualification of the type, (or the big to yourself)   < Identifier:lower bound-the lowest acceptable class type;
      
      
      This example shows: the highest acceptable cl The type of the person is the person, so the required parameters must be a person or a child of person (that is, worker)
      
      If it is not a person or a worker, it will be wrong to edit
      
4. View Bounds, you can do some kind of mystical transformation, and you can convert your type into a target type without knowing it, you might think that the view Bounds is a complementary version of the upper and lower bounds, for example, in the core of the Sparkcontext Spark.T <% WritableThe code of the Code, which is shown by the T must be writable type (the highest acceptable class is writable), but T does not inherit to the writable interface, this time need to pass through the "Implicit"Way to actually do this.
   
   The first point: to write the inheritance structure of person, Worker and Dog,The worker is the person who inherits the person (the person is the parent, the worker is the child), and the Dog has no relationship with the person;
   The second point: what is the Type Variance of the class Cub, in this case [T <% person], which means to accept all the subclasses or the person who inherit the person, that is to say in this caseaccept the person or the child of the person, that is, the Worker；
   The third point: check the correctness of the parameters, if T itself is a person, that is, you can pass in person and worker, if T itself is a worker, it is only possible to pass in worker, but the requirement is to get into the dog, because this is the View Bound so can be usedImplicit conversionsImplicit,dog because there is no relationship with person, it will find the key word implicit to determine whether they can be converted into a person category!Implicit def Dog2person(Dog: Dog) = New Person (Dog.name), found that it will automatically convert into a person and then into the communicate method! Implicit conversion succeeded!
   
6. T:classtag, such as the RDD class Rdd[t:classtag in the Spark source code] This is also a type of conversion system, only when the type of information is not enough to compile, you need to use the JVM runtime to pass through the information to obtain a complete type of information, which It is very important in spark that the programming and running of the spark program is divided between Driver and Executor, and only the complete type information is known when it is run.
   
   
7. Covariance and contravariance: [-t] and [+t] e.g. Expert is a subclass of engineer, so the inverse covariant isthe sub-category of oneself/oneself; Inverter contravariant ishimself/herself above the parent。
   The Following are examples of covariance:
   
   The first point: to write the inheritance structure between Engineer and Expert, Expert is the successor Engineer (Engineer is the father, the expert is the child);
   Second point: Determine the Type of class meeting Variance is what, in this example is [+t], meaning is to accept all inherited t of the subclass or T itself, that is to say in this exampleaccept the Engineer or the Engineer of the child, namely the Expert；
   The third point: Determine what type of method needs to be passed in, and in this case it is required to pass inMeeting[engineer]；
   Point four: Check the correctness of the parameters, if T itself is Engineer, that is can be passed to Engineer and expert; If T itself is expert, it is only possible to pass in the expert, if there is this time pass in Engineer, will compile error!
   
   The following are examples of contravariance:
   The first point: to write the inheritance structure between Engineer and Expert, Expert is the successor Engineer (Engineer is the father, the expert is the child);
   Second point: Determine the Type of class meeting Variance is, in this example is contravariant [-t], meaning to accept all t itself or the parent of T, that is to say in this case onlyAccept Engineer;
   The third point: Determine what type of method needs to be passed in, and in this case it is required to pass inMeeting[engineer]；
   Point four: Check the correctness of the parameters, if the T itself is an expert, it is possible to pass in Engineer and expert; If T itself is Engineer, it is only possible to pass in Engineer; If you pass in an expert at this time, you will compile an error!
   
9. Context bound,t:ordering This syntax will be able to program Ordering[t] this way.
   

Reference:programming Scala Chapter 10Suppose A method takes an arguments of type list[anyref], can pass a List[strin G] Value (should a list[string] be considered as a subtypeLIST[ANYREF]. If true, this kind of variance is called covariance. We can also has types that was contravariant, where x[string] is a Super-typeof X[any]. For covariance type parameters, + are usedfor contravariant type parameters,-is used covariance invariance contravariant [+a]: Own/own sub-category [a]: self [-a]: their own/their own father is the difference between Type Variance, View Bound and Context Bound? Why do we need to apply the Type System in programming? What is the pros and cons?

Big Data Spark mushroom cloud prequel lesson 15th: Scala type parameter programming combat and spark source appreciation (study notes)