Akka Source Analysis-akka-streams-materializer (1)

This blog gradually analyze the source of Akka streams, of course, must be gradual, and estimates will be divided into many, after all, Akka streams is more complex.

Implicit val system = Actorsystem ("QuickStart") implicit val Materializer = Actormaterializer ()

When using streams-related APIs, the above two objects must be created. Actorsystem no longer said, we came to see Actormaterializer.

Actormaterializer put a flow-calculated blueprint (outline, blueprint?) ) into a running stream, which is simply to compile the streaming API provided by Akka. It inherits a class and a trait. Materializerloggingprovider will not read, is to provide log-related functions.

/** * Materializer SPI (Service Provider Interface) * Binary compatibility is not guaranteed on Materializer internals. * * Custom Materializer implementations should be aware, the Materializer SPI * is not yet final and could change in pat CH releases of Akka. Does not impact end-users of Akka streams, only implementors of the custom materializers, * with whom The Akka team co-ordinates such changes. * * Once The SPI is final this notice'll be removed. */abstract class Materializer {/** * the ' nameprefix ' shall is used for deriving the names of processing * entities That is created during materialization.   This was meant to aid * logging and failure reporting both during materialization and while the * stream is running.  */def Withnameprefix (name:string): Materializer/** * This method interprets the given Flow description and creates The running * stream. The result can be highly implementation specific, ranging from * local actor chainsto remote-deployed processing networks. */def Materialize[mat] (Runnable:graph[closedshape, Mat]): Mat/** * This method interprets the given Flow Descriptio N and creates the running * stream using a explicitly provided [[Attributes]] as top level (least specific) Attributes   That * 'll is defaults for the materialized stream. * The result can be highly implementation specific, ranging from local actor chains to remote-deployed * processing NETW   Orks. */def Materialize[mat] (Runnable:graph[closedshape, Mat), @deprecate Dname (' Initialattributes) defaultattributes:attributes): Mat/** * Running A flow graph would require execution RESOURC ES, as would computations * within Sources, Sinks, etc. this [[[Scala.concurrent.ExecutionContextExecutor]] * Can is used by parts of the flow to submit processing jobs for exec   Ution, * run future callbacks, etc. * * Note that this was not necessarily the same execution conteXT The stream operator itself is running on. */Implicit def executioncontext:executioncontextexecutor/** * Interface for operators the need timer services for Their functionality.   Schedules A * Single task with the given delay. * * @return A [[akka.actor.Cancellable]] that allows cancelling the timer.   Cancelling is best effort, if the event * has been already enqueued it won't have a effect. */def scheduleonce (Delay:finiteduration, task:runnable): cancellable/** * Interface for operators that need timer Services for their functionality.   Schedules A * Repeated task with the given interval between invocations. * * @return A [[akka.actor.Cancellable]] that allows cancelling the timer.   Cancelling is best effort, if the event * has been already enqueued it won't have a effect. */def scheduleperiodically (Initialdelay:finiteduration, Interval:finiteduration, task:runnable): Cancellable}

Materializer is very important, so the full code is posted here. Materializer is an SPI (service Provider Interface), Materializer internal does not guarantee binary compatibility, that is, the version may not be compatible. But this definition is rather strange, since all the methods are not implemented, it is not used trait more appropriate? Why is it an abstract class?

Materializer Altogether 6 methods, we look one by one.

Withnameprefix is used when deriving processing entities, in simple terms, each computational entity (source/sink/flow/runnablegraph) in the stream is mapped into an actor, and the actor names need a prefix, The Withnameprefix is used to set this prefix.

The materialize method has two implementations. This method is used to interpret a given flow definition and create a running stream. The result of the creation is highly dependent on the specific implementation, possibly the local actor chain or the process of remote deployment.

The ExecutionContext is used to provide an asynchronous execution environment.

Scheduleonce/scheduleperiodically is used to provide timed scheduling functions, after all, there are some operations that require time service.

Materializer seems very simple, only the above several interfaces, but its core interface is the materialize method, after all, this is used to compile Akka stream. If you've ever used storm, you know that storm has a topology compiler. Its function is similar to this one.

We'll see Actormaterializer again.

The Actormaterializer also provides three more important interfaces: Actorof/system/supervisor. Where Actorof receives materializationcontext, props as a parameter, creates a actor;system that returns Actormaterializer associated with Actorsystem ; Supervisor function is studied later. Materializationcontext This parameter is still interesting and can be understood as a physical (compiled stream) context.

/** * Context parameter to the ' create ' methods of sources and sinks. * * INTERNAL API */@InternalApiprivate [Akka] Case class Materializationcontext (  materializer:        Materializer,  effectiveattributes:attributes,  islandname:          String)

Materializationcontext altogether three variables, Materializer no longer said, is the current context associated Materializer. Effectiveattributes are used to provide parameters, but are encapsulated as attributes. Islandname is more interesting, simply from the name to translate, it is "island names." What is the island, then? After the analysis, you will also encounter this "island", we have a little attention here on the line.

In addition, in Materializer this abstract class materialize method has a parameter that we need to study: Runnable:graph[closedshape, Mat].

/** * Not intended to being directly extended by user classes * * @see [[akka.stream.stage.GraphStage]] */trait Graph[+s < : Shape, +m]

Graph is a trait, it has two types of parameters: s/m. Can get a good name, SM, haha. where S is the subclass of shape, from the name it seems to be the shape of graph. A simple analysis of graph's main code reveals that it has two methods that are particularly important: shape, traversalbuilder. Where shape returns the shape of the graph, although we don't yet know what the shape is, but Closedshape is a form of it; Traversalbuilder

/**   * INTERNAL API.   *   * Every materializable element must is backed by a stream layout module   */  Private[stream] def Traversalbuil Der:traversalbuilder

Traversalbuilder returns a Traversalbuilder, commenting that each element that is physically capable must be supported by the Flow layout module. Traversalbuilder in terms of name, it is a traversal compiler. Estimation is a topological sort.

Let's see what shape is.

A shape describes the entrance and exit of graph, and according to the philosophy that graph is the blueprint for free reuse, it is important that graph be linked from the outside, otherwise it is just a black box. Let's just say it's used to define the input and output of graph. And the two most important method of this trait is inlets/outlets.

  /**   * Scala api:get a list of all input ports   */  def inlets:immutable. Seq[inlet[_]]  /**   * Scala api:get A list of all output ports   */  def outlets:immutable. Seq[outlet[_]]

What about the inlet and outlet?

Final class Inlet[t] Private (Val s:string) extends InPort {def carboncopy (): inlet[t] = {val in = Inlet[t] (s) i   N.mappedto = this in}/** * INTERNAL API.    */def As[u]: inlet[u] = This.asinstanceof[inlet[u]] Override def tostring:string = s + "(" + This.hashcode + S ")" + (if (mappedto eq this) "" Else S "mapped to $mappedTo")} /** * An input port of a streamlayout.module. This is the type logically belongs * into the Impl package but must live here due to how ' sealed ' works. * It is also used in the Java DSL for "untyped inlets" as a work-around * for otherwise unreasonable existential types. */sealed abstract class InPort {self:inlet[_]⇒final override def Hashcode:int = Super.hashcode final override def E Quals (that:any): Boolean = this eq that.asinstanceof[anyref]/** * INTERNAL API */@volatile Private[stream] Var ID : Int =-1/** * INTERNAL API */@volatile Private[stream] var mappedto:inport = this/** * INTERNAL API */P Rivate[stream] def inlet:inlet[_] = this} 

Inlet does not seem to provide any method ah, seems to be more important at present, there is only one ID field, the other variables are returned, copied.

It seems that the analysis here, Materializer, Shape, graph are also more abstract, but also to see the concrete implementation of Materializer, after all, it is only a trait.

def apply (Materializersettings:actormaterializersettings, nameprefix:string) (implicit context:actorreffactory): Actormaterializer = {    val haveshutdown = new Atomicboolean (FALSE)    val system = Actorsystemof (context)    new Phasedfusingactormaterializer (      system,      materializersettings,      system.dispatchers,      Actorofstreamsupervisor (materializersettings, Context, Haveshutdown),      Haveshutdown,      flownames (System). Name.copy (Nameprefix))  }

By analyzing Actormaterializer's apply method, we found that the final call to the above version of apply, you can see, eventually created the Phasedfusingactormaterializer, and is new out, So this class must be a concrete class, which means that all methods and fields have corresponding implementations and definitions. Phasedfusingactormaterializer's name is actually quite interesting, literal translation is a segmented fuse actor materialized device. Paragraph good understanding, the fuse does not know how to understand, or translation is wrong? Haha, I don't know either.

@InternalApi Private[akka] Case class Phasedfusingactormaterializer (  system:                Actorsystem,  override Val Settings:actormaterializersettings,  dispatchers:           Dispatchers,  Supervisor:            actorref,  Haveshutdown:          Atomicboolean,  flownames:             

Phasedfusingactormaterializer is actually a case class, and what else is new? It inherits the Extendedactormaterializer. Extendedactormaterializer source code is no longer posted out, it is to re-cover the Actormaterializer of several methods, and implemented a method actorof. We're going to see this actorof.

@InternalApi Private[akka] Override def actorof (Context:materializationcontext, props:props): Actorref = {    val Effe Ctiveprops = props.dispatcher Match {case      dispatchers.defaultdispatcherid⇒        props.withdispatcher ( Context.effectiveattributes.mandatoryattribute[actorattributes.dispatcher].dispatcher) Case      Actorattributes.iodispatcher.dispatcher⇒        //This one was actually not a dispatcher but a relative config key pointing containing the actual dispatcher name        Props.withdispatcher (settings.blockingiodispatcher) case      _⇒props    }    actorof (Effectiveprops, context.islandname)  }

In fact, it replaces the value in the props with a specific dispatcher, and then calls another actorof to create the actor.

@InternalApi Private[akka] def actorof (Props:props, name:string): Actorref = {    Supervisor Match {case      Ref:loca Lactorref⇒        ref.underlying.attachChild (props, name, Systemservice = false) Case      Ref:repointableactorref⇒        if (ref.isstarted)          ref.underlying.asinstanceof[actorcell].attachchild (props, name, Systemservice = False)        else {          implicit val timeout = Ref.system.settings.CreationTimeout          val f = (supervisor? Streamsupervisor.materialize (props, name)). Mapto[actorref]          Await.result (f, timeout.duration)        }      Case Unknown⇒        throw new IllegalStateException (S "Stream supervisor must is a local actor, was [${unknown.getclass.get Name}] ")    }  

This actorof we are not in-depth analysis, anyway, is to create the actor. Let's look at the implementation of materialize in Phasedfusingactormaterializer.

Override Def Materialize[mat] (Graph:graph[closedshape, Mat], defaultattributes:attributes, default Phase:phase[any], Phases:map[islandtag, Phase[any]): Mat = {if (isshutdown) throw new Illegalsta Teexception ("Trying to materialize stream after materializer have been shutdown") Val islandtracking = new Islandtrackin g (Phases, settings, defaultattributes, Defaultphase, this, Islandnameprefix = Createflowname () + "-") var Current:trav Ersal = graph.traversalBuilder.traversal val attributesstack = new Java.util.arraydeque[attributes] (8) Attributessta Ck.addlast (Defaultattributes and graph.traversalBuilder.attributes) val traversalstack = new java.util.arraydeque[      Traversal] (+) Traversalstack.addlast (current) Val matvaluestack = new Java.util.arraydeque[any] (8) if (Debug) { println (S "---materializing layout:") traversalbuilder.printtraversal (current) println (S "---Start material Ization ")}//Due to How Concat works, we need a stack.    This probably can is optimized for the most common cases.        while (!traversalstack.isempty) {current = Traversalstack.removelast () and (current NE emptytraversal) {            var nextstep:traversal = emptytraversal Current Match {case materializeatomic (mod, outtoslot) ⇒ if (Debug) println (S "materializing module: $mod") val matandstage = IslandTracking.getCurrentPhase.materi            Alizeatomic (mod, attributesstack.getlast) val logic = matandstage._1 val matvalue = matandstage._2 if (Debug) println (S "materialized value is $matValue") matvaluestack.addlast (Matvalue) v Al Stageglobaloffset = Islandtracking.getcurrentoffset wireinlets (islandtracking, mod, logic) Wireou Tlets (islandtracking, mod, logic, Stageglobaloffset, Outtoslot) if (Debug) println (S "PUSH: $matValue = = $mat   Valuestack ") Case Concat (first, next) ⇒         if (next ne emptytraversal) traversalstack.add (next) NextStep = First case pop⇒v Al popped = Matvaluestack.removelast () if (Debug) println (S "pops: $popped = = $matValueStack") Case P Ushnotused⇒matvaluestack.addlast (notused) if (Debug) println (S "push:notused = $matValueStack" ) Case Transform:transform⇒val prev = matvaluestack.removelast () Val result = Transform ( prev) matvaluestack.addlast (Result) if (Debug) println (S "TRFM: $matValueStack") Case Compos            E:compose⇒val second = Matvaluestack.removelast () val first = Matvaluestack.removelast () Val result = Compose (first, second) matvaluestack.addlast (result) if (Debug) println (S "COMP: $mat            Valuestack ") Case pushattributes (attr) ⇒attributesstack.addlast (Attributesstack.getlast and attr) if (Debug) println (S "ATTR PUSH: $attr") Case Popattributes⇒attributesstack.removelast () if (Debug) printl          N (S "ATTR POP") case Enterisland (tag) ⇒islandtracking.enterisland (tag, attributesstack.getlast)    Case Exitisland⇒islandtracking.exitisland () case _⇒} current = NextStep} } def shutdownwhilematerializingfailure = new IllegalStateException ("Materializer shutdown while materializing St Ream ") try {islandTracking.getCurrentPhase.onIslandReady () Islandtracking.allnestedislandsready () if (D Ebug) println ("---finished materialization") Matvaluestack.peeklast (). Asinstanceof[mat]} finally {if (issh Utdown) throw Shutdownwhilematerializingfailure}}

Because this method is too important, I have posted the entire code. We found that it has two parameters that have not been seen: Defaultphase:phase[any], Phases:map[islandtag, Phase[any]]. This involves two types of Islandtag, Phase. Phase better understanding is the stage, that Islandtag is what? The "Island" label? What is "island"?!

@DoNotInherit Private[akka] Trait Phase[m] {  def apply (    settings:            actormaterializersettings,    Effectiveattributes:attributes,    materializer:        phasedfusingactormaterializer,    islandname:          String): Phaseisland[m]}

Phase This trait there is only one apply, he is according to each parameter, returned to the Phaseisland, Stage Island?

@DoNotInherit Private[akka] Trait Phaseisland[m] {  def name:string  def materializeatomic (mod:atomicmodule[ Shape, any], attributes:attributes): (M, any)  def assignport (In:inport, Slot:int, logic:m): Unit  def Assignpor T (Out:outport, Slot:int, logic:m): Unit  def createpublisher (Out:outport, logic:m): Publisher[any]  def takepub Lisher (Slot:int, Publisher:publisher[any]): Unit  def onislandready (): unit}

Phaseisland There are two ways to be very important: CreatePublisher, Takepublisher. Why is it important? Because it seems to be creating publisher, what is Publisher? Flipping through my last blog? This is the interface inside the Reactivestreams, which is important because it involves the underlying.

There is another islandtag, what is this?

@DoNotInherit Private[akka] Trait Islandtag

What elements are not, is it only used to do type matching? The ghost knows. It is estimated that the island is labeled.

There are a lot of materialize method code, logic is complex, but there are several important points to note. It creates a islandtracking and accesses the Graph.traversalbuilder through a while loop, doing some operations on islandtracking, and finally calls the two methods of islandtracking.

IslandTracking.getCurrentPhase.onIslandReady () Islandtracking.allnestedislandsready ()

So islandtracking and these two methods are very important, because the two methods of this class after the call, the flow will be able to run properly ah.

@InternalApi Private[akka] class islandtracking (  val phases:       Map[islandtag, Phase[any]],  val settings:     actormaterializersettings,  attributes:       attributes,  defaultphase:     Phase[any],  Val Materializer:phasedfusingactormaterializer,  

Islandtracking Island tracking, incredibly a line of comments are not, this let me how to analyze ah, hemp eggs!

Because of the time, the first analysis here today, it is obvious Akka streams code is very complex, see the source is very difficult, I hope I can still watch. Ha ha.