Spark closures and serialization

This article source: http://blog.csdn.net/bluishglc/article/details/50945032 prohibited any form of reprint, otherwise will be entrusted to csdn the official maintenance rights.

Spark's official documentation repeatedly emphasizes the actions that will be performed on the RDD, whether they are a function or a piece of code, they are "closures", and Spark distributes the closure to the individual worker nodes, which involves a problem that is easily overlooked: the "serialization" of closures.

Obviously, closures are stateful, which mainly refers to the free variables it involves and the other variables that the free variables depend on, so Before passing a simple function or a piece of code (that is, a closure) to an operation like Rdd.map, Spark needs to retrieve all of the variables involved in the closure (including passing dependent variables), and correctly serialize the variables before passing them to the worker node and deserializing them for execution. If any of the variables involved do not support serialization or do not indicate how to serialize yourself, you will encounter such an error:

Org.apache.spark.SparkException:Task not serializable

In the following example, we continue to receive JSON messages from Kafka and parse the strings into corresponding entities in spark-streaming:

<code class= "Language-scala hljs has-numbering" style= "display:block; padding:0px; Color:inherit; Box-sizing:border-box; font-family: ' Source Code Pro ', monospace;font-size:undefined; White-space:pre; border-radius:0px; Word-wrap:normal; background:transparent; " ><span class= "Hljs-class" style= "Box-sizing:border-box;" ><span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >object</span> <span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >App</span> {</span> <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box ;" >private</span> <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >val</span> config = configfactory.load (<span class= "hljs-string" style= "Color:rgb (0, 136, 0); Box-sizing: Border-box; " > "my-streaming.conf" </span>) <span class= "Hljs-class" style= "Box-sizing:border-box;" &Gt;<span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >case</span> <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >class</span> <span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >Person</span> <span class= "Hljs-params" style= "Color:rgb (102, 0, 102); Box-sizing:border-box; " > (firstname:string,lastname:string) </span></span> <span class= "Hljs-keyword" style= "Color:rgb (0 , 0, 136); Box-sizing:border-box; " >def</span> Main (args:array[string]) {<span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); box-s Izing:border-box; " >val</span> Zkquorum = config.getstring (<span class= "hljs-string" style= "Color:rgb" (0, 136, 0); Box-sizing: Border-box; " > "Kafka.zkquorum" </span>) <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >val</span> myTopic = confiG.getstring (<span class= "hljs-string" style= "Color:rgb (0, 136, 0); box-sizing:border-box;" > "Kafka.mytopic" </span>) <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >val</span> MyGroup = config.getstring (<span class= "hljs-string" style= "Color:rgb" (0, 136, 0); Box-sizing: Border-box; " > "Kafka.mygroup" </span>) <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >val</span> conf = <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >new</span> sparkconf (). Setappname (<span class= "hljs-string" style= "Color:rgb (0, 136, 0); Box-sizing: Border-box; " > "my-streaming" </span>) <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >val</span> SSC = <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >new</span> StreamingContext (conf, SecOnds (<span class= "Hljs-number" style= "Color:rgb (0, 102, 102); box-sizing:border-box;" >1</span>) <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >val</span> lines = Kafkautils.createstream (SSC, Zkquorum, MyGroup, Map (MyTopic-<span class= " Hljs-number "style=" Color:rgb (0, 102, 102); Box-sizing:border-box; " >1</span>) <span class= "hljs-comment" style= "Color:rgb (136, 0, 0); Box-sizing:border-box; " >//this Val was a part of closure, and it's not serializable!</span> implicit <span class= "Hljs-keyword "Style=" Color:rgb (0, 0, 136); Box-sizing:border-box; " >val</span> formats = defaultformats <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >def</span> parser (json:string) = Parse (JSON). Extract[person].firstname Lines.map (_._2). Map (parser). Prin T .... Ssc.start () Ssc.awaitterminationortImeout (<span class= "Hljs-number" style= "Color:rgb (0, 102, 102); box-sizing:border-box;" >10000</span>) Ssc.stop ()}}</code><ul class= "pre-numbering" style= "Box-sizing:border-box ; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li><li style= "Box-sizing:border-box; padding:0px 5px; " >2</li><li style= "Box-sizing:border-box; padding:0px 5px; " >3</li><li style= "Box-sizing:border-box; padding:0px 5px; " >4</li><li style= "Box-sizing:border-box; padding:0px 5px; " >5</li><li style= "Box-sizing:border-box; padding:0px 5px; " >6</li><li style= "Box-sizing:border-box; padding:0px 5px; " >7</li><li style= "Box-sIzing:border-box; padding:0px 5px; " >8</li><li style= "Box-sizing:border-box; padding:0px 5px; " >9</li><li style= "Box-sizing:border-box; padding:0px 5px; " >10</li><li style= "Box-sizing:border-box; padding:0px 5px; " >11</li><li style= "Box-sizing:border-box; padding:0px 5px; " >12</li><li style= "Box-sizing:border-box; padding:0px 5px; " >13</li><li style= "Box-sizing:border-box; padding:0px 5px; " >14</li><li style= "Box-sizing:border-box; padding:0px 5px; " >15</li><li style= "Box-sizing:border-box; padding:0px 5px; " >16</li><li style= "Box-sizing:border-box; padding:0px 5px; " >17</li><li style= "Box-sizing:border-box; padding:0px 5px; " >18</li><li style= "Box-sizing:border-box; padding:0px 5px; " >19</li><li style= "Box-sizing:border-box; padding:0px 5px; " >20</li><li style= "Box-sizing:border-box; padding:0px 5px; " >21</li></ul> 

This code, when executed, will report the following error:

Org.apache.spark.SparkException:Task not serializable
caused by:java.io.NotSerializableException: org.json4s.defaultformats$

The crux of the problem is that closures have no way of serializing. In this case, the scope of the closure is: The function parser and the implicit argument it relies on: formats, and the problem is on this implicit argument, whose type is defaultformats, which does not provide a description of the serialization and deserialization itself, Therefore, spark cannot serialize the formats and cannot push the task to the remote execution.

The implicit parameter formats is prepared for extract, and its argument list is as follows:

<code class= "Hljs markdown has-numbering" style= "display:block; padding:0px; Color:inherit; Box-sizing:border-box; font-family: ' Source Code Pro ', monospace;font-size:undefined; White-space:pre; border-radius:0px; Word-wrap:normal; background:transparent; " >org.json4s.extractablejsonastnode#extract[<span class= "Hljs-link_label" style= "Box-sizing:border-box;" >a</span>] (<span class= "Hljs-link_url" style= "Box-sizing:border-box;" >implicit formats:formats, mf:scala.reflect.manifest[a]</span>): A = ... </code><ul class= " Pre-numbering "style=" Box-sizing:border-box; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul>

The

finds the root cause of the problem and resolves it. In fact, we don't need to serialize the formats at all, and for us it's stateless. So, we just need to declare it as a global static variable to bypass serialization. So the way to change is simply to move the declaration of implicit Val formats = Defaultformats from inside the method to the field location of the App object.

<code class= "Language-scala hljs has-numbering" style= "display:block; padding:0px; Color:inherit; Box-sizing:border-box; font-family: ' Source Code Pro ', monospace;font-size:undefined; White-space:pre; border-radius:0px; Word-wrap:normal; background:transparent; " ><span class= "Hljs-class" style= "Box-sizing:border-box;" ><span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >object</span> <span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >App</span> {</span> <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box ;" >private</span> <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >val</span> config = configfactory.load (<span class= "hljs-string" style= "Color:rgb (0, 136, 0); Box-sizing: Border-box; " > "my-streaming.conf" </span>) <span class= "Hljs-class" style= "Box-sizing:border-box;" &Gt;<span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >case</span> <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >class</span> <span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >Person</span> <span class= "Hljs-params" style= "Color:rgb (102, 0, 102); Box-sizing:border-box; " > (firstname:string,lastname:string) </span></span> <span class= "hljs-comment" style= "Color:rgb ( 136, 0, 0); Box-sizing:border-box; " >//as Object field, Global, static, no need to serialize</span> implicit <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >val</span> formats = defaultformats <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >def</span> Main (args:array[string]) {<span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); box-s Izing:boRder-box; " >val</span> Zkquorum = config.getstring (<span class= "hljs-string" style= "Color:rgb" (0, 136, 0); Box-sizing: Border-box; " > "Kafka.zkquorum" </span>) <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >val</span> myTopic = config.getstring (<span class= "hljs-string" style= "Color:rgb" (0, 136, 0); Box-sizing: Border-box; " > "Kafka.mytopic" </span>) <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >val</span> MyGroup = config.getstring (<span class= "hljs-string" style= "Color:rgb" (0, 136, 0); Box-sizing: Border-box; " > "Kafka.mygroup" </span>) <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >val</span> conf = <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >new</span> sparkconf (). Setappname (<span class= "hljs-string" style= "Color:rgb (0, 136, 0); BoX-sizing:border-box; " > "my-streaming" </span>) <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >val</span> SSC = <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >new</span> StreamingContext (conf, Seconds (<span class= "Hljs-number" style= "Color:rgb (0, 102, 102); Box-sizing:border-box; " >1</span>) <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >val</span> lines = Kafkautils.createstream (SSC, Zkquorum, MyGroup, Map (MyTopic-<span class= " Hljs-number "style=" Color:rgb (0, 102, 102); Box-sizing:border-box; " >1</span>) <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >def</span> parser (json:string) = Parse (JSON). Extract[person].firstname lines. Map (_._2). Map (parser). Print ... ssc.start () ssc.awaitterminationortimeout (&ltSpan class= "Hljs-number" style= "Color:rgb (0, 102, 102); Box-sizing:border-box; " >10000</span>) Ssc.stop ()}}</code><ul class= "pre-numbering" style= "Box-sizing:border-box ; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li><li style= "Box-sizing:border-box; padding:0px 5px; " >2</li><li style= "Box-sizing:border-box; padding:0px 5px; " >3</li><li style= "Box-sizing:border-box; padding:0px 5px; " >4</li><li style= "Box-sizing:border-box; padding:0px 5px; " >5</li><li style= "Box-sizing:border-box; padding:0px 5px; " >6</li><li style= "Box-sizing:border-box; padding:0px 5px; " >7</li><li style= "Box-sizing:borDer-box; padding:0px 5px; " >8</li><li style= "Box-sizing:border-box; padding:0px 5px; " >9</li><li style= "Box-sizing:border-box; padding:0px 5px; " >10</li><li style= "Box-sizing:border-box; padding:0px 5px; " >11</li><li style= "Box-sizing:border-box; padding:0px 5px; " >12</li><li style= "Box-sizing:border-box; padding:0px 5px; " >13</li><li style= "Box-sizing:border-box; padding:0px 5px; " >14</li><li style= "Box-sizing:border-box; padding:0px 5px; " >15</li><li style= "Box-sizing:border-box; padding:0px 5px; " >16</li><li style= "Box-sizing:border-box; padding:0px 5px; " >17</li><li style= "Box-sizing:border-box; padding:0px 5px; " >18</li><li style= "Box-sizing:border-box; padding:0px 5px; " >19</li><li style= "Box-sizing:border-box; padding:0px 5px; " >20</li><li style= "Box-sizing:border-box; padding:0px 5px; " >21</li><li StYle= "Box-sizing:border-box; padding:0px 5px; " >22</li></ul>

Another good example is provided here:

This example illustrates a scenario that solves a similar problem: "Copy a class member variable to a closure", or the entire object needs to be serialized!

Finally, let's summarize how to properly handle the serialization problem of spark task closures. First you need to have a clear understanding of the boundaries of the closures involved in the task, to try to control the scope of the closures and the free variables involved, a very alarming point is: Try not to directly reference a class's member variables and functions in the closure, which causes the entire class instance to be serialized. Examples of this are also mentioned in the Spark documentation, as follows:

<code class= "Language-scala hljs has-numbering" style= "display:block; padding:0px; Color:inherit; Box-sizing:border-box; font-family: ' Source Code Pro ', monospace;font-size:undefined; White-space:pre; border-radius:0px; Word-wrap:normal; background:transparent; " ><span class= "Hljs-class" style= "Box-sizing:border-box;" ><span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >class</span> <span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >MyClass</span> {</span> <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); box-sizing:border- box; "
  >def</span> func1 (s:string): String = {...} <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >def</span> Dostuff (rdd:rdd[string]): rdd[string] = {Rdd.map (func1)}}</code><ul class= " Pre-numbering "style=" Box-sizing:border-box; Position:absolute; width:50px; top:0px; Left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li><li style= "Box-sizing:border-box; padding:0px 5px; " >2</li><li style= "Box-sizing:border-box; padding:0px 5px; " >3</li><li style= "Box-sizing:border-box; padding:0px 5px; " >4</li></ul>

Then, a good way to organize code is: Except for those very short functions, try to encapsulate complex operations to a single global function body: A global static method or a Function object

If you do need an instance of a class to participate in the calculation process, you need to do the related serialization work.