Hadoop Source Code Analysis (iii) serialization of objects

Because of the need for communication between the MapReduce and HDFs of Hadoop, the communication object needs to be serialized. Hadoop does not use Java serialization, but rather introduced its own system. A large number of serializable objects are defined in Org.apache.hadoop.io, and they all implement the writable interface. A typical example of a writable interface is implemented, such as Under Java Code 1. Public class Mywritable implements writable { 2.//Some Data 3. private int counter; 4. Private long timestamp; 5. 6. public void Write (DataOutput out) throws IOException { 7. Out.writeint (counter); 8. Out.writelong (timestamp); 9.} 10. One. public void ReadFields (Datainput in) throws IOException { Counter = In.readint (); timestamp = In.readlong (); 14.} 15. Mywritable public static-read (Datainput in) throws IOException { mywritable w = new mywritable (); W.readfields (in); . Return W; 20.} 21.} The Write and readfields respectively implement the functions of serializing and deserializing objects, and are the two methods of writable interface definition. Gives a huge The relationship of the objects in the Org.apache.hadoop.io. <ignore_js_op> Here, I marked the objectwritable as red because it has a different status than other objects. When we talk about RPC for Hadoop, we'll mention RPC The information exchanged must be the basic type of Java, the implementation class for the String and writable interfaces, and an array of elements of the type above. Objectwritable to The image stores the type information of an object and an object that can be transmitted on the RPC. In this way, we have a universal, can be used for client/server transfer between the The writable object. For example, to use the object in the example above as an RPC request, we need to create a objectwritable based on mywritable, Objectwritable will write the following information in the flow Object class name length, object class name, object's own serialization result So, to the opposite end, objectwritable can create the corresponding object according to the object class name and solve the serial. It should be noted that objectwritable relies on writablefactories, which The plant that corresponds to the writable sub-class is stored. We need to keep the Mywritable factory in Writablefactories (via Writablefactories.setfactory).

Hadoop Source Code Analysis (iii) serialization of objects