Research on the serialization of Hadoop

Hadoop differs from the Java-brought serialization mechanism by providing a set of serialization system interfaces and classes.

For basic data types, the writable interface represents the data that can be serialized, and this interface defines 2 methods, where the Write method can serialize the data to the DataOutput byte array given by the parameter. The Readfield method can read the serialized byte array from the Datinput and deserialize it into Hadoop data:

Public interface Writable { 
  /**  
   * Serialize The fields of this object to <code>out</code>. 
   *  
   * @param out <code>DataOuput</code> to serialize this object into. 
   * @throws IOException 
   *
  /Void Write (DataOutput out) throws IOException; 
     
  /**  
   * Deserialize The fields of this object from <code>in</code>.   
   *  
   * <p>for efficiency, implementations should attempt to re-use storage in the  
   * existing object where POS sible.</p> 
   *  
   * @param in <code>DataInput</code> to deseriablize this object from. 
   * @throws IOException 
   *
  /void ReadFields (Datainput in) throws IOException; 
}

But in Hadoop, the serialization process is generally used for map-reduce, and we don't see the intermediate artifacts of serialization. To capture the serialized trajectory, we wrote a tool method to serialize it into a byte array so that we could print the contents of the byte array and get the serialized product:

* */package com.charles.hadoop.serial; 
Import Java.io.ByteArrayOutputStream; 
Import Java.io.DataOutputStream; 
     
Import java.io.IOException; 
     
Import org.apache.hadoop.io.Writable; /** * * Description: This class provides a tool method to record the serialized trajectory * because, in Hadoop, serialization and deserialization are done in the writable interface, writable is the serialized Hadoop object * So we put the serialized The product is stored in a byte array to capture the content * * @author Charles.wang * @created June 2, 9:32:41 AM * */public class Hadoopserializa 
    Tionutil {//This method serializes the object of Hadoop (writable, which is serializable) into a byte array,//then returns the contents of the byte array to/from the parameter, the serialized numeric object 
         Return value: Serialized byte array public static byte[] Serialize (writable writable) throws IOException {//Create a byte array 
         Bytearrayoutputstream out = new Bytearrayoutputstream (); 
         Creates a dataoutputstream and wraps an array of bytes to store the serialized byte stream DataOutputStream dataout = new DataOutputStream (out); 
         Let the Hadoop object serialize to the byte array corresponding to the stream of bytes writable.write (dataout); 
         Dataout.close (); Returns the serialized byte stream
         return Out.tobytearray (); } 
         
      
     
}