Hadoop serialization and Writable interface (i)

Serialization

Serialization (serialization) is the process of converting a structured object into a byte stream so that it can be transmitted over a network or written to a hard disk for permanent storage;

In a distributed system, the process serializes objects into a byte stream, travels over the network to another process, and another process receives a stream of bytes, which, by deserializing, returns to the structured object to achieve interprocess communication. In Hadoop, communication between mapper,combiner,reducer and other phases requires the use of serialization and deserialization techniques. For example, Mapper produces intermediate results (<key:http://www.aliyun.com/zixun/aggregation/9541.html ">value1, value2...> Needs to be written to the local hard drive, which is a serialization process that converts a structured object into a byte stream. and write to the hard disk), while the reducer phase reads the intermediate result of the mapper is a deserialization process (reads the byte stream file stored on the hard drive and turns it into a structured object), and it should be noted that Can be transmitted on the network can only be a byte stream, the intermediate results of Mapper shuffle between different hosts, the object will undergo serialization and deserialization of two of processes.

Serialization is part of the Hadoop core, and in Hadoop, the writable interface in the Org.apache.hadoop.io package is the implementation of the Hadoop serialization format.

Writable interface

The Hadoop writable interface is a serialization protocol based on Datainput and DataOutput implementations, compact (efficient use of storage space), and fast (with little overhead for reading and writing data, serialization, and deserialization). The keys (key) and value (value) in Hadoop must be objects that implement the writable interface (the key must also implement writablecomparable for sorting).

The following is a declaration of the writable interface in Hadoop (using Hadoop 1.1.2):

Writable class

Hadoop itself provides a variety of specific writable classes, including common Java Basic Types (Boolean, Byte, short, int, float, long, and double) and collection types (byteswritable, Arraywritable and mapwritable etc.). These types are located in the Org.apache.hadoop.io package.

Custom writable Class

Although Hadoop has built a variety of writable classes to provide user choice, Hadoop implements the Rawcomparable interface for Java base-type wrapper writable classes, allowing these objects to be sorted at the byte stream level without the need to deserialize the process. This greatly shortens the time overhead of comparisons, but when we need more complex objects, the built-in writable class of Hadoop does not meet our needs (note that the writable collection type provided by Hadoop does not implement the Rawcomparable interface. So it doesn't meet our needs), so we need to customize our writable class, especially when it's key (key), in order to achieve more efficient storage and faster comparisons.

The following example shows how to customize a writable class, a custom writable class must first implement a writable or Writablecomparable interface, and then write write for a custom writable class (DataOutput Out) and ReadFields (Datainput in) method to control how the custom writable class translates into a byte stream (the Write method) and how to return it from bytes to the writable object.

To be continued, the next article describes the length of the bytes used by the writable object serialized as a byte stream and the composition of its byte sequence.