Deferred deserialization: A way to improve the performance of MapReduce programs

Deferred deserialization: A way to improve the performance of MapReduce programs

Jia

Based on the MapReduce parallel Computing framework provided by Hadoop, the user can realize parallel computation conveniently. Typically, the user resolves the input data into a string delimited by a particular delimiter in the MapReduce program, and the cost of deserialization and serialization is too high, and the conversion of the data type during the calculation can incur additional overhead. In this paper, a delay deserialization technology is integrated into the MapReduce program, and the performance of the MapReduce program is improved obviously. The core of this technique is to deserialize a specific field in a record into a corresponding Java object only when necessary.

Deferred deserialization: A way to improve the performance of MapReduce programs