In-depth hadoop Research: (16)-Avro serialization and deserialization

Reprinted please indicate Source Address: http://blog.csdn.net/lastsweetop/article/details/9773233

All source code on GitHub, https://github.com/lastsweetop/styhadoop

In many cases, Avro is used to transform the original system. The framework format has been defined. We can only use Avro to integrate the original data. (If you are creating a new system, you 'd better use Avro's datafile. The next chapter describes datafile)

Prepare to save the schema as the stringpair. AVSC file and put it in the src/test/resources directory.

{    "type":"record",    "name":"StringPair",    "doc":"A pair of strings",    "fields":[        {"name":"left","type":"string"},        {"name":"right","type":"string"}    ]}

The latest Avro package 1.7.4 depends on Org. codehaus. JACKSON: Jackson-core-Asl: 1.8.8 package, but this version is not available in the maven library, so you need to replace it with another version.

    <dependency>                <groupId>org.codehaus.jackson</groupId>                <artifactId>jackson-core-asl</artifactId>                <version>1.9.9</version>            </dependency>

If you use hadoop 1.0.4 (or another version), it depends on Jackson-mapper-Asl, if it is different from the Jackson-core-Asl version, the method cannot be found. Otherwise, you need to introduce the same version.

            <dependency>                <groupId>org.codehaus.jackson</groupId>                <artifactId>jackson-mapper-asl</artifactId>                <version>1.9.9</version>            </dependency>

The generic method is described in the Code section.

Package COM. sweetop. styhadoop; import JUnit. framework. assert; import Org. apache. avro. schema; import Org. apache. avro. generic. genericdata; import Org. apache. avro. generic. genericdatumreader; import Org. apache. avro. generic. genericdatumwriter; import Org. apache. avro. generic. genericrecord; import Org. apache. avro. io. *; import Org. JUnit. test; import Java. io. bytearrayoutputstream; import Java. io. file; import Java. io. ioexception;/*** created with intellij idea. * User: lastsweetop * Date: 13-8-5 * Time: * to change this template use file | Settings | file templates. */public class testgenericmapping {@ test public void test () throws ioexception {// extract the schema from stringpair. add schema to the AVSC file. parser = new schema. parser (); schema = parser. parse (getclass (). getresourceasstream ("/stringpair. AVSC "); // create a record example genericrecord datum = new genericdata according to the schema. record (schema); datum. put ("Left", "L"); datum. put ("right", "R"); bytearrayoutputstream out = new bytearrayoutputstream (); // datumwriter converts genericrecord into a type that edncoder can understand. datumwriter <genericrecord> writer = new genericdatumwriter <genericrecord> (schema); // encoder can write data into a stream, binaryencoder the second parameter is the reused encoder, which is not reused here. It is used to pass the encoder = encoderfactory. get (). binaryencoder (Out, null); writer. write (datum, encoder); encoder. flush (); out. close (); datumreader <genericrecord> reader = new genericdatumreader <genericrecord> (schema); decoder = decoderfactory. get (). binarydecoder (Out. tobytearray (), null); genericrecord result = reader. read (null, decoder); assert. assertequals ("L", result. get ("Left "). tostring (); assert. assertequals ("r", result. get ("right "). tostring ());}}

Result. Get returns the UTF-8 format. The tostring method must be called to be consistent with the string.

Specific: first, use Avro-Maven-plugin to generate code and configure pom.

  <plugin>                    <groupId>org.apache.avro</groupId>                    <artifactId>avro-maven-plugin</artifactId>                    <version>1.7.0</version>                    <executions>                        <execution>                            <id>schemas</id>                            <phase>generate-sources</phase>                            <goals>                                <goal>schema</goal>                            </goals>                            <configuration>                                <includes>                                    <include>StringPair.avsc</include>                                </includes>                                <sourceDirectory>src/test/resources</sourceDirectory>                                <outputDirectory>${project.build.directory}/generated-sources/java</outputDirectory>                            </configuration>                        </execution>                    </executions>                </plugin>

The Avro-Maven-plugin plug-in is bound to the generate-sources stage. You can call MVN generate-sources to generate the source code. Let's take a look at the generated source code.

package com.sweetop.styhadoop;/** * Autogenerated by Avro * <p/> * DO NOT EDIT DIRECTLY */@SuppressWarnings("all")/** A pair of strings */public class StringPair extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {    public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse("{\"type\":\"record\",\"name\":\"StringPair\",\"doc\":\"A pair of strings\",\"fields\":[{\"name\":\"left\",\"type\":\"string\",\"avro.java.string\":\"String\"},{\"name\":\"right\",\"type\":\"string\"}]}");    @Deprecated    public java.lang.CharSequence left;    @Deprecated    public java.lang.CharSequence right;    public org.apache.avro.Schema getSchema() {        return SCHEMA$;    }    // Used by DatumWriter.  Applications should not call.    public java.lang.Object get(int field$) {        switch (field$) {            case 0:                return left;            case 1:                return right;            default:                throw new org.apache.avro.AvroRuntimeException("Bad index");        }    }    // Used by DatumReader.  Applications should not call.    @SuppressWarnings(value = "unchecked")    public void put(int field$, java.lang.Object value$) {        switch (field$) {            case 0:                left = (java.lang.CharSequence) value$;                break;            case 1:                right = (java.lang.CharSequence) value$;                break;            default:                throw new org.apache.avro.AvroRuntimeException("Bad index");        }    }    /**     * Gets the value of the 'left' field.     */    public java.lang.CharSequence getLeft() {        return left;    }    /**     * Sets the value of the 'left' field.     *     * @param value the value to set.     */    public void setLeft(java.lang.CharSequence value) {        this.left = value;    }    /**     * Gets the value of the 'right' field.     */    public java.lang.CharSequence getRight() {        return right;    }    /**     * Sets the value of the 'right' field.     *     * @param value the value to set.     */    public void setRight(java.lang.CharSequence value) {        this.right = value;    }}

To ensure compatibility with previous versions, a set of get and put methods are generated. After 1.6.0, The getter/setter method is added. There is also a class with builder, which has been deleted by me.

In addition, the namespace can be used in the name in Schama, such as com. sweetop. styhadoop. stringpair, so that the source code generated will contain package.

Let's see if the generated class is different from the generic method:

Package COM. sweetop. styhadoop; import JUnit. framework. assert; import Org. apache. avro. schema; import Org. apache. avro. io. *; import Org. apache. avro. specific. specificdatumreader; import Org. apache. avro. specific. specificdatumwriter; import Org. JUnit. test; import Java. io. bytearrayoutputstream; import Java. io. ioexception;/*** created with intellij idea. * User: lastsweetop * Date: 13-8-6 * Time: * to change this template use file | Settings | file templates. */public class testsprecificmapping {@ test public void test () throws ioexception {// The schema is no longer used because the source code of stringpair has been generated, call setter and getter to stringpair datum = new stringpair (); datum. setleft ("L"); datum. setright ("R"); bytearrayoutputstream out = new bytearrayoutputstream (); // schema is no longer required. stringpair is directly used as the model and parameter, datumwriter <stringpair> writer = new specificdatumwriter <stringpair> (stringpair. class); encoder = encoderfactory. get (). binaryencoder (Out, null); writer. write (datum, encoder); encoder. flush (); out. close (); datumreader <stringpair> reader = new specificdatumreader <stringpair> (stringpair. class); decoder = decoderfactory. get (). binarydecoder (Out. tobytearray (), null); stringpair result = reader. read (null, decoder); assert. assertequals ("L", result. getleft (). tostring (); assert. assertequals ("r", result. getright (). tostring ());}}

To sum up the differences,Schema-> stringpair. Class, genericrecord-> stringpair