Java read/write avro example, java read/write avro

Java read/write avro example, java read/write avro

1. avro is a data serialization framework that can be efficiently serialized and deserialized. It supports C, C ++, C #, Java, PHP, Python, and Ruby languages. Now we use Java to read and write data.

II. Environment Construction

1. DownloadAvro-1.7.7.jarAndAvro-tools-1.7.7.jar two jar packages, put in the specified file directory. Http://www.trieuvan.com/apache/avro/avro-1.7.7/java/

I saved it to the D: \ soft \ avro folder, and created a new java folder under the directory to store the generated Java code.

2. Create a directoryUser. avsc file with the following content:　　

{"namespace": "example.avro", "type": "record", "name": "User", "fields": [     {"name": "name", "type": "string"},     {"name": "favorite_number",  "type": ["int", "null"]},     {"name": "favorite_color", "type": ["string", "null"]}

 ]}   

3. Open cmd and enter the directory. Run the command to generate the User class.

java -jar avro-tools-1.7.7.jar compile schema user.avsc java .

　　

The User. Java file is generated in the./example/avro/directory of the java file in this folder.

4. Use eclipse to create a maven project and add avro dependencies to pom. xml.

<dependency>  <groupId>org.apache.avro</groupId>  <artifactId>avro</artifactId>  <version>1.7.7</version></dependency>

3. The content of the generated User. java file is as follows.

Copy the generated User. java class to the project. Note that the User class generated in User. java and the package name of its internal class areUser. avscInNamespace value,

In this example, example. avro is used. Replace all with your own package name.

The simplest method is to replace example. avro in User. java with the package name.

/** * Autogenerated by Avro *  * DO NOT EDIT DIRECTLY */package example.avro;  @SuppressWarnings("all")@org.apache.avro.specific.AvroGeneratedpublic class User extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {  public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse("{\"type\":\"record\",\"name\":\"User\",\"namespace\":\"example.avro\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"favorite_number\",\"type\":[\"int\",\"null\"]},{\"name\":\"favorite_color\",\"type\":[\"string\",\"null\"]}]}");  public static org.apache.avro.Schema getClassSchema() { return SCHEMA$; }  @Deprecated public java.lang.CharSequence name;  @Deprecated public java.lang.Integer favorite_number;  @Deprecated public java.lang.CharSequence favorite_color;  /**   * Default constructor.  Note that this does not initialize fields   * to their default values from the schema.  If that is desired then   * one should use <code>newBuilder()</code>.    */  public User() {}  /**   * All-args constructor.   */  public User(java.lang.CharSequence name, java.lang.Integer favorite_number, java.lang.CharSequence favorite_color) {    this.name = name;    this.favorite_number = favorite_number;    this.favorite_color = favorite_color;  }  public org.apache.avro.Schema getSchema() { return SCHEMA$; }  // Used by DatumWriter.  Applications should not call.   public java.lang.Object get(int field$) {    switch (field$) {    case 0: return name;    case 1: return favorite_number;    case 2: return favorite_color;    default: throw new org.apache.avro.AvroRuntimeException("Bad index");    }  }  // Used by DatumReader.  Applications should not call.   @SuppressWarnings(value="unchecked")  public void put(int field$, java.lang.Object value$) {    switch (field$) {    case 0: name = (java.lang.CharSequence)value$; break;    case 1: favorite_number = (java.lang.Integer)value$; break;    case 2: favorite_color = (java.lang.CharSequence)value$; break;    default: throw new org.apache.avro.AvroRuntimeException("Bad index");    }  }  /**   * Gets the value of the 'name' field.   */  public java.lang.CharSequence getName() {    return name;  }  /**   * Sets the value of the 'name' field.   * @param value the value to set.   */  public void setName(java.lang.CharSequence value) {    this.name = value;  }  /**   * Gets the value of the 'favorite_number' field.   */  public java.lang.Integer getFavoriteNumber() {    return favorite_number;  }  /**   * Sets the value of the 'favorite_number' field.   * @param value the value to set.   */  public void setFavoriteNumber(java.lang.Integer value) {    this.favorite_number = value;  }  /**   * Gets the value of the 'favorite_color' field.   */  public java.lang.CharSequence getFavoriteColor() {    return favorite_color;  }  /**   * Sets the value of the 'favorite_color' field.   * @param value the value to set.   */  public void setFavoriteColor(java.lang.CharSequence value) {    this.favorite_color = value;  }  /** Creates a new User RecordBuilder */  public static example.avro.User.Builder newBuilder() {    return new example.avro.User.Builder();  }    /** Creates a new User RecordBuilder by copying an existing Builder */  public static example.avro.User.Builder newBuilder(example.avro.User.Builder other) {    return new example.avro.User.Builder(other);  }    /** Creates a new User RecordBuilder by copying an existing User instance */  public static example.avro.User.Builder newBuilder(example.avro.User other) {    return new example.avro.User.Builder(other);  }    /**   * RecordBuilder for User instances.   */  public static class Builder extends org.apache.avro.specific.SpecificRecordBuilderBase<User>    implements org.apache.avro.data.RecordBuilder<User> {    private java.lang.CharSequence name;    private java.lang.Integer favorite_number;    private java.lang.CharSequence favorite_color;    /** Creates a new Builder */    private Builder() {      super(example.avro.User.SCHEMA$);    }        /** Creates a Builder by copying an existing Builder */    private Builder(example.avro.User.Builder other) {      super(other);      if (isValidValue(fields()[0], other.name)) {        this.name = data().deepCopy(fields()[0].schema(), other.name);        fieldSetFlags()[0] = true;      }      if (isValidValue(fields()[1], other.favorite_number)) {        this.favorite_number = data().deepCopy(fields()[1].schema(), other.favorite_number);        fieldSetFlags()[1] = true;      }      if (isValidValue(fields()[2], other.favorite_color)) {        this.favorite_color = data().deepCopy(fields()[2].schema(), other.favorite_color);        fieldSetFlags()[2] = true;      }    }        /** Creates a Builder by copying an existing User instance */    private Builder(example.avro.User other) {            super(example.avro.User.SCHEMA$);      if (isValidValue(fields()[0], other.name)) {        this.name = data().deepCopy(fields()[0].schema(), other.name);        fieldSetFlags()[0] = true;      }      if (isValidValue(fields()[1], other.favorite_number)) {        this.favorite_number = data().deepCopy(fields()[1].schema(), other.favorite_number);        fieldSetFlags()[1] = true;      }      if (isValidValue(fields()[2], other.favorite_color)) {        this.favorite_color = data().deepCopy(fields()[2].schema(), other.favorite_color);        fieldSetFlags()[2] = true;      }    }    /** Gets the value of the 'name' field */    public java.lang.CharSequence getName() {      return name;    }        /** Sets the value of the 'name' field */    public example.avro.User.Builder setName(java.lang.CharSequence value) {      validate(fields()[0], value);      this.name = value;      fieldSetFlags()[0] = true;      return this;     }        /** Checks whether the 'name' field has been set */    public boolean hasName() {      return fieldSetFlags()[0];    }        /** Clears the value of the 'name' field */    public example.avro.User.Builder clearName() {      name = null;      fieldSetFlags()[0] = false;      return this;    }    /** Gets the value of the 'favorite_number' field */    public java.lang.Integer getFavoriteNumber() {      return favorite_number;    }        /** Sets the value of the 'favorite_number' field */    public example.avro.User.Builder setFavoriteNumber(java.lang.Integer value) {      validate(fields()[1], value);      this.favorite_number = value;      fieldSetFlags()[1] = true;      return this;     }        /** Checks whether the 'favorite_number' field has been set */    public boolean hasFavoriteNumber() {      return fieldSetFlags()[1];    }        /** Clears the value of the 'favorite_number' field */    public example.avro.User.Builder clearFavoriteNumber() {      favorite_number = null;      fieldSetFlags()[1] = false;      return this;    }    /** Gets the value of the 'favorite_color' field */    public java.lang.CharSequence getFavoriteColor() {      return favorite_color;    }        /** Sets the value of the 'favorite_color' field */    public example.avro.User.Builder setFavoriteColor(java.lang.CharSequence value) {      validate(fields()[2], value);      this.favorite_color = value;      fieldSetFlags()[2] = true;      return this;     }        /** Checks whether the 'favorite_color' field has been set */    public boolean hasFavoriteColor() {      return fieldSetFlags()[2];    }        /** Clears the value of the 'favorite_color' field */    public example.avro.User.Builder clearFavoriteColor() {      favorite_color = null;      fieldSetFlags()[2] = false;      return this;    }    @Override    public User build() {      try {        User record = new User();        record.name = fieldSetFlags()[0] ? this.name : (java.lang.CharSequence) defaultValue(fields()[0]);        record.favorite_number = fieldSetFlags()[1] ? this.favorite_number : (java.lang.Integer) defaultValue(fields()[1]);        record.favorite_color = fieldSetFlags()[2] ? this.favorite_color : (java.lang.CharSequence) defaultValue(fields()[2]);        return record;      } catch (Exception e) {        throw new org.apache.avro.AvroRuntimeException(e);      }    }  }}

4. Implement serialization in Java, that is, write avro files.

Create a Java class

Public static void main (String [] args) throws IOException {
// Declare and initialize the User object
// Method 1
User user1 = new User (); user1.setName ("zhangsan"); user1.setFavoriteNumber (21); user1.setFavoriteColor (null );
// Method 2 Use constructor user2 = new User ("Ben", 7, "red ");
// Method 3: Build // Construct via builder User user3 = User. newBuilder (). setName ("Charlie "). setFavoriteColor ("blue "). setFavoriteNumber (null ). build (); String path = "D: \ tmp \ user. avro "; // avro file storage directory DatumWriter <User> userDatumWriter = new SpecificDatumWriter <User> (User. class); DataFileWriter <User> dataFileWriter = new DataFileWriter <User> (userDatumWriter); dataFileWriter. create (user1.getSchema (), new File (path ));
// Write the generated user object to the avro file dataFileWriter. append (user1); dataFileWriter. append (user2); dataFileWriter. append (user3); dataFileWriter. close ();
}

Run the code to check whether the avro file is generated in the specified file directory.

5. Read avro files in Java to implement avro deserialization.

public static void main(String[] args) throws IOException {       DatumReader<User> reader = new SpecificDatumReader<User>(User.class);       DataFileReader<User> dataFileReader = new DataFileReader<User>(new File("D:\\tmp\\user.avro"), reader);       User user = null;       while (dataFileReader.hasNext()) {            user = dataFileReader.next();            System.out.println(user);       }      }

Running result:

{"name": "zhangsan", "favorite_number": 21, "favorite_color": null}{"name": "Ben", "favorite_number": 7, "favorite_color": "red"}{"name": "Charlie", "favorite_number": null, "favorite_color": "blue"}

Now, the example has been written.