Java read/write avro example, java read/write avro
1. avro is a data serialization framework that can be efficiently serialized and deserialized. It supports C, C ++, C #, Java, PHP, Python, and Ruby languages. Now we use Java to read and write data.
II. Environment Construction
1. DownloadAvro-1.7.7.jarAndAvro-tools-1
reversal becomes thrift more efficient. And when the BLOB scale in the data structure becomes larger, the difference between Avro and thrift is decreasing. Conclusion Thrift is suitable for program-to-Program static data exchange, which requires schema prediction and relative fixation. Avro adds support for schema dynamics on a thrift basis and does not lose performance on thrift. The
in the avpr protocol.After AvroHandler is assembled and processed by the business logic, it returns results. I created some data in the updateRespond/searchRespond2 methods to simulate the data retrieval results.3. tools, protocol parsing tool (AvroUtils. java ).4. user. avpr transfer protocol file.After the code is imported into the IDE environment, run AvroServer. java to start the server, and then run ClientHandler. java to execute the client call
Rpc for getting started with avro,
For more information about avro rpc, visit https://github.com/phunt/avro-rpc-quickstart.
The usage in java is summarized as follows:
Reference: http://www.iteblog.com/archives/1008
Http://my.oschina.net/zhzhenqin/blog/151040
Http://www.bianceng.cn/Servers/web/201411/46469.htm
I am not using maven here, and the jar packages I add
schema definition for a given version number.
Compared with thrift and Pb, Avro is more complex and difficult to use. Of course, it has the following advantages...
At first glance it may seem that Avro's approach suffers from greater complexity, because you need to go to the additional effort of distributing schemas. However, I am beginning to think that Avro's approach also has some distinct advantages:
Object container files are wonderful
with code generation Compiling schema
Here, because the Avro plugin is used, the MAVEN plugin automatically generates the class file for us by entering the following command directly:
MVN clean Install
The corresponding classes are then generated in the directory that you just configured, as follows:
If you do not use Plug-ins, you can also use Avro-tools to bu
document, so thrift uses more people...
Data Type comparison
Obviously, thrift supports more types, especially the direct support for container, which is very powerful.
IDL comparison
Actually, they are similar and different.
The field ID is in different forms. Thrift starts with 1:, while Pb ends with 1.
For container support, thrift uses list directly, while Pb can only be represented by repeated.
Performancesize comparison
It can be seen that thrift uses tcompactprotocol and Pb equival
About Avro RPC get started using, the official information is too little, this link https://github.com/phunt/avro-rpc-quickstart have specific instructionsThe following summary is now available for use in Java:Reference: http://www.iteblog.com/archives/1008http://my.oschina.net/zhzhenqin/blog/151040Http://www.bianceng.cn/Servers/web/201411/46469.htmI do not use Maven here, directly in the project to add the
Note: This is a good Chinese description of the previous blog.
Avro is a subproject of hadoop. It is developed by Doug cutting, founder of hadoop (also the founder of Lucene, nutch, and other projects. Avro is a data serialization system designed for applications that support mass data exchange. It supports binary serialization to process large amounts of data conveniently and quickly. Dynamic languages are
uploading Avro files to HDFs using flume
Scenario Description: Upload the Avro file under a folder to HDFs. Source uses HDFs, which is used by Spooldir,sink. Configure flume.conf
# memory channel called CH1 on Agent1 agent1.channels.ch1.type = memory # source Agent1.sources.spooldir-source1.channels = Ch1 Agent1.sources.spooldir-source1.type = Spooldir Agent1.sources.spooldir-source1.spooldir=/home/yang/da
Avro Introduction
Schema
File composition
Header and DataBlock declaration code
Test code
Serialization and deserialization
Specific
Generic
Resources
Avro IntroductionAvro is a data serialization system created by Doug Cutting (the father of Hadoop) designed to address the lack of writeable types: language portability. To suppo
Reprinted please indicate Source Address: http://blog.csdn.net/lastsweetop/article/details/9773233
All source code on GitHub, https://github.com/lastsweetop/styhadoop
In many cases, Avro is used to transform the original system. The framework format has been defined. We can only use Avro to integrate the original data. (If you are creating a new system, you 'd better use Avro's datafile. The next chapter d
Avro is one of the Hadoop projects. It is mainly used to support data-intensive applications. It defines a data format and supports this format in multiple programming languages. We primarily use Cassandra and RPC to implement mutual calls between languages. There are many lightweight python scripts in the architecture. For example, if PHP receives a URL, it needs to call
Avro is one of the Hadoop projects.
In recent projects to serialize data sent to Kafka using Avro, there are two ways to serialize with Avro: one is to serialize in memory and the other is a data file format.
Change how to choose it.
If you want to integrate Avro into an existing system, it is better to serialize memory.
In other cases, consider using the Avro
Added native thrift and avro support for dubbo/dubbox,
(Facebook) thrift/(hadoop) avro/(google) probuf (grpc) is a highly eye-catching and efficient serialization/rpc framework in recent years. Although the dubbo framework supports thrift, however, earlier versions of the dependency only support 0.8.0, and some extensions are also made to the Protocol, not the native thrift protocol.
Although some friends o
Reprinted please indicate Source Address: http://blog.csdn.net/lastsweetop/article/details/9448961
All source code on GitHub, https://github.com/lastsweetop/styhadoop
Avro is a multi-language data serialization framework that supports C, C ++, C #, Python, Java, PHP, Ruby, and Java.
He was born mainly to make up for the shortcomings that writable only supports the Java language.
Many people will ask thrift and protocol for similar framewo
I. OverviewThrift : Is by Facebook Leading development of a cross-platform, multi-language support, by defining IDL file, a tool that automatically generates RPC client and server-side communication code to build in C + +, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C #, Cocoa, JavaScript, node. js, Smalltalk, and OCaml are seamlessly integrated and efficient services among these programming languages. Thrift defines the interface and data type of RPC through an intermediate language (IDL,
Apache Avro is a data serialization system that is a high performance middleware based on binary data transmission.1. Provide the following characteristics
A rich data structure
A simple, compact, fast binary data format
A file container for persistent data storage
Remote Procedure Call (RPC)
Simple dynamic language combination, Avro and dynamic language, both read and write data fi
1, spark SQL can directly load the Avro file, followed by a series of operations, examples: 1sparkconf sparkconf =NewSparkconf (). Setappname ("Spark Job");2Javasparkcontext Javasparkcontext =NewJavasparkcontext (sparkconf);3 4SqlContext SqlContext =NewSqlContext (javasparkcontext);5 6String Format_class = "Com.databricks.spark.avro";7 8 //Avro the path on the HDFs9String Path = "/sqoopdb/pcdas/*.
https://avro.apache.org/docs/current/IntroductionApache Avro? is a data serialization system.Avro provides:
Rich data structures.
A Compact, fast, binary data format.
A container file, to store persistent data.
Remote procedure Call (RPC).
Simple integration with dynamic languages. Code generation is not required to read or write data files and to the use or implement RPC protocols. Code generation as an optional optimization,
The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion;
products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the
content of the page makes you feel confusing, please write us an email, we will handle the problem
within 5 days after receiving your email.
If you find any instances of plagiarism from the community, please send an email to:
info-contact@alibabacloud.com
and provide relevant evidence. A staff member will contact you within 5 working days.