The relationship and distinction of several technologies in Hadoop: Hive, pig, hbase relationships and differences

Z Excerpt from: http://www.linuxidc.com/Linux/2014-03/98978.htm

Hadoop Eco-Circle

Pig

A lightweight scripting language that operates on Hadoop, originally launched by Yahoo, but is now on the decline. Yahoo itself slowly withdrew from the maintenance of pig after the open source of its contribution to the open source community by all enthusiasts to maintain. But some companies are still using it, but I don't think it's better to use hive than using pig. ：）

Pig is a data flow language used to quickly and easily handle huge data.

Pig consists of two parts: Pig Interface,pig Latin.

Pig can handle HDFs and hbase data very easily, and like hive, Pig can handle what it needs to do very efficiently, and can save a lot of labor and time by directly manipulating pig queries. When you want to do some conversion on your data, and you don't want to write mapreduce jobs, you can use pig.

Hive

Friends who do not want to develop mapreduce in a programming language such as DB, friends who are familiar with SQL can use hive to open offline data processing and analysis work.

Note that hive is now suitable for offline data manipulation, which means that it is not suitable for real-time online queries or operations in real-world production environments because one word is "slow". Instead

Originated in the role of facebook,hive as a data warehouse in Hadoop. Build on top of the Hadoop cluster and manipulate the SQL-like interface for data stored on the Hadoop cluster. You can do select,join with HIVEQL, and so on.

If you have data warehousing requirements and you're good at writing SQL and don't want to write mapreduce jobs, you can use hive instead.

HBase

HBase runs on HDFs as a column-oriented database, and HDFs lacks the read and write operations that HBase does. HBase is modeled on Google bigtable and stored as a key-value pair. The goal of the project is to quickly locate and access the required data within billions of rows of data in the host.

HBase is a database, a NoSQL database that provides the ability to read and write like other databases, Hadoop does not meet real-time needs, and HBase is ready to meet. If you need real-time access to some data, put it into hbase.

You can use Hadoop as a static data warehouse, HBase as the data store, and the data that makes some things change.

Pig VS Hive

Hive is more suitable for data warehouse tasks, and hive is used primarily for static structures and for work that requires frequent analysis. The similarity between hive and SQL makes it an ideal intersection for Hadoop to be combined with other BI tools.

Pig gives developers more flexibility in large data sets and allows for the development of concise scripts to transform data streams for embedding into larger applications.

Pig is relatively lightweight compared to hive, and its main advantage is that it can drastically reduce the amount of code compared to using Hadoop Java APIs directly. Because of this, pig still attracts a lot of software developers.

Both hive and pig can be used in combination with hbase, and hive and pig also provide high-level language support for HBase, making data statistics processing on hbase very simple

Hive VS HBase

Hive is a batch system built on top of Hadoop to reduce the work of mapreduce jobs, and HBase is a project to support the shortcomings of Hadoop for real-time operations.

Imagine you are operating Rmdb database, if it is a full table scan, use Hive+hadoop, if it is indexed access, use Hbase+hadoop.

The hive query is that mapreduce jobs can be more than 5 minutes to hours, hbase is very efficient and certainly more efficient than hive.

The relationship and distinction of several technologies in Hadoop: Hive, pig, hbase relationships and differences