Hadoop self-study notes (I) Overview of common Hadoop-related projects

This self-study note is from the video Hadoop series on Yutube. url:

Https://www.youtube.com/watch? V =-TaAVaAwZTs (one of them)

I will not repeat it later

Self-taught notes are prone to various errors. Please be careful.

Hadoop is still used in the big data era. Hadoop is not a separate tool, but an entire ecosystem that contains a series of tools. Therefore, we should first introduce Hadoop-related tools and concepts, which will be frequently used in the future.

1. Hadoop Core

A) HDFS Hadoop distributed file system, one of the core of Hadoop, splits the TB, PB, and ZB data into 64 MB (default) data blocks of different sizes are stored on different hard disks and backed up (3 copies by default)

B) MapReduce Hadoop's data processing model. To put it simply, the processing can be divided into the Map stage and the Reduce stage. The Map stage finds the data to be processed, and the Reduce stage processes the data to get the desired result.

C) The future version of YARN is equivalent to MapReduce2.0. For example, in the original MapReduce system, there are JobTracker and TaskTracker. The former focuses on Name Node, and the latter focuses on TaskNode. In YARN, JobTracker will be more detailed, into several

2. Hadoop Projects

A) Pig and Hive: responsible for data analysis. Similar to the SQL tool in Hadoop. This is mainly because it is too difficult to write MapReduce programs every time you use them (and not everyone likes java). Therefore, these tools are available to provide SQL-like support. The difference is that Pig provides a scripting language and converts it to MapReduce, while Hive provides a SQL-like language called HiveQL.

B) HBase and Cassandra. HBase are NOSQL databases. Hadoop data storage method. It is seamlessly integrated with Pig and Hive. Based on google's big table, it can support millions of columns and hundreds of millions of rows of data. Cassandra mainly involves tools for instant interaction with Hadoop data.

C) HCatalog, Lucene, hamand Crunch: HCatalog is a log tool that records query records of Pig and Hive. Lucene provides the search function for various types of searches. Hama is often used to analyze scientific research data, and Crunch is used to represent the pipeline of MapReduce. The preceding four tools provide visual interfaces.

D) Avro and Thrift: Provide data serialization/, so that we can share data in different programs after serialization. Avro is the main data serialization tool of Hadoop. Thrift is mainly used for data serialization between different development languages.

E) Drill and Mahout: intelligent data. Drill is used for data mining. Mahout is used for data classification, for example, intelligent recommendation based on your music.

3. Hadoop Incubation:

A) Sqoop: data exchange between relational databases and hadoop

B) Flume: Real-time log processing system

C) Chukwa:

D) Zookeeper: Synchronize servers and different clusters, and provide a unified management interface.

E) Oozie

F) Knok, security control

G) HDT. Use eclipse to develop Hadoop, such as MapReduce.

H) Spark: It is declared that it is times faster than Hadoop. It mainly puts hadoop hard disk operations into memory. Shark is similar to Spark and corresponds to hive.

I) Ambari: a tool for managing the entire project

J) for details about each tool, you can go to incubator.apache.org/projects/