Impala's Hidden Secrets

As the source of big data, meta-data plays a very important role. But there's a secret hidden in impala? and meta-data has a deep roots, we come back together!

I. Impala Architecture

(1) Each slave node runs a Impala process, and the datanode process of HDFS works together

(2) Two other processes running on the master node to support query execution

1 , State Store process: Provides lookup services for Impaladaemon and periodically checks the status of Impala processes

2 , Catalog process: Synchronizing metadata changes to all impaladaemons

650) this.width=650; "Src=" https://s2.51cto.com/wyfs02/M02/8C/8B/wKiom1hvSFiQB97eAAD7OoUSkmM978.png-wh_500x0-wm_ 3-wmp_4-s_1201672807.png "title=" 11.png "alt=" Wkiom1hvsfiqb97eaad7oouskmm978.png-wh_50 "/>

Ii.How Impala executes queries

(1) Impala daemon Query execution process

The client (Impala-shell or hue) connects to the Impala Daemon, which is coordinator. Coordinator requests other Impala daemons from the state store, distributes the query to other Impala daemons, and finally returns the results to the client.

650) this.width=650; "Src=" https://s3.51cto.com/wyfs02/M00/8C/8B/wKiom1hvSGnAg_GlAAD8mSsMxSY065.png-wh_500x0-wm_ 3-wmp_4-s_2431350356.png "title=" 22.png "alt=" Wkiom1hvsgnag_glaad8mssmxsy065.png-wh_50 "/>

Third, metadata cache

Based on Impala's query function, we can know that there is a cache of metadata in Impala. Metadata is stored in the Metastore of Hive, what is the role of the metadata cache? is when Impala starts, cache the metadata to the node of Impala Daemon. When Impala daemon initiates metadata changes, such as creating a table, deleting a table, or modifying the definition of a table, these can be cached. If we go to query the table information, can be found from the cache, directly in the node query, so in each Impala daemon inside the metadata cache node as a cache.

650) this.width=650; "Src=" https://s4.51cto.com/wyfs02/M00/8C/88/wKioL1hvSHWSZssvAABqzejoX0s238.png-wh_500x0-wm_ 3-wmp_4-s_3073393867.png "title=" 33.png "alt=" Wkiol1hvshwszssvaabqzejox0s238.png-wh_50 "/>

When the Impala process changes the metadata, the catalog service will be prompted to notify all Impala daemons to update the cache.

650) this.width=650; "Src=" https://s1.51cto.com/wyfs02/M01/8C/8B/wKiom1hvSIawcaVTAAFa0gdUm7c623.png-wh_500x0-wm_ 3-wmp_4-s_3648960010.png "title=" 44.png "alt=" Wkiom1hvsiawcavtaafa0gdum7c623.png-wh_50 "/>

Iv. external changes and meta-data caching

We're talking about internal changes, but there are often external changes, but external changes are unknown to Impala, such as:

(1) Hive, Hcatalog, or Huemetadata manager changed meta data

(2) Data added directly to the HDFs directory

650) this.width=650; "Src=" https://s5.51cto.com/wyfs02/M02/8C/88/wKioL1hvSVeCXQbKAAEs8-9q8hI201.png-wh_500x0-wm_ 3-wmp_4-s_1440135879.png "title=" 55.png "alt=" Wkiol1hvsvecxqbkaaes8-9q8hi201.png-wh_50 "/>

External changes will cause the Impala metadata cache to become unavailable, and we must manually refresh or validate the Impala's metadata cache.

Of course, this change varies depending on the content and nature of the change, and there are different ways to update it, such as:

650) this.width=650; "Src=" https://s5.51cto.com/wyfs02/M00/8C/88/wKioL1hvSWuDDo50AAJQHQLzZXs072.png-wh_500x0-wm_ 3-wmp_4-s_2791913476.png "title=" 66.png "alt=" Wkiol1hvswuddo50aajqhqlzzxs072.png-wh_50 "/>

In doubt, welcome to the "Big Data CN", the public number every day peer hi stop, there are more resources waiting for you!

This article is from the "11872756" blog, please be sure to keep this source http://11882756.blog.51cto.com/11872756/1889736

Impala's Hidden Secrets