Hive metastore Thrift Server partition Stress test

Test environment:

Virtual Machine 10.1.77.84

Quad-Core Intel (R) Xeon (r) CPU E5649 @ 2.53ghz,8g Memory

Metastore Server from Single station

JVM args:-XMX7000M-XX:PARALLELGCTHREADS=8-XX:+USECONCMARKSWEEPGC

Test table Schema:

Table name:metastore_stress_testing

# col_name Data_type Comment

A string None

# Partition Information

# col_name Data_type Comment

PT String None

# Detailed Table Information

Database:default

Owner:hadoop

Createtime:wed

May 11:29:30 CST 2013

Lastaccesstime:unknown

Protect Mode:none

retention:0

Location:hdfs://10.1.77.86/user/hive9/warehouse/metastore_stress_testing

Table type:managed_table

Table Parameters:

transient_lastddltime1369798170

# Storage Information

Serde Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

InputFormat:org.apache.hadoop.mapred.TextInputFormat

OutputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat

Compressed:no

Num Buckets:-1

Bucket Columns: []

Sort Columns: []

Storage Desc Params:

Serialization.format1

Test method:

Hive

When parsing a query, if you don't add table partition predicate (such as a direct select

* from Dpdw_traffic_base), which will be invoked in the physical execution plan, partitionpruner in the Physics plan optimization link hive

Metastore

Client's Getpartition (table) method to obtain the list<partition> of the entire table;, as each Partition

It can be seen as a child table under a large table containing separate inputformatclass, Outputformatclass,

Deserializer, Column Schema,

Storagedescriptor,parameters and so on, if the table structure is more complex, column more, partitions and more, will cause the return of the

The object is very large and will be Metastore

Server creates a lot of pressure.

We test performance performance by simulating the Hivemetastoreclient listpartitions method, hiveconf the default setting Metastore

Client socket timeout timeout of 20 seconds (hive.metastore.client.socket.timeout). To prevent read timeout during the test, we set this value very large

Generate call stack for physical execution plan: