A small example of an hbase data read optimization

Today, an experienced God in the group raised a question (@ dust matter) and recorded it.

A June set up a small cluster, 7 Datanode, a master node. The demand is this: through the MR Program to read the data in the HBase table, the program runs after the discovery speed is relatively slow, through the query task detailed discovery, the cumulative need 1542 map completed, there are currently 14 map in execution. Customer is not satisfied with this speed, how can I optimize it?

First, through job, there are 1542 maps that need to be executed, stating that HBase has 1542 partitions (one map per partition), which is a knowledge point.

Data does not have hot spots, hbase processing performance is not a problem

There are 1542 maps that need to be executed, but only 14 in parallel, indicating that the map has insufficient data.

After asking, he did not configure, the default per machine only 2 map, so the degree of parallelism is 7*2=14, that is, running 14 map at the same time

So prompt him to go to configure the file reconfiguration, each machine configures the amount of map data is CPU, because 8 CPUs, so configure 8 map. Then the parallel processing capability is 7*8=56 (56 maps at the same time).

Run 4 times times faster.

Parameter modification:

Conf/mapred-site.xml

Version 1.0 is this parameter mapred.tasktracker.map.tasks.maximum configuration map

Mapred.tasktracker.reduce.tasks.maximum Configuring reduce

Https://hadoop.apache.org/docs/r1.0.4/mapred-default.html

Version 2.0 is this parameter mapreduce.tasktracker.map.tasks.maximum configuration map

Mapreduce.tasktracker.reduce.tasks.maximum Configuring reduce

Http://hadoop.apache.org/docs/r2.4.1/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

Reflection:

1. Hadoop clusters are built and must be optimized (optimization needs to be optimized according to your different business scenarios)

2. A region corresponding to a map, then in turn, bulk import data to hbase, how to optimize it?

A small example of an hbase data read optimization