Hbase+hadoop experience sharing-products and technologies

At present, the system on the line, constantly have new problems, but also constantly summing up some experience, hair here, do mark, but also for hbase users to provide some ideas.

1, a single rowkey can support millions of columns, but it is not recommended to do so, because the real one rowkey hung so much data will be in two places problems, one is the query when the number of Single-line record data is too large, client timeout The second problem is that the process of regionserver the compact may be caused by a single rowkey of data stuffed with memory oom

2, the number of Datanode handler not the default value, too little, to be around 100, the other Regionserver handler the number of 200 or so more appropriate

3, Resultscannser use finished remember to close

4, Htablepool can manage htable. After the end of the query directly with Htablepool.put (Htable T) to reclaim, storage after the end Remember Table.flushcommit () after recycling

5, Mapred.local.dir This parameter is the MapReduce will be on the machine's hard disk to do temporary file storage, if mounted on more than one disk, please match each disk, improve the overall performance

6, the map phase of the output can be configured Lzo compression, which can effectively save the map->reduce phase of network bandwidth, but will increase the CPU occupancy rate

7. Multi-NIC binding can effectively improve the network bandwidth of Hadoop system

8, Bulkload current test performance in a single node 51gb/hour or so, I hope that other students have done to exchange the bulkload aspects of the problem.