Implement Hbase paging and Hbase Paging

Implement Hbase paging and Hbase Paging

Author: Source of R xingyue: http://www.cnblogs.com/rxingyue. You are welcome to reprint it. Please keep this statement. Thank you!

In a project, hbase paging is performed because the data volume is large and scheduled incremental analysis is required. The version used in the project is hbase1.1. Need to enable coprocessor Aggregation

1. Start global aggregation to manipulate data on all tables. By modifying the hbase-site.xml file, you just need to add the following code:

<property>   <name>hbase.coprocessor.user.region.classes</name>   <value>org.apache.hadoop.hbase.coprocessor.AggregateImplementation</value> </property>

2. enable aggregation for a specific table only. Implemented through HBase Shell.

(1) disable specifies the table. Hbase> disable 'mytable'

(2) Add aggregation hbase> alter 'mytable', METHOD => 'table _ att ', 'coprocessor' => '| org. apache. hadoop. hbase. coprocessor. aggresponimplementation |'

(3) restart the specified table hbase> enable 'mytable'

 

Hbase client call code example

1. Obtain the total number of hbase table structures.

Public int getTotalRecord (Table keyIndexTable, String nowTime ){
Int count = 0;
AggregationClient aggregationClient = new AggregationClient (config );
Scan scan = new Scan ();
Scan. setStopRow (nowTime. getBytes (); // less than the current time
Try {
Long rowCount = aggregationClient. rowCount (keyIndexTable, new LongColumnInterpreter (), scan );
AggregationClient. close ();
Count = rowCount. intValue ();
} Catch (Throwable e ){
E. printStackTrace ();
}
Return count;
}

2. Implement Paging

 

Public Map <String, String> getIndexTableInfo (Table table, String tableName, String nowTime, String startRow, Integer currentPage, Integer pageSize) {Map <String, string> communtiyKeysMap = new TreeMap <String, String> (); result?margin = null; // This is the encapsulated class object created by pagination, the following shows the specific attribute try {// obtain the maximum number of returned results if (pageSize = null | pageSize = 0L) pageSize = 100; if (currentPage = null | currentPage = 0) currentPage = 1; // calculate the start AND end pages Integer nowPageSize = pageSize + 1; // MUST_PASS_ALL (condition AND) MUST_PASS_ONE (condition OR) FilterList filterList = new FilterList (FilterList. operator. MUST_PASS_ALL); Filter filter1 = new PageFilter (nowPageSize); filterList. addFilter (filter1); // if (tableName. equals ("COMMUNITY_KEYS_INDEX") {// Filter filter2 = new QualifierFilter (CompareOp. EQUAL, new BinaryComparator (Bytes. toBytes ("communitykey "))); // FilterList. addFilter (filter2); //} Scan scan = new Scan (); scan. setFilter (filterList); scan. setMaxResultSize (nowPageSize); scan. setStartRow (Bytes. toBytes (startRow); if (! NowTime. equals ("") {scan. setStopRow (nowTime. getBytes ();} bytes = table. getScanner (scan); int I = 1; // traverses the scanner object and retrieves the row key of the data to be queried for (Result result: Success) {String row = new String (result. getRow (); for (Cell cell: result. rawCells () {// System. out. println ("columnfamily:" + new String (CellUtil. cloneQualifier (cell) + ">>>" + new String (CellUtil. cloneValue (cell); if (I = nowPageSize) {communtiyKeysMap. p Ut ("nextStart", row. substring (0, row. lastIndexOf (":"); break;} communtiyKeysMap. put (row, new String (CellUtil. cloneValue (cell);} I ++;} catch (IOException e) {e. printStackTrace ();} finally {if (response! = Null) response. close ();} return communtiyKeysMap ;}

3. Process and jump to the next page in this page

For (int page = 1; page <= pageNum; page ++) {// page List <String> pageList = new ArrayList <String> (); // Analysis of subclass calls // 1. map <String, String> communtiyKeysMap = getIndexTableInfo (hTable, hbaseIndexTabel, nowTime, startRow, page, pageSize); for (String communitykey: communtiyKeysMap. keySet () {String rowKeyIndex = communitykey; String cellValue = communtiyKeysMap. get (rowKeyIndex); if (communitykey. equals ("nextStart") {startRow = cellValue; continue; // jump to the next page }}// call specific analysis // implement paging processing}

This process consists of three steps. 1. Set the table's coprocessor Aggregation to enable the table to achieve statistical functions. 2. Retrieve 1001 pieces of data on a page, with 1000 pieces of data per page and 1,001st rowkey as the startrowkey on the next page as the marker "nextStart ". 3. Search for associated data and perform logical analysis.