Hbase+springboot+redis Realizing paging

Implementation principle:

1. Fetch one more piece of data per page of hbase data. Such as: pagination is 10 page, the first query hbase, take 10+1 bar data, and then the first and last rowkey data stored in Redis, Redis key is the user's token+url. namely Token.set (token+url:list<string>);

2, the front desk click on the next page, query the current page (Currentpagae) in the Redis list whether there is List.get (currentpage) Rowkey. If present, take the 10+1 bar before the Startrowkey, and save the last one in Redis. Does not exist then query error, prompt re-query, in theory will not appear, unless Redis hangs.

3, if the query data Startrowkey and Stoprowkey in token can be found. You only need to query the range data.

3. When do you clear these redis data? First, set the Redis validity period. Second, this is important to ensure the accuracy of the first three data . Clears the current user's data in Redis when the user taps the Previous button on a page other than the next .

4, that is, there are paging, of course, there is data statistics count, this I use HBase coprocessor coprocessor. Of course, count is also time-consuming every time. When the first query is made, count is saved in the user's redis. The removal of Redis is still the process of step (3).

5, can do so there is a very important premise: the front-end interface of the paging, only provides two buttons: the previous page and the next page. This is the basis for ensuring the viability of this scheme.

The following code:

The code for the method in the controller, where the basic framework uses the Renren Rapid development Kit

@RequestMapping ("/list") Public R list (@RequestParam map<string, object> params,httpservletrequest HttpRequest) {Long time = System.currenttimemillis (); Hbasepage page= Null;try{string token = httprequest.getheader ("token");//If additional filters are required, define and use them yourself. Buildfilter (Filter) method to Hbasequery query = new Hbasequery (params). Buildscancount (). Buildpagerowkey (token). Finish ();p age = Service.query (query). Buildredisrowkey (token);} catch (Exception e) {e.printstacktrace ();} Long time2 = System.currenttimemillis (); System.out.println ("time2-time==list=" + (Time2-time)); return R.ok (). Put ("page", page);

　　

The

class that handles query parameters Hbasequery, because for business reasons, all HBase queries have two required conditions: the start date and the end date, so I encapsulate the two parameters directly in the Hbasequery.

public class Hbasequery {private static final long Serialversionuid = 1l;//Current page number private int page;    Number of pages per page private long limit;    Private map<string, object> params;    Private list<string> Pagestartrowkeys;    Private Date StartDate;    Private Date endDate;    Private Scan Countscan;    Private scan datascan= new scan ();    private int cache = 10;    Private DateFormat SF =new simpledateformat ("Yyyy-mm-dd");    private filterlist fl = new filterlist (FilterList.Operator.MUST_PASS_ALL);    private filterlist COUNTFL = new filterlist (FilterList.Operator.MUST_PASS_ALL);        Public Hbasequery (map<string, object> params) throws exception{this.params = params;        String temp2 = (string) params.get ("StartDate");        StartDate = Sf.parse (TEMP2);        String temp = (string) params.get ("EndDate");        EndDate = Sf.parse (temp);        EndDate = Dateutils.adddays (enddate,1);        This.page = Integer.parseint (params.get ("page"). toString ()); This.Limit = Integer.parseint (Params.get ("Limit"). ToString ());        cache = limit>5000?5000: ((int) limit+1);//plus 1, because each time will take limit+1 data params.remove ("StartDate");        Params.remove ("EndDate");        Params.remove ("page");    Params.remove ("Limit");        } public Hbasequery Buildscancount () throws exception{countscan= new Scan ();        Countscan.setmaxversions ();        Countscan.setcaching (5);        Long Startlong = Long.max_value-startdate.gettime ();        Countscan.setstoprow ((startlong+ "-"). GetBytes ());        Long Endlong = long.max_value-(Enddate.gettime ()-1);        Countscan.setstartrow ((endlong+ "-"). GetBytes ());    return this;        } public hbasequery Buildpagerowkey (String token) throws exception{datascan.setmaxversions ();        Long Startlong = Long.max_value-startdate.gettime ();        Datascan.setstoprow ((startlong+ "-"). GetBytes ());        Long Endlong = long.max_value-(Enddate.gettime ()-1); Datascan.setstartrow ((endlong+ "-"). GetbyTES ());        Redisutils redisutils = (redisutils) springcontextutils.getbean ("Redisutils");        list<string> Pagestartrowkeys = Redisutils.get (Token,list.class); Click the previous or next page if (Params.get ("Pageicon")!=null&&! ( (string) params.get ("Pageicon")). Equals ("")) {//And Startrowkeys in Redis is not an empty String Pageicon = (string) pa            Rams.get ("Pageicon");                if (pagestartrowkeys!=null) {String Startrowkey = Pagestartrowkeys.get (this.page-1); if (Pageicon.equals ("Next") &&pagestartrowkeys.size () ==this.page) {Datascan.setstartrow (StartRowK                    Ey.getbytes ());                    Filter pagefilter=new pagefilter (cache);                Fl.addfilter (Pagefilter); }else if ((Pageicon.equals ("Next") &&pagestartrowkeys.size () >this.page) | |                    Pageicon.equals ("prev")) {String Stoprowkey = Pagestartrowkeys.get (this.page); Datascan.Setstartrow (Startrowkey.getbytes ());                    Datascan.setstoprow (Stoprowkey.getbytes ());                    Filter pagefilter=new Pagefilter (This.getlimit ());                Fl.addfilter (Pagefilter);            }}else{throw new Exception ("Dot is paging, but there is no data in Redis, this program is definitely problematic");            }}else{//Click on the non-paged button, then remove the paging information Redisutils.delete (token) in Redis;            Filter pagefilter=new Pagefilter (This.getlimit () +1);        Fl.addfilter (Pagefilter);        } datascan.setcaching (cache);    return this;        Public Hbasequery Builddatafilter (filter filter) {Fl.addfilter (filter);    return this;        Public Hbasequery Buildcountfilter (filter filter) {Fl.addfilter (filter);    return this;        } public Hbasequery Finish () {countscan.setfilter (COUNTFL);        Datascan.setfilter (FL);    return this; }}

　　

Methods for querying HBase. Note: Before using the coprocessor for HBase, make sure that the table is enabled for this feature.

The HBase table opens the co-processing function method (Shell command):

(1) Disable the specified table. hbase> Disable ' mytable '
(2) Add aggregation hbase> alter ' mytable ', METHOD = ' Table_att ', ' coprocessor ' and ' = ' | org.apache.hadoop.hbase.coprocessor.aggregateimplementation| | '
(3) Restart the specified table hbase> enable ' mytable '

Public hbasepage Query (hbasequery query) {Long time = System.currenttimemillis ();        map<string,string> Communtiykeysmap = new hashmap<> ();        Hbasepage page = new Hbasepage (Query.getlimit (), Query.getpage ());        Final String tableName = This.gettablename ();             if (Query.getcountscan ()!=null) {aggregationclient ac = new Aggregationclient (Hbasetemplate.getconfiguration ()); try{Long Count = Ac.rowcount (tablename.valueof (TableName), New Longcolumninterpreter (), query.ge                Tcountscan ());            Page.settotalcount (count);            }catch (Throwable e) {e.printstacktrace ();        }} Long time2 = System.currenttimemillis ();            List rows = Hbasetemplate.find (TableName, Query.getdatascan (), new rowmapper<object> () {@Override Public Object Maprow (result result, int i) throws Exception {Class clazz = Reflectmap.get (tableName); Table names and entities are made hereThe mapping of the bean.                if (i==0) {communtiykeysmap.put ("Curpagestart", New String (Result.getrow ())); } if (I==query.getlimit ()) {communtiykeysmap.put ("Nextpagestart", New String (Result.getro                W ()));                } hbaseresultbuilder HRB = new hbaseresultbuilder<object> ("SF", result, clazz);            Return Hrb.buildall (). Fetch ();        }        });        if (Rows.size () >0&&page.getpagesize () <rows.size ()) {Rows.remove (Rows.size ()-1);        } page.setlist (rows);        Page.setnextpagerow (Communtiykeysmap.get ("Nextpagestart"));        Page.setcurpagerow (Communtiykeysmap.get ("Curpagestart"));        Long Time3 = System.currenttimemillis ();        System.out.println ("time2-time==getcount=" + (time2-time));        System.out.println ("Time3-time2==getdata=" + (time3-time2));    return page; }

Code for

/pagination class Hbasepage

public class Hbasepage implements Serializable {private static final long Serialversionuid = 1l;//Total number of records protected long Tota lcount;//records per page protected long pagesize;//total number of pages protected int totalpage;//current page protected int currpage;//data protected list <Object> list;private string nextpagerow;//next rowkeyprivate string curpagerow;//the beginning of the current page rowkey/** * Paging * @param PageSize number of records per page * @param currpage current page */public hbasepage (long pageSize, int currpage) {this.list = List;this.totalco UNT = Totalcount;this.pagesize = Pagesize;this.currpage = Currpage;} public void Settotalcount (long totalcount) {this.totalcount = Totalcount;this.totalpage = (int) Math.ceil ((double) totalcount/pagesize);} Public Hbasepage Buildredisrowkey (String token) {redisutils redisutils = (redisutils) Springcontextutils.getbean (" Redisutils "); list<string> Pagestartrowkeys = Redisutils.get (Token,list.class); list<string> Pagerowkeys = Redisutils.get (Token,list.class); if (This.getlist (). Size () >0) {if (pageRowKeys== null| | PageROwkeys.size () <=0) {Pagerowkeys = new arraylist<> ();p Agerowkeys.add (This.getcurpagerow (). substring (0, This.getcurpagerow (). IndexOf ("-") +1);p Agerowkeys.add (This.getnextpagerow (). substring (0,this.getnextpagerow (). IndexOf ("-") +1)); Redisutils.set (Token,pagerowkeys);} Else{if (Pagerowkeys.size () >this.getcurrpage ()) {//donothing}else if (pagerowkeys.size () ==this.getCurrPage ()) { Pagerowkeys.add (This.getnextpagerow (). substring (0,this.getnextpagerow (). IndexOf ("-") +1); Redisutils.set (token, Pagerowkeys);}}}　　 return this;}}

Attention:

1, my rowkey set rule is (Long_max-new Date (). GetTime () + "-" +id), so watch Startrowkey and stoprowkey with special attention.

If there is any better way or code defects, welcome to discuss the message.

Hbase+springboot+redis Realizing paging