HBase Filter and Rowkey range retrieval uses and processes the result set __hbase

HBase The result set is processed after the new API operates the HBase filter spark temporary tables are used for subsequent processing.

Note that in HBase, the data type stored in the table must be one by one corresponding, or it will cause the final result to be very confusing because of the decoding problem .

The code is as follows: (if there is a great God to see, please enlighten me, Exchange mailbox 1970497138@qq.com)

 Val show_info_one:array[(string,string,string) = Array (///Find the columns you want to use here ("CF", "user_id", "Long"),
  .........
  ) Registered table name: ywapp_bigdata_user_score_tmp filterrowkeygethtable (sp, "Db_oss_ywgl:ywapp_bigdata_user_score", Show_info _one,getdatetime (0)) sp.sql ("Select * from Ywapp_bigdata_user_score_tmp limit"). Show () sp.sql ("SELECT count (1) from Y" Wapp_bigdata_user_score_tmp "). Show () Val show_info_two:array[(string,string,string)] = Array (//Find the column you want to fill here ()
  CF "," user_id "," Long "), ...) Registered table name: ywapp_bigdata_user_indicator_tmp filterrowkeygethtable (sp, "Db_oss_ywgl:ywapp_bigdata_user_indicator",  Show_info_two,getdatetime (0)) sp.sql ("Select * from Ywapp_bigdata_user_indicator_tmp limit"). Show () Sp.sql ("Select Count (1) from Ywapp_bigdata_user_indicator_tmp "). Show () 

/** * Data filtering based on Rowkey * Test: Use scan Setstartrow to set the starting position of Rowkey, * You can specify Rowkey prefix to match query, * can not specify row
    Key in the middle of the data filter * @param sparksesson * @param tbl_nm table name * @param show_info display column name (cf,col_nm,valuetype) * @param tuple Rowkey The criteria for date filtering, Starttime,endtime interval is [starttime,endtime) * @param filters filter Element (cf,col_nm,filt Ervalue), default is null * Register as temporary table: tabl_nm_tmp/Private def filterrowkeygethtable (Sparksesson:sparksessi on,tbl_nm:string,show_info:array[(string,string,string)],tuple:tuple2[string,string],filters:array[(String, string,string)]=array (("", "," ")): Unit ={val sparkcontext = sparksesson.sparkcontext val sqlcontext = sparksess
    On.sqlcontext val hbaseconf = hbaseconfiguration.create () hbaseconf.set (TABLEINPUTFORMAT.INPUT_TABLE,TBL_NM) Val table = new Htable (hbaseconf,tbl_nm) Val scan = new scan () Scan.setstartrow (Bytes.tobytes (tuple._1)) scan

 . Setstoprow (Bytes.tobytes (tuple._2))   Add custom Filter if (filters.length>0) {for (filter<-filters) {val f = new Singlecolumnvaluefilter (Byte S.tobytes (filter._1), Bytes.tobytes (filter._2), Comparefilter.compareop.equal,new binarycomparator ( Filter._3)) Scan.setfilter (f)}} for (I <-show_info) {scan.addcolumn (bytes.tobytes),
    Bytes.tobytes (i._2))} val scannerres = Table.getscanner (scan) Val Columnvaluescanner = Scannerres.iterator ()
      /** *-------------------------Key-----------------------------* Convert the resulting value of the scan to a rdd[row used to match the registry] * use * * Here for the Columnvaluescanner processing is not perfect, need to follow up the modification, find effective method * * val listrow = new util. Arraylist[row] () var flag = True while (columnvaluescanner.hasnext) {try{//each result is a row of data VA L R:result = Columnvaluescanner.next () val arr = Arraybuffer[any] ()//Normal Here you need to specify a class based on the element type in the Structtype and the type of the element in the table
   , but for some unknown reason, conversion to other types will be an error, so the unified conversion to string//Get Rowkey     Arr+=bytes.tostring (r.getrow)//*************---------------------------Here you need to specify the type of each column for pattern matching processing for (COL&L T;-show_info) {col._3 match {case "Float" =>{val v = r.getvalue (Col._1.getbytes (), Col._2.getbytes ())//This section requires processing of NULL pointer data if (v!=null) arr+=bytes.tofloat (v) Else a, because there is a call to the bytes.length operation inside the type conversion
              RR+=V} case "Int" =>{val v = r.getvalue (Col._1.getbytes (), Col._2.getbytes ()) if (v!=null) Arr+=bytes.toint (v) Else arr+=v} case "Double" =>{val v =
            R.getvalue (Col._1.getbytes (), col._2.getbytes ()) if (v!=null) arr+=bytes.todouble (v) Else arr+=v} Case "Long" =>{val v = r.getvalue (Col._1.getbytes (), col._2.getbytes ()) if (v!=nu ll) Arr+=bytes.tolong (v) Else arr+=v} case "String" =>{val v = r.getvalue (col._1. GetBytes (), Col._2.getbytES ()) if (v!=null) arr+=bytes.tostring (v) Else Arr+=v} case _=>{Val
            v = r.getvalue (Col._1.getbytes (), col._2.getbytes ()) if (v!=null) arr+=bytes.tostring (v) Else arr+=v }} val row = Row.fromseq (arr.toseq) listrow.add (ROW)}catch{case E: Exception=> e.printstacktrace ()}///build Rdd//val Seqrdd = Sc.makerdd (arr) val schema = Structty PE ({val list = new util. Arraylist[structfield] () List.add (Structfield ("Rowkey", Stringtype,true)) for (Col<-show_info) {col._ 3 match{case "Float" =>list.add (Structfield (col._2,floattype,true)) case "Int" =>list.add (struc  Tfield (col._2,integertype,true)) case "Double" =>list.add (Structfield (col._2,doubletype,true)) case "Long" =>list.add (Structfield (col._2,longtype,true)) case "String" =>list.add (Structfield (col._2,sTringtype,true) Case _=>list.add (Structfield (col._2,stringtype,true))} list} )//Create dataframe val df = sqlcontext.createdataframe (Listrow,schema) val tbl = Tbl_nm.split (":") (1) df.cr Eatetempview (tbl+ "_tmp")//println ("----------------------------------" +sqlcontext.sql (S "SELECT * from $tbl" + "_tmp "). Count ()) Scannerres.close ()}