2--implementation part of the Crown Font query system based on HBase

1. Software version and Deployment

maven:3.3.9,jdk:1.7, struts2:2.3.24.1,hibernate:4.3.6,spring:4.2.5,mysql:5.1.34,junit:4,myeclipse:2014;

hadoop2.6.4,hbase1.1.2

SOURCE Download: https://github.com/fansy1990/ssh_v3/releases

Deployment reference: http://blog.csdn.net/fansy1990/article/details/51356583

Data download: http://download.csdn.net/detail/fansy1990/9540865 or http://pan.baidu.com/s/1dEVeJz7
Please refer to the previous blog: Based on HBase, the Crown size query system of the theory part;
2. system function & Core Implementation 2.1 System home

Crown size query system, mainly includes two functions: 1. Import raw data from HDFs to HBase via the Mr Process, providing a common interface; 2. Provide the search function of Crown size, provide deposit and withdrawal function; 2.2 Data load function
The data load function is a common hbase table data import function, the user only need to provide the original data path (HDFS), HBase table name (the table needs to exist first), column description (refer to the previous blog This parameter explanation), field delimiter, time format can be;
2.2.1 Background Implementation

public void SubmitJob () {map<string, object> jsonmap = new hashmap<string, object> (); if ( Hadooputils.getmrlock (). Equals (mrlock.notlocked)) {//Without a lock, you can submit code//Lock Hadooputils.setmrlock (mrlock.locked);// Empty Mr Task Cache Hadooputils.initmrcache ();//Submit Task New Thread (New Hdfs2hbaserunnable (Hdfsfile, Tablename,coldescription, Splitter, DateFormat)). Start (); Jsonmap.put ("Flag", "true"), Jsonmap.put ("JobId", Hadooputils.getjobid ());} else {jsonmap.put ("flag", "false"); Jsonmap.put ("msg", "Already running Mr Program, please confirm!") ");} Utils.write2printwriter (json.tojsonstring (Jsonmap)); return;}

Here is a mrlock, plus this lock is to prevent the task is running after the task is submitted, and there are other programs to repeat the task (monitoring will be problematic); At the same time, the use of multi-threading, after the submission of the task, immediately return to the foreground, the front desk receives the returned information, according to judge, whether the window to monitor the

ret = Callbyajax ("Hadoop/hadoop_submitjob.action", {hdfsfile:hdfs,tablename:table,coldescription:coldescription, Splitter:splitter,dateformat:dateformat}) if (ret.flag== "false") {//Submit task failed $.messager.alert (' hint ', ret.msg, ' warn '); return;} $.messager.progress ({title: ' Hint ', msg: ' Import data ... ', interval:0//disable auto Update Progress Value});//had The RET returned by oop_submitjob.action contains JobId, Ret.jobidif (typeof (EventSource)!== "undefined") {Console.info ("JobId:" +  Ret.jobid);  var source=new EventSource ("hadoop/hadoop_getmrprogress.action" + "? jobid=" + ret.jobid);    Source.onmessage=function (event) {console.info (event.data);  TODO Judge Event.data indexOf error, parse: The following value, display, and prompt for task error if (event.data.indexOf ("error") >-1) {source.close ();  $.messager.progress (' close ');  $.messager.alert (' Hint ', ' task run failed! ', ' warn ');  }//TODO judge Event.data as success, then the task is successful, other emptying will be the task progress if (Event.data = = "Success") {Source.close ();  $.messager.progress (' close ');  $.messager.alert (' Hint ', ' task run successfully! ', ' warn '); }  var bar = $.messager.progress (' bar ');      Bar.progressbar (' SetValue ', event.data); };

Note Here the Jobid gets:
1) The job variable is set to the external static class when the task is submitted;
2) After the thread thread submits the task, go to get Jobid
It is important to note that Jobid is initialized only when the task is running, so after the task is committed (Jobid is initialized in thread run); 3) while Loop gets Jobid:

public static String Getjobid () {Long start = System.currenttimemillis (), while (Nojobid ()) {try {thread.sleep ($);} CATC H (interruptedexception e) {e.printstacktrace ();} Log.info ("Getting Job ID ..."); Long end = System.currenttimemillis (); Log.info ("Get Jobid, Time consuming:" + (End-start) * 1.0/1000 + "s"); return Currjob.getjobid () . toString ();}

private static Boolean nojobid () {if (Currjob = = NULL | | currjob.getjobid () = = null) return True;return false;

At the same time, the HTML5 server is used here to send events, for this technology please refer to: http://www.w3school.com.cn/html5/html_5_serversentevents.asp;
After the 2.2.2 instance submits the task, it reflects the progress of the task in real time:
Background log:
Run successfully:

2.3. Query Crown Font Size
Here the query Crown size contains two parts: 1) Randomly generated: refers to randomly generated crown font size (according to the number can generate different number of Crown font size, comma separated);
2) query, according to the Crown font size to query, if the HBase table in the exist field of the record is 1, the explanation exists, otherwise the query to change the Crown font size is taken out of the state, corresponding to the suspected banknote crown size;
3) Detailed query: Query the Crown size of all information, you can query multiple version information:


2.3.1 Background Implementation

/** * Check the given crown size for the presence of suspected banknote crown font size * * @param stumbers * @return * @throws illegalargumentexception * @throws IOException */public Map<string, string> checkstumbersexist (String stumbers) throws IllegalArgumentException, IOException {String[] Stumbersarr = Stringutils.split (stumbers, Utils.comma); Connection Connection = Hadooputils.gethbaseconnection (); Table table = connection.gettable (tablename.valueof (utils.identify_rmb_records)); map<string, string> map = new hashmap<> (); Get get = null;try {list<get> gets = new Arraylist<> (); for (String stumber:stumbersarr) {get = new get (stumb Er.trim (). GetBytes ()); Gets.add (get);} Result[] results = table.get (gets); String exist; StringBuffer existstr = new StringBuffer (); StringBuffer notexiststr = new StringBuffer (); for (int i = 0; i < results.length; i++) {exist = new String (results[i].g Etvalue (utils.family,utils.col_exist)); if ("1". Equals (EXIST)) {existstr.append (Stumbersarr[i]). Append ( Utils.comma);} else if ("0". Equals (exist) {notexiststr.append (Stumbersarr[i]). Append (Utils.comma);} else {log.info ("Crown font:" + stumbersarr[i] + "value exist field value is abnormal!) ");}} if (existstr.length () > 0) {map.put ("exist", existstr.substring (0, Existstr.length ()-1));} else {map.put ("exist", "N OData "); if (notexiststr.length () > 0) {map.put ("Notexist", notexiststr.substring (0, Notexiststr.length ()-1));} else { Map.put ("Notexist", "NoData");}} catch (Exception e) {e.printstacktrace ();} return map;}

Implementation of the table Java API directly using HBase;

Gets a record of the given Rowkey and version number, as well as using HBase's table Java API to implement

/** * Query data based on Rowkey and version number * @param tableName * @param cfs * @param rowkeys * @param versions * @return * @throws IOException */public list

2.4 Deposit:

1) The deposit needs to enter the user ID, bank, the crown size, of course, can also be randomly generated;
2) access is using the Checkandput function of table, regarding this function to store data consistency, reference: http://blog.csdn.net/fansy1990/article/details/51451583

Due to the AAAR5912 's crown font size, its exist status is 1, indicating that the font size of the HBase table in the storage state, can not be stored again, that is, the discovery of the suspected counterfeit banknotes of the Crown size;

2.5 Withdrawals
1) Withdrawals also have a random generation function, similar to the above:
Of course, only the users and banks are randomly generated here; 2) Withdrawals: Withdrawals are based on the withdrawal amount:

The withdrawal process is as follows:
1) According to the number num of the size of the given withdrawal crown, the num*3 record of the Op_www:exist field corresponding to the rowkey is randomly searched for the value of 1;
2) Update with Hbase.checkandput, update the Op_www:exist field value to 0, and return the updated Rowkey, that is, the Crown font size;
3) If the num*3 record is updated, the updated crown font size is less than Num bar, then again randomly find the Crown font size corresponding to the Op_www:exist field value of 1 records, and update, return the updated crown font size, until the number of Crown font returned is num;

2.6 validation per second The query is made using a single thread:
Package Stumer;import Java.io.ioexception;import java.util.arraylist;import java.util.date;import java.util.List; Import Org.apache.hadoop.hbase.tablename;import Org.apache.hadoop.hbase.client.connection;import Org.apache.hadoop.hbase.client.get;import Org.apache.hadoop.hbase.client.table;import Org.apache.hadoop.hbase.util.bytes;public class Readtest {//private static String FAMILY = "info"; public static void main (string[] args) throws IOException {long size =10000;get (Utils.getconn (), Utils.generaterowkey (size));} public static void Get (Connection connection,list<byte[]> Rowkeys) throws IOException {System.out.println (new Date () + ": Start reading Records ..."); Long start =system.currenttimemillis (); Table table = connection.gettable (tablename.valueof (utils.table)); Get get = null, Long Count =0;try{for (byte[] rowkey:rowkeys) {count ++;//get = new Get (Bytes.tobytes ("")); get = new Get (ro Wkey); Table.get (get), if (count%1000==0) {System.out.println ("Count:" +count);}} Long end = System.currenttimemillis (); SysTem.out.println (New Date () + ":" +rowkeys.size () + "record, read time:" + (End-start) *1.0/1000+ "S");} catch (Exception e) {}finally{table.close ();}}}

Using multi-threaded queries:
package Stumer;import Java.util.date;import Java.util.list;import Org.apache.hadoop.hbase.client.get;import Org.apache.hadoop.hbase.client.table;public class Readthread implements Runnable {private list<byte[]> rks; Private table Table;public readthread (table Table,list<byte[]> rks) {this.table = Table;this.rks = rks;} @Overridepublic void Run () {System.out.println (Thread.CurrentThread (). GetName () + "" +new Date () + ": Start reading Records ..."); long Start =system.currenttimemillis (); Get get = null, Long Count =0;try{for (byte[] rowkey:rks) {count ++;//get = new Get (Bytes.tobytes ("")); get = new Get (Rowkey ); Table.get (get), if (count%1000==0) {System.out.println (Thread.CurrentThread (). GetName () + "Count:" +count);}} Long end = System.currenttimemillis (); System.out.println (Thread.CurrentThread (). GetName () + "" +new Date () + ":" +rks.size () + "record, read time:" + (End-start) *1.0/ 1000+ "S");} catch (Exception e) {}}} 

Multi-threaded Query main program:
Package Stumer;import Java.io.ioexception;public class Readthreadtest {public static void main (string[] args) throws Ioex ception {long datasize =500;int threadsize = 20;for (int i=0;i<threadsize;i++) {new Thread (new Readthread ( Utils.gettable (), Utils.generaterowkey (DataSize)). Start ();}}

Engineering Class Utils Program
Package Stumer;import Java.io.ioexception;import Java.text.decimalformat;import java.util.arraylist;import Java.util.date;import Java.util.list;import Java.util.random;import Org.apache.hadoop.conf.configuration;import Org.apache.hadoop.hbase.hbaseconfiguration;import Org.apache.hadoop.hbase.tablename;import Org.apache.hadoop.hbase.client.connection;import Org.apache.hadoop.hbase.client.connectionfactory;import Org.apache.hadoop.hbase.client.table;import Org.apache.hadoop.hbase.util.bytes;public class Utils {public static String TABLE = "records";p rivate static DecimalFormat df = new DecimalFormat ("0000");p ublic static string[] Crownsizepr Efixes =null;static Random random = new random (), static {crownsizeprefixes = new String[26*2];for (int i = 0; I < crown SIZEPREFIXES.LENGTH/2; i++) {Crownsizeprefixes[i] = "AAA" + (char) (+ + i); crownsizeprefixes[i+26] = "AAB" + (char) (+ + i);}} /** * Convert 0~9999 to 0000~9999 * @param num * @return */public static String formatcrownsizesuffix (int nUM) {return df.format (num);} public static Table getTable () throws Ioexception{return Getconn (). GetTable (Tablename.valueof (Table));} public static String Getrandomcrownsize () {return crownsizeprefixes[random.nextint (crownsizeprefixes.length)]+ Formatcrownsizesuffix (Random.nextint (10000));} public static Connection Getconn () throws IOException {Configuration conf = hbaseconfiguration.create (); Conf.set (" Hbase.master "," node2:16000 ");//Specify Hmasterconf.set (" Hbase.rootdir "," hdfs://node1:8020/hbase ");// Specifies that HBase stores path Conf.set ("Hbase.zookeeper.quorum", "Node2,node3,node4") on HDFs,//Specifies the Zookeeper cluster conf.set (" Hbase.zookeeper.property.clientPort "," 2181 ");//Specify the port using zookeeper Cluster Connection connection = Connectionfactory.createconnection (conf);//get even return connection;} public static list<byte[]> Generaterowkey (long size) {System.out.println (new Date () + "start generating" +size + "records ..."); long Start =system.currenttimemillis (); list<byte[]> Rowkeys = new arraylist<> (); for (int i=0;i<size;i++) {Rowkeys.add (Bytes.tobytes (Utils.getrandomcrownsize ()));} Long end =system.currenttimemillis (); System.out.println (New Date () + ":" +rowkeys.size () + "record, Generation Time:" + (End-start) *1.0/1000+ "S"); return Rowkeys;}}

3. Summary 1) based on the Crown size query system based on the existing hbase of the crown size of non-counterfeit banknotes, if there is already a crown size contains counterfeit banknotes, the storage and money-taking function will be problematic; 2) the original data (user information, the trade information of the Crown size) is regular in a certain program, and for big data, or small Need to be tested on a larger data set; 3) user account information (total amount of money stored) is not reflected in the system, follow-up can be considered; 4) query Crown font size, deposit, withdrawal function in the first click on the initialization time is longer, consider the pop-up window display; 5) query Crown size, storage, The detailed query in the withdrawal function can be at the custom point, for example, can only query the data of a certain column or column;
Share, grow, be happy
Down-to-earth, focus
Reprint Please specify blog address: http://blog.csdn.net/fansy1990







2--implementation part of the Crown Font query system based on HBase