The theory part of the Crown Font query system based on HBase

1. Software version and Deployment

maven:3.3.9,jdk:1.7, struts2:2.3.24.1,hibernate:4.3.6,spring:4.2.5,mysql:5.1.34,junit:4,myeclipse:2014;

hadoop2.6.4,hbase1.1.2

SOURCE Download: https://github.com/fansy1990/ssh_v3/releases

Deployment reference: http://blog.csdn.net/fansy1990/article/details/51356583

Data download: http://download.csdn.net/detail/fansy1990/9540865 or http://pan.baidu.com/s/1dEVeJz7
2. Background & ideas currently for banknote recognition, is generally used to see, touch, listen, test four ways, here use a more objective way to identify the class. The construction of the Crown Font Management query system, with the Crown size query as a means to effectively solve the bank's foreign false payment of counterfeit money. Solve the problem of counterfeit banknotes from the source.

The system is to use an objective method to verify counterfeit banknotes. This system adopts the scheme is based on the Crown font size, each of the Chinese yuan is unique, if there is a large table can be all the renminbi and the operation of the renminbi (at what time, where to deposit or access) record, so that in the access can be based on the Crown font size first query, Look at the current crown font size corresponding to the preservation of banknotes in the large table, so that you can determine whether the current crown size corresponding banknotes are counterfeit banknotes (this assumes that all crown sizes in the large table corresponding banknotes are real banknotes).

Here is the storage scenario:

	Deposit/Withdraw	Recent status (there are none in the table)	Real banknotes/counterfeit banknotes
Scenario 1	Save	Yes	Counterfeit banknotes
Scenario 2	Save	No	Real banknotes
Scenario 3	Take	Have (no stateless at this time)	Real banknotes

At present, the traditional database is based on the storage of data generally in the Tens (limited by the performance of queries, etc.), but if you want to store all the money and information stored or acquired records, then traditional databases are certainly not competent. So this system is based on HBase.

3. Functional indicators

? Storage of information for users of all levels;

? storing millions banknote information;

? Support front-end service every second, real-time query request;

? Data storage and computation can be expanded;

? Provide a unified interface to support the front-end related query services;

Note: Among the first two, the million-level user information and millions banknote information is determined according to the data, which can be adjusted according to the data and the size of the cluster (if the cluster is large enough, the storage information can be very large); 4. Architecture
The Crown size query system consists of the following 5 layers:

? Data layer: Including basic data MySQL, documents, web data and so on;

? Data processing layer: The main is the load, including Mr Loading mode, Java API loading model, Sqoop loading mode, etc.

? Data storage layer: mainly HBase storage, including all the information of banknotes and user information;

? Data Service layer: Mainly provide query, storage and other interface services;

? Data Application layer: To access the banknote system, to design the banknote recognition when the banknote is deposited, and other application systems;

5. Table Design 5.1 raw data: The Crown Font storage record (the Crown font size, the table has the Crown font size (0 means no, 1 means there), store or fetch time, store or fetch the bank number, user ID):

User Information table (user ID, name, date of birth, gender, address, mobile number, bound bank number):


5.2 Crown Font size record to the data into a preliminary exploration, found that the law of the Crown is as follows: aaa[a~z][0000~9999]
AAB[A~Z][0000~9999]
If the cluster has four nodes, and the region is initially set to 4, then three split points are: aaam9999,aaaz9999,aabm9999; Assuming that each user takes 10 operations per day, if you want to save 100 days of data, set the number of versions to 1000. The following table statements are created:

Create ' records ',{name=> ' info ', versions=>1000},splits =>[' AAAM9999 ', ' AAAZ9999 ', ' AABM9999 '

The table structure is described as follows:

Primary key/Column cluster	Field name	Field meaning	Example field values	Note
Rowkey	-	Table primary KEY (banknote crown size)	AAAA0000
Timestamp	-	Time stamp	1414939140000	Long (can store user operation time)
Info	-	Column families	-	What the WHO, when, where did
	Exist	Whether there is	1	If the user is a storage behavior, then at the end of the behavior, the value is 1
	Uid	User ID	4113281991xxxx9919
	Bank	Bank of banknote access	Spdbcnsh	Bank number

5.3 User information in the preliminary exploration of data, found that the law of user information as follows: 41132819[89~92]xxxx[0000~9999]
If the cluster has four nodes and the region is initially set to 4, then the three split points are: 4113281990xxxx0000,4113281991xxxx0000,4113281992xxxx0000; The table statement is as follows:

Create ' user ',{name=> ' info '},splits =>[' 4113281990xxxx0000 ', ' 4113281991xxxx0000 ', ' 4113281992xxxx0000 '

The table structure is described as follows:

Primary key/Column cluster	Field name	Field meaning	Example field values	Note
Rowkey	-	User primary key (Social Security number)	4113281991xxxx9919
Timestamp	-	Time stamp	1414939140000	Long type
Info	-	Column families	-	User Information
	Name	User name	JACO
	Gender	User gender	Femail
	Bank	User Registration Bank	Spdbcnsh	Bank number
	Address	User Address	exx-o94-1319151759
	Birthday	User Birth date	1981-10-20 09:12

6. When the data loading system is put into use, historical data is already present, and the historical data needs to be imported into the system in batches. When the renminbi is first issued, it also needs to be imported into the system. The import here is directly using the Mr Import.
The MR is designed as a generic data import from HDFs into the mr:6.1 main class of HBase:

Package Ssh.mr;import Org.apache.hadoop.conf.configuration;import Org.apache.hadoop.conf.configured;import Org.apache.hadoop.fs.path;import Org.apache.hadoop.hbase.tablename;import Org.apache.hadoop.hbase.mapreduce.tablemapreduceutil;import Org.apache.hadoop.mapreduce.job;import Org.apache.hadoop.mapreduce.lib.input.fileinputformat;import Org.apache.hadoop.mapreduce.lib.input.textinputformat;import Org.apache.hadoop.util.tool;import ssh.util.hadooputils;/** * JOB Driver Driver class * * @author Fansy * */public class Importtohbase extends configured Implements Tool {public static final string SPLITTER = "SPLITTER";p ublic static final String colsfamily = "colsfamily";p ublic static Final String DateFormat = "DateFormat"; @Overridepublic int run (string[] args) throws Exception {if (args.length! = 5) {Sys Tem.err.println ("usage:\n demo.job.ImportToHBase <input> <tableName> <splitter> <rk,ts,col1:q1 ,col2:q1,col2:q2> <date_format> "); return-1;} if (args[3] = = NULL | | args[3].length () < 1) {System.err.println ("column family can ' t be null!"); return-1;} Configuration conf = getconf (); Conf.set (SPLITTER, args[2]); Conf.set (colsfamily, args[3]); Conf.set (DateFormat, args[4 ]); TableName TableName = tablename.valueof (args[1]); Path InputDir = new Path (args[0]); String jobName = "Import to" + tablename.getnameasstring (); Job Job = job.getinstance (conf, jobName); Job.setjarbyclass (Importmapper.class); Fileinputformat.setinputpaths (Job, InputDir); Job.setinputformatclass (Textinputformat.class); Job.setMapperClass ( Importmapper.class); Tablemapreduceutil.inittablereducerjob (tablename.getnameasstring (), null, job); job.setnumreducetasks (0); Hadooputils.setcurrjob (Job);//Set external static Jobreturn job.waitforcompletion (True)? 0:1;}}

The main class of the Run method is used in the traditional Mr Import HBase code, just set the additional parameters, here the main class parameter meaning is interpreted as follows: Input:hdfs input data path; Splitter: Enter data field delimiter; TableName: Table name; <rk, Ts,col1:q1,col2:q1>: Column description, RK for Rowkey and Rowkey columns, TS for timestamp and their columns; Sample data describes the original data, the first column is Rowkey, the second columns are timestamp, The third column belongs to the column family col1, the column name is Q1, the 4th column is the column family col2, and the column name is q1;date_format:timestamp date format, if there is no TS in the column description, then there is no timestamp in the original data, then this parameter has no meaning;
6.2 Mapper:

Package Ssh.mr;import Java.io.ioexception;import Java.text.parseexception;import java.text.simpledateformat;import Java.util.arraylist;import Org.apache.hadoop.hbase.client.put;import Org.apache.hadoop.hbase.io.immutablebyteswritable;import Org.apache.hadoop.hbase.util.bytes;import Org.apache.hadoop.io.longwritable;import Org.apache.hadoop.io.text;import org.apache.hadoop.mapreduce.mapper;/** * Mapper class, receiving HDFS data, writing to hbase table * @author Fansy * */public class Importmapper extends Mapper<longwritable, Text, immutabl Ebyteswritable, put>{private static final String COMMA = ",";p rivate static final string colon= ":";p rivate String Split ter = null;//private String colsstr = null;private int rkindex = 0; Rowkey subscript private int tsindex = 1; Timestamp subscript Private Boolean hasts = false; Whether the raw data has timestampprivate simpledateformat SF = null;private arraylist<byte[][]> colsfamily= null;private put put =null;immutablebyteswritable Rowkey = new immutablebyteswritable (); @Overrideprotected voidSetup (mapper<longwritable, Text, immutablebyteswritable, Put>. Context context) throws IOException, interruptedexception {splitter = Context.getconfiguration (). Get ( Importtohbase.splitter, ","); String colsstr = Context.getconfiguration (). get (importtohbase.colsfamily,null); SF = Context.getconfiguration (). Get ( Importtohbase.dateformat,null) ==null? New SimpleDateFormat ("Yyyy-mm-dd hh:mm"): New SimpleDateFormat (Context.getconfiguration (). Get ( Importtohbase.dateformat)); string[] cols = Colsstr.split (COMMA,-1); colsfamily =new arraylist<> (); for (int i=0;i< cols.length;i++) {if ("RK ". Equals (Cols[i])) {rkindex= i;colsfamily.add (null); continue;} if ("Ts". Equals (Cols[i])) {tsindex = I;colsfamily.add (null); hasts = true;//raw data includes tscontinue;} Colsfamily.add (Getcol (Cols[i));}} /** * Get family:qualifier byte array * @param col * @return */private byte[][] Getcol (String col) {byte[][] Fam_qua = new byte [2] []; string[] Fam_quastr = Col.split (COLON,-1); fam_qua[0]= bytes.tobytes (fam_quastr[0]); fam_qua[1]= Bytes.tobytes (fam_quastr[1]); return Fam_qua;} @Overrideprotected void Map (longwritable key, text value,mapper<longwritable, text, immutablebyteswritable, put> . Context context) throws IOException, interruptedexception {string[] words = value.tostring (). Split (splitter,-1); if ( Words.length!=colsfamily.size ()) {System.out.println ("line:" +value.tostring () + "does not compatible!"); return;} Rowkey.set (Getrowkey (Words[rkindex]));p ut = GetValue (words,colsfamily,rowkey.copybytes ()); Context.write (Rowkey, Put);} /** * Get Put value * @param words * @param colsfamily * @param BS * @return */private Put getValue (string[] words, arraylist< Byte[][]> colsfamily, byte[] bs) {Put put = new put (BS); for (int i=0;i<colsfamily.size (); i++) {if (Colsfamily.get (i) ==null) {//RK or tscontinue;//the next column}if (Words[i]==null | | words[i].length () ==0) {//not added, directly down a valuecontinue;} The record of the date exception is also added if (hasts) {//Inserts the data containing the time Put.addcolumn (Colsfamily.get (i) [0], Colsfamily.get (i) [1],getlongfromdate ( Words[tsindex]), Bytes.tobytes (worDs[i]));} else{//contains no time data put.addcolumn (Colsfamily.get (i) [0], Colsfamily.get (i) [1], bytes.tobytes (Words[i]));}} return put;} Private Long Getlongfromdate (String datestr) {Try{return sf.parse (datestr). GetTime ();} catch (ParseException e) {System.out.println (datestr+ "conversion failed!"); return 0;}} /** * Get RowKey byte array * @param rowKey * @return */private byte[] Getrowkey (String rowKey) {return bytes.tobytes (RowKey);} }

Mapper is the core of the entire process and is primarily responsible for data parsing and the work of importing from HDFs into HBase tables with the following functions:

? Setup (): Gets the input data field delimiter, gets the column family, the column name, gets the Rowkey column label, obtains the TS format and the column label (if not, follows the insertion data time setting);

? Map (): Parse, filter and extract data (required field data), generate put object, write to HBase;

6.3 For Records,user Mr Import:
You only need to put together the parameters and then call them directly.
7. Real-time Data loading

Use the Java API to manipulate HBase database and complete the real-time HBase database update, including the features such as Crown size query, access payment and so on.

Share, grow, be happy

Down-to-earth, focus

Reprint Please specify blog address: http://blog.csdn.net/fansy1990

The theory part of the Crown Font query system based on HBase