Boolean Operation--java Bitmap search implementation

Objective

Boolean operations are the great Boolean inventions of the algebra operations, only simple logic and or not, at first people did not find useless, and later on the impact of the computer is too large, from infrastructure to search engines everywhere.

Scene

As a yard farmer, I also met the needs involved in my daily work. The scenario is that our backend service has a complex configuration that involves matching multiple dimensions of the user, because the changes are not very frequent,

Querying the database every time is obviously not cost-effective, the amount of data is not too much, less than the level of people, configuration. This is very natural, cache to the server's memory bar, but can not brute force of a match it, too verbose, the efficiency will be very low, and logical judgment will be a bit complex, the configuration is mainly four dimensions: model, channel, country, version, each dimension is an array, and some value is a ["all"], It means all matches;

Introduction

So think of the boolean operation, it is relatively simple, each value corresponds to a long string of digits, how many data, each value of each dimension is the maximum number of bits.

For a total of 10,000 configurations, the "China" corresponds to a bit vector of up to 10,000 bits, for "China" This value, the 100th digit is 1 means that the 100th configuration contains this "China" dimension value.
The query country is "China" is to take "China" from the map of the corresponding bit vector and "all" corresponding to the bit vector do or operations. Java comes with the implementation of a large integer: BigInteger; You can pass a byte array representing bits to the construction method, and the length of the byte array is obviously: (Number of configuration bars/8) +1,biginteger internal will do some processing, mainly to remove the left continuous 0;

Start right now.

Since we are using the MongoDB database, the operation is using the Spring-data to MongoDB encapsulation. There is the API for criteria and query, which is spring's consistent style.

Thus, the implementation of a common bitmap query, and the criteria of the API similar, so that the code changes can be very small (import spring package to their own almost finished), not verbose, on the code:

First the test case (the example is a little low, will look at it):

Import Java.util.arrays;import Java.util.list;import Java.util.concurrent.atomic.atomicinteger;import Org.junit.assert;import Org.junit.test;import Bitmapsearch.bitmapsearcher;import BitmapSearch.Criteria;public Class Bitmapsearchtest {@Testpublic void Testquery () {list<car> cars = arrays.aslist (new car (),//new car ("VW"). Set Toareas ("All"),//new car ("Mouse Car", 3). Settoareas ("a Mountain"),//new car ("Eastern Red", 6). Settoareas ("China")); Bitmapsearcher searcher = new Bitmapsearcher (Cars, New Bitmapsearcher.indexcreator<car, string> () {@ Overridepublic string[] Indexkeys () {return new string[] {"id", "brand", "Legs", "toareas", "desc"}; @Overridepublic object[] Fieldvalue (Car bean, String indexkey) {if ("id". Equals (Indexkey)) {return new object[] {bean.id };} else if ("brand". Equals (Indexkey)) {return new object[] {Bean.getbrand ()};} else if ("Legs". Equals (Indexkey)) {return NE W object[] {bean.getlegs ()};} else if ("Toareas". Equals (Indexkey)) {return Bean.gettoareas ();} else if ("desc". EqualS (indexkey)) {return new object[] {bean.getdesc ()};} return null;}}); Car rs1 = Searcher.findone (Criteria.where ("legs"). Is (6)//.andoperator (new Criteria (). Oroperator (//criteria.where (" ID "). is (4),//criteria.where (" desc ")) (" MadeInChina ")))//, NULL),//assert.asserttrue (Rs1.brand.equals (" Eastern Red ")); list<car> rs2 = Searcher.find (Criteria.where ("Toareas"). In ("China", "all")); Assert.asserttrue (rs2! = null && rs2.size () = = 2);} private Static class Car {final int id;static atomicinteger Id_gen = new Atomicinteger (); String brand = "QQ"; int legs = 4; String[] Toareas; String desc = "";p ublic Car () {super (); id = Id_gen.incrementandget ();} Public Car (String brand) {this (); this.brand = brand;} Public Car (String brand, int. legs) {this (); This.brand = Brand;this.legs = legs;} Public String Getbrand () {return brand;} public void Setbrand (String brand) {This.brand = brand;} public int Getlegs () {return legs;} public void setlegs (int legs) {this.legs = legs;} Public Car settoareas (String ... toareas) {thiS.toareas = Toareas;return This;} Public string[] Gettoareas () {return toareas;} Public String GetDesc () {return desc;} public void Setdesc (String desc) {this.desc = desc;}}}

Where an internal class indexcreator is abstracted, the work of constructing indexes and how to get indexed field values is thrown to the user

</pre><p></p><p></p><pre name= "code" class= "HTML" >/** * */package bitmapSearch; Import Java.math.biginteger;import java.util.arraylist;import java.util.collections;import java.util.Comparator; Import Java.util.hashmap;import java.util.list;import java.util.map;/** * Universal Bitmap Search Tool Class * * @author Houkangxi * */public FINA L class Bitmapsearcher {/** * object list, read only. */@SuppressWarnings ("Rawtypes") Private final List beanslist;/** * index used for search: K:index, value:<fieldval--bits> */ Private Map<object, Map<object, biginteger>> indexmap;/** * Index Builder */@SuppressWarnings ("Rawtypes") private Final indexcreator indexcreator;/** * Index Builder * * * * @param <t,<span style= "font-family:arial, Helvetica, Sans-seri F; " >INDEX</span>> */public Static Interface indexcreator<t, index> {/** * Returns a set of indexes * * @return */index[] Ind Exkeys ();/** * Gets the field value corresponding to the specified index name * * @param bean *-Object in list * @param indexkey *-index name * @return */obj Ect[] FIeldvalue (T Bean, INDEX indexkey);} /** * Construction Method * * @param objlist *-Object list * @param IC *-index builder */public <t, index> bitmapsear Cher (List<t> objlist, indexcreator<t, index> ic) {indexcreator = Ic;if (objlist! = null && objlist.siz E () > 0) {beanslist = Collections.unmodifiablelist (objlist); Createallindex ();} else {beanslist = Collections.emptylist ();}} Map<object, biginteger> Getbitmap (Object key) {return indexmap.get (key);} /** * Query a result * * @param <T> * * @param criteria *-Query criteria * @param Sorter *-The specified sequencer * @retur n */public <T> T findOne (criteria, comparator<t> sorter) {list<t> List = Find (criteria), if (LIS t = = NULL | | List.isEmpty ()) {return null;} if (sorter! = null) Collections.sort (list, sorter);//Take out the first return list.get (0);} /** * Query List * * @param <T> * @param criteria * @return */@SuppressWarnings ("unchecked") public <T> list<t > Find (Criteria){if (beanslist = = NULL | | beanslist.isempty ()) {return null;} BigInteger bInt = Criteria.proc (this, null), if (bInt = = null) {return null;} arraylist<integer> indexes = new arraylist<integer> (); int Idx;while ((idx = Bint.getlowestsetbit ()) >= 0) {indexes.add (IDX);//The value of the current value minus 1 to do & operation, just next time you can get the rightmost 1bInt = Bint.and (Bint.subtract (Biginteger.one));} Ordinal indexes is naturally arranged from small to large, and will not repeat, because each time is to find the rightmost 1if (Indexes.isempty ()) {return null;}                @SuppressWarnings ("Rawtypes") ArrayList rslist = new ArrayList (Indexes.size ()); for (int i:indexes) {Rslist.add (Beanslist.get (i));} return rslist;} private void Createallindex () {map<object, map<object, byte[]>> t_index = new Hashmap<object, Map< Object, byte[]>> (); object[] keynames = Indexcreator.indexkeys (); for (int i = 0; i < keynames.length; i++) {T_inde X.put (Keynames[i], new Hashmap<object, byte[]> ());} int i = 0;final int SUM = Beanslist.size (); for (Object o:beanslist) {createindex (O, T_index, sum, I, keynames); i++;} Indexmap = new Hashmap<object, Map<object, biginteger>> (T_index.size ()); Bytes2biginteger (T_Index, INDEXMAP);} private void CreateIndex (Object o, Map<object, Map<object, byte[]>> t_index, Final int SUM, final int index,ob Ject[] indexes) {if (o = = null) {return;} Final int byteslen = (SUM >> 3) + 1;final int byteindex = bytesLen-1-(index >> 3); final int value = 1 <  < (index% 8); for (int i = 0; i < indexes.length; i++) {Object key = Indexes[i]; @SuppressWarnings ("Unchecked") Object Fieldvalues[] = Indexcreator.fieldvalue (o, key); if (fieldvalues = = null) {continue;} Map<object, byte[]> bintmap = T_index.get (key); for (Object fieldvalue:fieldvalues) {if (fieldvalue! = null) {byte[ ] BInt = Bintmap.get (Fieldvalue), if (bInt = = null) {Bintmap.put (fieldvalue, bInt = new Byte[byteslen]);} Bint[byteindex] |= value;}}} @SuppressWarnings ("unchecked") private void Bytes2biginteger (Map<object, Map<object, byte[]>> T_index, Map<object, Map<object, biginteger>> bigints) {for (Map.entry<object, Map<object, byte[]>> Entry:t_Index.entrySet ()) {Object key = Entry.getkey (); Map<object, byte[]> value = Entry.getvalue (); if (value = = NULL | | value.isempty ()) {continue;} @SuppressWarnings ("Rawtypes") Map ov = value;for (Map.entry<object, byte[]> v:value.entryset ()) {Ov.put (V.getkey (), New BigInteger (V.getvalue ()));} Bigints.put (key, ov);}}}

Here are some of the implementation classes for criteria:

/** * */package bitmapsearch;import java.math.biginteger;import java.util.linkedlist;import java.util.List;/** *  General Query Constraints * * @author Houkangxi * */public class Criteria {protected Object key;protected list<criteria> chain;private Criteria prev = This;public criteria () {chain = new linkedlist<criteria> ();} Public Criteria (Object key) {this (); this.key = key;} criteria (int noinitchain) {}public static criteria where (Object key) {return new criteria (key);} Private Criteria Addtochain (criteria c) {prev = C;chain.add (c); return this;} Public Criteria are (Object val) {prev.addtochain (new Criteriaopis (Prev.key, Val)); return this;} Public Criteria NE (Object val) {prev.addtochain (new Criteriaopnot (Prev.key, Val)); return this; Public Criteria in (Object ... val) {Prev.addtochain (new Criteriaopin (Prev.key, Val)); return this; Public Criteria and (String key) {return Addtochain (new Criteriaopand (key));} Public Criteria or (String key) {return Addtochain (new Criteriaopor (key));} Public Criteria AndopErator (Criteria ... o) {return addtochain (new Criteriaopand (O));} Public criteria Oroperator (criteria ... o) {return addtochain (new Criteriaopor (O));} BigInteger proc (Bitmapsearcher Sea, BigInteger prev) {if (chain = = null) {return null;} BigInteger rs = prev;for (Criteria c:chain) {rs = C.proc (sea, RS);} return RS;} @Overridepublic String toString () {return getclass (). Getsimplename () + "@key =" + Key;}}

/** *  */package bitmapsearch;import java.math.biginteger;import java.util.arrays;/** * @author Houkangxi * */ Abstract class Criteriachain extends Criteria {criteriachain (String key) {super (key);} Criteriachain (criteria[] list) {super (0); this.chain = Arrays.aslist (list);} Protected abstract BigInteger op (BigInteger O1, BigInteger O2); @Overrideprotected final BigInteger proc (bitmapsearcher Sea, BigInteger prev) {if (chain = = NULL | | chain.isempty ()) {return null;} BigInteger h = chain.get (0). Proc (sea, NULL); for (int i = 1; i < chain.size () && h! = null; i++) {h = op (H, Chai N.get (i). Proc (Sea, h)); if (prev = = null) {return h;} else if (h! = null) {return op (prev, h);} return null;}}

/** * */package bitmapsearch;import java.math.biginteger;/** * and (and) query * @author Houkangxi * */class Criteriaopand Exte NDS Criteriachain {Criteriaopand (String key) {super (key);} Criteriaopand (criteria[] list) {super (list);} @Overrideprotected BigInteger op (BigInteger O1, BigInteger O2) {if (O2 = = NULL | | o1 = NULL) {return null;} Return O1.and (O2);}} /** * */package bitmapsearch;import java.math.biginteger;import java.util.map;/** * in Query * @author Houkangxi * */class C Riteriaopin extends Criteria {object[] colls; Criteriaopin (Object key, object[] colls) {super (key); this.colls = Colls;} @Overrideprotected BigInteger proc (Bitmapsearcher Sea, BigInteger prev) {map<object, biginteger> bitmap = Sea.getbitmap (key), if (bitmap = = NULL | | colls = = NULL) {return null;} BigInteger bit = null;for (int i = 0; i < colls.length; i++) {Object val = colls[i]; BigInteger I = Bitmap.get (val), if (I! = null) {bit = bit = = null? I:bit.or (I);}} return bit;}} /** * */package bitmapsearch;import java.Math. Biginteger;import java.util.map;/** * is query * * @author Houkangxi * */class Criteriaopis extends Criteria {private Object Ov Criteriaopis (object K, object ov) {super (k); This.ov = ov;} @Overrideprotected BigInteger proc (Bitmapsearcher Sea, BigInteger prev) {map<object, biginteger> Bimap = Sea.getbitmap (key); if (Bimap = = null) {return null;} Return Bimap.get (ov);}} /** * */package bitmapsearch;import java.math.biginteger;import java.util.map;/** * NOT (non) query * @author Houkangxi * */CLA SS Criteriaopnot extends Criteria {private Object ov; Criteriaopnot (object K, object ov) {super (k); This.ov = ov;} @Overrideprotected BigInteger proc (Bitmapsearcher Sea, BigInteger prev) {map<object, biginteger> Bimap = Sea.getbitmap (key); if (Bimap = = null) {return null;} BigInteger B = bimap.get (ov); if (b = = null) {return null;} return B.not ();}} /** * */package bitmapsearch;import java.math.biginteger;/** * OR (OR) query * @author Houkangxi * */class Criteriaopor Extend S Criteriachain {Criteriaopor (StRing k) {super (k);} Criteriaopor (criteria[] list) {super (list);} @Overrideprotected BigInteger op (BigInteger O1, BigInteger O2) {if (O2 = = null) {return O1;} if (O1 = = null) {return O2;} Return o1.or (O2);}}

Boolean Operation--java Bitmap search implementation