(implemented) similarity to big Data find the MySQL article match some ideas and improve the query speed

requirements, the recent realization of the article's originality detection function, processing ideas one is to do search engine matching floating red, on the other hand is a quantitative phrase, according to articles, paragraphs, sentences to do database query, the function basically meet the actual needs.

Next, it is necessary to quickly find the most relevant articles and paragraphs in a large amount of data.

The previous essay recorded some of the ideas of the time, this afternoon by the idea of concrete implementation and testing, speed than the direct packet query is certainly a lot faster, review my implementation steps:

Compressing "corpus, that is, extracting feature words or word frequency, and doing quantitative processing to save to the database in the form of" column vector ", and then press the first n groups of words into a vector group for query use, that is, the combination of 1 to n word combination, quantified in the form of" row vector "saved to the database (currently MySQL), When calculating and querying similarity, we first extract the features, then quantify and query the long numeric fields, and the speed should be faster than the general query.

application example: [PM realized the specific idea, the current system is processing data, is expected to be in 80 million rows of data sets, believe that the query speed should also be]

Query test queries the following characteristics

dictionary<Stringint> words =New dictionary<StringInt>(); Words. ADD ("Wubi",1); Words. ADD ("Pinyin",1); Words. ADD ("Stroke",1); Words. ADD ("Other",1); Words. ADD ("English plate",1); Words. ADD ("American Dish",1); Words. ADD ("French plate",1);//List<dictionary<int, long>> WordList = new List<dictionary<int, long>> ();//for (int i = 0; i < i++)//{//Wordlist.add (getwordsecurity (words, i + 1));//}//Intuitive view of data dictionary<Intlong> R1 = getwordsecurity (words,1); dictionary<Intlong> R2 = getwordsecurity (words,2); dictionary<Intlong> R3 = getwordsecurity (words,3); dictionary<Intlong> R4 = getwordsecurity (words,4); dictionary<Intlong> R5 = getwordsecurity (words,5); dictionary<Intlong> R6 = getwordsecurity (words,6); dictionary<Intlong> R7 = getwordsecurity (words,7); dictionary<Intlong> R8 = getwordsecurity (words,8); dictionary<Intlong> R9 = getwordsecurity (words,9); dictionary<int, long> R10 = getwordsecurity (words, 10); Dictionary<int, long> R11 = getwordsecurity ( Words, 11); Dictionary<int, long> R12 = getwordsecurity ( Words, 12); Dictionary<int, long> R13 = getwordsecurity ( Words, 13); Dictionary<int, long> R14 = getwordsecurity ( Words, 14);            

"Quantitative data" I chose Md5->long to quantify.

Wubi-868324650754601807257200751680446853546444854990336207024 Other-4797408270696495584 English plate- 174184988395034501141160942441067998905071717547464226258        

The query requires only one of the values in the following list to be the query criteria, depending on the actual demand (that is, the correlation requirement). That is, by word-line vector arrangement, feature column vector arrangement will map the article into ID, so that we

You can go through Select. From T Where long1= Value implements the relevance of the article query "query field can be changed at any time according to the relevant requirements LONGN"

Two words dictionary<Intlong> R1 = getwordsecurity (words,1);+  [0] {[1,-2963171339501332718]} system.collections.generic.keyvaluepair<Intlong>+ [1] {[2,-2238391517209811048]} system.collections.generic.keyvaluepair<Intlong>+ [2] {[3,4966089295467037960]} system.collections.generic.keyvaluepair<Intlong>+ [3] {[4,-6281813915328659238]} system.collections.generic.keyvaluepair<Intlong>+ [4] {[5, 922666897348189770]} System.collections.generic.keyvaluepair<int, long>+ [5] {[6, int,long>+ [6] {[7,-int,long>dictionary<int, long> R2 = getwordsecurity (words, 2);    

The above test in the afternoon to complete the coding and testing, and now my system is doing data capture and quantification processing, the initial estimate of data set 80 million rows or so, for several years to do the program, this is the first time we deal with millions of rows of data.

(implemented) similarity to big Data find the MySQL article match some ideas and improve the query speed