"Tech" POI label categories

Winter vacation boss gives the task, let me reproduce this experiment http://www.liuhaihua.cn/archives/15565.html. Oneself on casually tried, use is compare Classic (Lao) of algorithm and knowledge, record a bit.

I. Crawling POI data from a Web page

Find a lap, feel this site more reliable: http://www.poi86.com, because here only need poi tag, do not need other similar address AH what information, so this site is enough.

Crawl site using the Scrapy this open source Library, the core code is as follows:

1 ImportRe2 ImportJSON3 4  fromScrapy.spiderImportBasespider5 6 classSpider (basespider):7Name ="poi86"8Allowed_domains = ["poi86.com"]9indexes = Range (1,1000)TenStart_urls = [] One      forIinchindexes: AURL ='http://www.poi86.com/poi/category/43/%s.html'%I - start_urls.append (URL); -  the     defParse (self,response): -filename = Response.url.split ("/") [-1] -Open (filename,'WB'). Write (Response.body)

The above code only does the "download page and save" work, and the next code extracts the POI tag information from each page.

1 ImportOS2D ='/users/sunshineatnoon/documents/poi/train/'3Article_number = [0,344,1143,1943,2572,3137]4names = ['Finace','Hotel','resturant','Transport','amusment']5  forIinchRange (1, Len (article_number)):6Begin = Article_number[i-1]+17All_shops = []8      forKinchRange (begin,article_number[i]+1):9filename = str (k) +'. html';TenFile_object = open (d+filename) OneAll_text =File_object.read () AAll_text_split = All_text.split ('<table class= "table table-bordered table-hover" >') -Names_split = All_text_split[1].split ('title= "" >') -          forJinchRange (1, Len (names_split)): theShop_name_list = Names_split[j].split ('</a></td>') - all_shops.append (shop_name_list[0]) -All_shops =list (set (All_shops)) -Output = open ('/users/sunshineatnoon/documents/poi/preprocess/'+names[i-1],'W') +      forIteminchAll_shops: -Output.write (item+'\ n') +Output.close ()

The final amount of data is roughly the same:

Category	Number of pages	Number of POI tags
Financial	344	9186
Hotel	799	37985
Catering	800	37248
Traffic	629	22012
Leisure and entertainment	565	25737
Total	3137	132168

Second, participle

The POI tag information that is downloaded is short text, where vector space is used to characterize the texts. So first of all to do participle, here use jieba participle, the tool has three types of Word segmentation: Precision mode, full mode, search engine mode; Here you use the exact mode. The core code of the participle is as follows:

1 seg_list = jieba.cut (line,cut_all=False)2"". Join (Seg_list)  3 word_list = Word_list.split ("")

After the completion of the participle, select words with a word frequency greater than or equal to 5 as the initial processing of the obtained dictionary.

Iii. Selecting feature words based on information gain

The information gain value of the words in the POI information is used as a measure of the effect of the words on the classification, and the information gain top10% (about 1000 words) are selected as the characteristics of the POI information. The formula for the information gain of the word is as follows:

The first item in the formula is the original information entropy in the training data set, and for all the words are the same, so here only the sum of the last two items in the above formula is computed, that is, according to the information entropy of the data after the T classification, and then select 1000 words as feature words according to the order from small to large.

The first 20 words of entropy are as follows: Department store, hotel, parking, branch, hotel, petrol station, China, bank, internet bar, postal savings, Agricultural Bank, KTV, Institute, Sub-branches, network, passenger terminal, hotel, property insurance, apartment; You can see that most of the above words are the above five categories (finance, Typical representative words for transportation, recreation, hotel, catering).

The polynomial naive Bayesian model

The above feature words form a set of "characteristics", according to this set of feature words, a POI information can be expressed as a 0-1 vector, the vector as the POI information vector space representation, input to the naïve Bayesian model classification. In subsequent experiments, the naive Bayesian model in Weka is used, so the vector is organized into the Weka input format, as follows:

121006class {1,2,3,4,5} @DATA0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 , 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 , 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 , 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 , 0,0,0,0,0,0,1 ...

The final 1 indicates that the sample belongs to the first class, the Finance class.

This paper uses the Weka classification tool to train the polynomial naive Bayesian classification model, and obtains the model accuracy by 10 cross-validation. The results are as follows:

The model accuracy rate is 88.6%.

Resources:

"1" http://www.liuhaihua.cn/archives/15565.html

"2" http://www.cnblogs.com/wentingtu/archive/2012/03/24/2416235.html

"3" http://scrapy.org

"4" http://www.cs.waikato.ac.nz/~ml/weka/

"Tech" POI label categories