關於最近研究的關鍵詞提取keyword extraction做的筆記

標籤：des   style   blog   http   color   os   使用   io   ar   

來源：http://blog.csdn.net/caohao2008/article/details/3144639

之前內容的整理

要求：第一: 首先找出具有proposal性質的paper,歸納出經典的方法有哪些. 第二:我們如果想用的話,哪種更實用或者易於實現? 哪種在研究上更有意義.

 

第一，      較好較全面地介紹keyword extraction的經典特徵的文章《Finding Advertising Keywords on Web Pages》.

基於概念的keywords提取，使用概念、分類來輔助關鍵詞抽取。較經典的文章《Discovering Key Concepts in Verbose Queries》,《A study on automatically extracted keywords in text categorization》

基於查詢日誌的keywords提取，有文章《Using the wisdom of the crowds for keyword generation》,《Keyword Extraction for Contextual Advertisement》

Keywords擴充，keywords產生《Keyword Generation for Search Engine Advertising using Semantic Similarity》, 《Using the wisdom of the crowds for keyword generation》,《n-Keyword based Automatic Query Generation》

第二，      較常用的特徵，之前研究者提到過的特徵：

《Finding Advertising Keywords on web pages》中提到過的特徵

1.語言特徵 詞性標註

2.首字母大寫

3.關鍵詞是否在hypertext裡

4.關鍵詞是否在meta data裡

5.關鍵詞是否在title裡

6.關鍵詞是否在url裡

7.TF,DF

8.關鍵詞所處位置資訊

9.關鍵詞所在句子長度及文檔長度

10.候選短語的長度

11.查詢日誌

 

我想到的特徵

1.周圍資訊含量，附近幾個詞甚至是一個句子的平均資訊含量。

2.語義距離，使用co-occurance.

3.NE。  曾經在IE抽取中使用過。

4.關鍵詞之間的關係，語義距離。divergance是越大愈好還是越小越好。或者沒有影響？

 

2.3.2.1 Lin: linguistic features.

The linguistic information used in feature extraction includes: two types of pos tags – noun (NN & NNS) andpropernoun (NNP & NNPS), and one type of chunk – noun phrase(NP). The variations used in MoS are: whether the phrase contain these pos tags; whether all the words in that phrase share the same pos tags (either proper noun or noun); and whether the whole candidate phrase is a noun phrase. For DeS, they are: whether the word has the pos tag; whether the word is the beginning of a noun phrase; whether the word is in a noun phrase, but not the first word; and whether the word is outside any noun phrase.

2.3.2.2 C: capitalization.

Whether a word is capitalized is an indication of being art of a proper noun, or an important word. This set of features for MoS is defined as: whether all the words in the andidate phrase are capitalized; whether the first word of he candidate phrase is capitalized; and whether the candidate phrase has a capitalized word. For DeS, it is imply

whether the word is capitalized.

2.3.2.3 H: hypertext.

Whether a candidate phrase or word is part of the anchor text for a hypertext link is extracted as the following features. For MoS, they are: whether the whole candidate phrase matches exactly the anchortext of a link; whether all the words of the candidate phrase are in the same anchor text; and whether any word of the candidate phrase belongs to the anchor text of a link. For DeS, they are: whether the word is the beginning of the anchor text; whether the word is in the anchor text of a link, but not the first word; and whether the word is outside any anchor text.

2.3.2.4 Ms: meta section features.

The header of an HTML document may provide additional information embedded in meta tags. Although the

text in this region is usually not seen by readers, whether a candidate appears in this meta section seems important. For MoS, the features are whether the whole candidate phrase is in the meta section. For DeS, they are: whether the word is the first word in a meta tag; and whether the word occurs somewhere in a meta tag, but not as the first word.

2.3.2.5 T: title.

The only human readable text in the HTML header is the TITLE, which is usually put in the window caption by the browser. For MoS, the feature is whether the whole candidate phrase is in the title. For DeS, the features are: whether the word is the beginning of the title; and whether the word is in the title, but not the first word.

2.3.2.6 M: meta features.

In addition to TITLE, several meta tags are potentially related to keywords, and are used to derive features. In the MoS framework, the features are: whether the whole candidate phrase is in the meta-description; whether the whole candidate phrase is in the meta-keywords; and whether the whole candidate phrase is in the meta-title. For DeS, the features are: whether the word is the beginning of the metadescription; whether the word is in the meta-description, but not the first word; whether the word is the beginning of the meta-keywords; whether the word is in the meta-keywords, but not the first word; whether the word is the beginning of the meta-title; and whether the word is in the meta-title, but not the first word.

2.3.2.7 U: URL.

A web document has one additional highly useful property – the name of the document, which is its URL. For MoS, the features are: whether the whole candidate phrase is in part of the URL string; and whether any word of the candidate phrase is in the URL string. In the DeS framework, the feature is whether the word is in the URL string.

2.3.2.8 IR: information retrieval oriented features.

We consider the TF (term frequency) and DF (document frequency) values of the candidate as real-valued features. The document frequency is derived by counting how many documents in our web page collection that contain the given term. In addition to the original TF and DF frequency numbers, log(TF + 1) and log(DF + 1) are also used as features. The features used in the monolithic and the decomposed frameworks are basically the same, where for DeS, the “term” is the candidate word.

2.3.2.9 Loc: relative location of the candidate.

The beginning of a document often contains an introduction or summary with important words and phrases. Therefore, the location of the occurrence of the word or phrase in the document is also extracted as a feature. Since the length of a document or a sentence varies considerably, we take only the ratio of the location instead of the absolute number. For example, if a word appears in the 10th position, while the whole document contains 200 words, the ratio is then 0.05. These features used for the monolithic and decomposed frameworks are the same. When the candidate is a phrase, its first word is used as its location. There are three different relative locations used as features: wordRatio – the relative location of the candidate in the sentence; sentRatio – the location of the sentence where the candidate is in divided by the total number of sentences in the document; wordDocRatio – the relative location of the candidate in the document. In addition to these 3 realvalued features, we also use their logarithms as features.Specifically, we used log(1+wordRatio), log(1+sentRatio), and log(1 + wordDocRatio).

2.3.2.10 Len: sentence and document length.

The length (in words) of the sentence (sentLen) where the candidate occurs, and the length of the whole document

(docLen) (words in the header are not included) are used as features. Similarly, log(1+sentLen) and log(1+docLen) are also included.

2.3.2.11 phLen: length of the candidate phrase.

For the monolithic framework, the length of the candidate phrase (phLen) in words and log(1+phLen) are included as features. These features are not used in the decomposed framework.

2.3.2.12 Q: query log.

The query log of a search engine reflects the distribution of the keywords people are most interested in. We use the

information to create the following features. For these experiments, unless otherwise mentioned, we used a log file

with the most frequent 7.5 million queries. For the monolithic framework, we consider one binary fea-

ture – whether the phrase appears in the query log, and two real-valued features – the frequency with which it appears and the log value, log(1 + frequency). For the decomposed framework, we consider more variations of this information: whether the word appears in the query log file as the first word of a query; whether the word appears in the query log file as an interior word of a query; and whether the word appears in the query log file as the last word of a query. The frequency values of the above features and their log values (log(1 + f), where f is the corresponding frequency value) are also used as real-valued features. Finally, whether the word never appears in any query log entries is also a feature.

 

和師姐討論完之後的內容整理

背景及應用：

基於內容的廣告詞推薦系統，例如google、yahoo、ebay等的線上廣告系統

問答系統

關鍵詞替代、擴充

冗餘查詢的精簡、調整、重新規整等

輔助分類

輔助話題追蹤

 

特徵選取：

1.語言特徵：使用POS(part-of-speech),標出詞性。如名詞、動詞、副詞、形容詞等。

2.title ： 該關鍵詞是否出現在document中的標題裡。

3.position ： 該關鍵詞在document中的位置，是否出現在整篇文章的首句、末句或段落的首句、末句等。《Automatic Keyword Extraction Using Linguistic Features》裡面詳細介紹了這種方法。

4.TF,IDF：最基本的資訊權衡特徵。

5.NE ： 該關鍵詞是否為具名實體，如人名、地名。是否為日期資訊，如年月日，時間等。

6.關鍵詞之間關係：關鍵詞之間的語義距離，是越大越好還是越小越好，還是沒有關係？

7.周圍詞資訊含量：該詞所在的位置附近幾個詞的資訊含量是否高？或者說該詞所在的句子在整篇文章中資訊含量情況如何？

8.該關鍵詞是否在其他關鍵詞中出現過：作為關鍵詞出現的機率

9.document所屬類別：可參考基於分類的關鍵詞提取和基於concept的關鍵詞提取

10.該詞是否出現在一個總結性句子中

 

關於NE的問題

1.       在paper《News-Oriented Automatic Chinese Keyword Indexing》中使用過

2.       NE的資訊含量非常高。

3.       NE的區分度非常高。

 

值得注意和探討的問題：

1.       關鍵詞的定義？是區分度最大還是資訊含量最大。

2.       由分詞帶來的影響。TF的粒度的問題。分詞本身存在的問題，《Chinese keyword extraction based on max-duplicated Strings of the Documents》找出重複的最大字串。

 

《News-Oriented Automatic Chinese Keyword Indexing》描寫中文關鍵詞抽取，非常經典的一篇文章。其提出了在分詞前先統計字元頻率，解決了分詞不準確及分詞粒度帶來的問題。提到了過濾關鍵詞的方法等等。使用POS標記詞串，然後過濾掉資訊含量比較低的詞性對應的詞彙。例如連詞，副詞等等。

關於選擇出來的特徵，如何選取最有效特徵，可以參考論文《Multi-Subset Selection for Keyword Extraction and Other Prototype Search Tasks Using Feature Selection Algorithms》

關於最近研究的關鍵詞提取keyword extraction做的筆記