Using Python to extract the feature of the article (ii)

This blog is a sequel to the feature extraction of the article using Python, and mainly introduces the construction of the article eigenvector with TF-IDF weights.

In [1]:

#   extended thesaurus with TF-IDF weights #  in the first document, the word base model is used to determine whether a word appears in the document. However, it has nothing to do with the order and frequency of words. Then the frequency of the word is more meaningful to the document. So this article adds the word frequency to the eigenvector

In [2]:

# 1. Calculate the frequency of a word in a document  from Import Countvectorizer

In [5]:

documents=['the dog ate a sandwich, the wizard transfigured a sandwich, and I ate a sandwich'
   
    ]vector=countvectorizer (stop_words=
    '
    中文版
    '
    )
    print
    ( Vector.fit_transform (documents). Todense ())
    print(vector.vocabulary_)
   

[[2 1 3 1 1]] {u ' sandwich ': 2, U ' Wizard ': 4, U ' dog ': 1, U ' transfigured ': 3, U ' ate ': 0}

in []:

#2. Reverse document rate (IDF)#But when you compare documents of different lengths, the problem becomes complicated. Sklearn provides the Tfdftransformer class to solve the problem,#the comparability of different document vectors is mainly realized by the normalization of the frequency eigenvector of CI. This class uses the L2 paradigm to normalization eigenvectors:#In addition, there are logarithmic word frequency adjustment method (logarithmically scaled term frequencies), the word frequency adjustment to a smaller range, or the word frequency amplification method (augmented term frequencies), Applies to eliminating the differences in longer documents. #Normalization, logarithmic adjustment word frequency and word frequency amplification three methods all eliminate the effect of different size of the document. #for the elimination of high frequency but very common words on the document effect, the introduction of the inverse document frequency (inverse documents FREQUENCY,IDF), used to measure the frequency of words in the document set. #The Tfdftransformer class returns the TF-IDF value by default, and its parameter USE_IDF is true by default#Sklearn provides the Tfidfvectorizer class to encapsulate Countvectorizer and Tfdftransformer classes together. The code is as follows:

In [6]:

 from  sklearn.feature_extraction.text   tfidfvectorizerdocuments  =[the dog ate a sandwich and I ate a sandwich  ,  the Wizard transfigured a sandwich   " ]vector  =tfidfvectorizer (stop _words=  english    )   ( Vector.fit_transform (documents). Todense ())  print  (Vector.vocabulary_ )

[[0.75458397  0.37729199  0.53689271  0.          0.        ] [0.          0.          0.44943642  0.6316672   0.6316672]]{u ' sandwich ': 2, U ' Wizard ': 4, U ' dog ': 1, U ' transfigured ': 3, U ' ate ': 0}

In [7]:

# 3. Using a hash table to achieve the eigenvector #  when calculating the word frequency and the IDF value of a document, the dictionary is created first, and then the eigenvector is established. However, if the document set is particularly large, it can cause memory exhaustion.  #  Therefore, you can take advantage of the hash table to solve the above problem. Sklearn provides the Hashingvectorizer to implement the code as follows:

In [9]:

 from Import hashingvectorizerdocuments=['thedog ate a sandwich and I ate a sandwich',' The Wizard transfigured a sandwich']vector=hashingvectorizer (n_features=5)  Print(vector.transform (documents). Todense ())

[[0.33333333  0.66666667  0.         -0.66666667  0.        ] [0.9486833   0.          0.         -0.31622777  0.        ]

in []:

# The hashing technique is non-stationary (stateless), which maps arbitrary blocks of data to a fixed number of positions, and guarantees that the same input must produce the same output, and that different inputs produce different outputs as much as possible. It can be used for parallel, on-line, streaming to create eigenvectors, because it initializes that no corpus input is required. N_features is an optional parameter, and the default value is 2 of the 20-time side.  # with the symbolic hash function, the probability of the hash collision of the word block can be cancelled out, and the information loss is better than the information loss of the information # redundancy. One disadvantage of the hashing technique is that the results of the model are more difficult to see because the hash function cannot show which word block maps to where the feature # vector is.  #  Special thanks to "Mastering machine learning with Scikit-learn" book. 

Using Python to extract the feature of the article (ii)