Computer Search Course Design-Automatic Indexing

I have always thought that information science/library learning is meaningless, but I have never understood it in depth before. Information Retrieval Technology is the cornerstone of the current search engine technology. The famous Elsevier sciencedirect, springerlink, EBSCO academic/business source Premier are the powerful search tools for journal papers in the network environment; there are also popular e-commerce recommendations such as Amazon. Books and movie recommendations in Douban.com are all at the core of data mining, such as collaborative filtering algorithms. There are also knowledge discovery, spam filtering and so on.

It took four days to complete the course assignment (source code). The requirement was simple:

1. Merge and use the deprecated Word Table and keyword table as Word Segmentation words, and use the reverse longest match method to separate all words and give the word segmentation result corresponding to each article name. Displays the sequence number, name, and word splitting result on the screen.
2. Remove deprecated words (displayed on the screen ).
3. Using TF-IDF Word Frequency Weighting formula, calculate the weight of each word, and display the weight of each word in each article on the screen.
4. Determine the index words based on the input threshold and display the index words on the screen.
5. Determine the indexing words based on the input indexing depth and display the indexing words on the screen.

The implementation can also be simple. After all, the data volume is not very large, so the data is stored in the map Container Using ADO. The keyword table uses the key and the value is the number of entries containing the keyword. Then we use the inverse largest matching method combined with disabled Word Table for word segmentation, and then use the TF-IDF Word Frequency Weighting formula for weighting. Finally, by setting the index threshold, the indexing depth shows the selected index words. As follows:

The entire program class diagram is as follows (Er, a bit distorted ...) :

After finishing this assignment, I read the topic article "programmer" "view algorithms through products" and realized that algorithms are the core of the program, the applications that make our lives simple are backed by excellent algorithms. However, unfortunately, many colleges and universities in China only offer data structure courses, but they do not offer more important algorithm courses. It is not enough to understand the data structure to complete an application. In Chinese word segmentation, the dictionary method is simple and effective for maximum forward/reverse matching. The weighted words are mainly inverse document (IDF) Weighting and Shannon information theory. Without these previous algorithms, this program is difficult to complete in a short time. Back to the beginning, those applications, especially algorithms related to Information Science/library science, are the true significance of this profession.

Algorithm? Algorithm!