Word Frequency Statistics

1)

After a day of thinking, I wrote this design report.
Before I started writing a project, I conducted a lot of queries and tests, mainly related to the usage of the C ++ language. Although I have learned it before, I have no experience in computer engineering, so it will be unfamiliar.
The second is the design of the algorithm. It is not difficult to count the frequency of words. You can use map, but it is difficult to count phrases. Although you can use the method of calculating word frequency, however, the storage space and time cost will be huge, because only the first 10 phrases that are most frequently used are required. Therefore, some optimizations are required. My optimization method is to eliminate some phrases in advance. Specifically, an ordered set is created in descending order of the number of occurrences. Each time a new phrase is added, 2 (or 3) in the phrase is checked first) the minimum value of Word Frequency minfreq is compared with the number of occurrences of 10th phrases in the Set tenthfreq. If minfreq <tenthfreq, it indicates the new phrase (the number of occurrences of this phrase targetfreq <minfreq) it must not be the 10 phrases that appear most frequently.
This project is divided into the following sub-processes. The time in parentheses is assumed to be the time required to master C/C ++.
1. traverse the given directory and return the String Array (10 m) 18 with the absolute path of the required text file
_ Findfirst
_ Findnext
2. Get the word and create a Word Frequency Statistics map (20 m) 17
Ifstream + Getline read each row of each text file in sequence
Map counts word frequency, and vector <pair> sorts word frequency.
3. Obtain the phrase and establish an ordered set of the occurrence times of the phrase (60 m) 120
Pair + List
4. Main Function (30 m)
Each sub-process is assembled into a complete program
5. Testing and debugging (120 m)
Construct 10 Test Cases
6. Performance analysis (60 m)

2)

Actual use of each function module
1. traverse directory: 18 m
2. Obtain the word and create the Word Frequency Statistics map: 17 m
3. Obtain the phrase and establish an ordered set of 120 m corresponding to the occurrence times of the phrase (not implemented)

Each sub-process is assembled into a complete program 60 m
Due to improper design of statistical phrases, A lot of time is wasted and the same method as statistical words is used.

Testing and debugging: 1 day
The algorithm was constantly adjusted during the test, and some classes in C ++ were incorrectly used, so it took a long time.
Construct 10 Test Cases

Performance analysis (2 hours)

3)

Count Microsoft Visual Studio 11.0 \ common7 under the vs11 installation directory.

According to the analysis results of the first version, the results are weak.

Without the performance analysis function, I have no idea that the bottleneck is actually on a regular expression.

So I wrote a class for reading words, and the results were faster than an order of magnitude.

I fixed the bottleneck.

However, add can be optimized again, but I use the map in C ++. Map uses the structure of the red and black trees, rather than a faster hash, so there is no room for improvement.

4)

Time is tight. I used all other test cases.

I will try again later

5)

Great gains

First, I got the exercises to write projects in C ++ language. Although I learned C ++ for a semester, I didn't know how to program them. I forgot a lot of theoretical things, the best way to learn a language is to write it by yourself.

Second, I am more familiar with vs. I have been using vs since my freshman year, but I still know how to use performance analysis, I believe there are still many powerful functions waiting for me to explore.

Word Frequency Statistics