Individual project-word_frequency

0x00 preparation and time planning

1. Because Visual Studio 2013 is used and C # is ready for learning, it is expected that it can be basically used in one day.

3. Understand the requirements and design the basic data structure and general process for 20 minutes

2. implement simple mode 30 min as proposed

3. Extended simple mode to complete extend Mode 1 h

 

0x01 actual use time and implementation process

1. after learning about C #, I checked that the basic mode is similar to C ++ and Java. In addition, I can familiarize myself with the language more quickly during code writing, in fact, it takes only 20 minutes to prepare for the attack.

2. Understand requirements, understand requirements, and design basic data structures and general processes.

1) Demand Analysis is fast. The core function is string processing and Word Frequency Statistics, note the case-sensitive processing, sorting, word length, and word mode ("^ [A-Za-Z] [0-9] *").

2) The first problem is the recursive scanning of files. You can use the following code to obtain the names of all files that meet the requirements.

Directory.GetFiles(path, "*.*", SearchOption.AllDirectories).Where(s => s.EndsWith(".txt") || s.EndsWith(".cpp") || s.EndsWith(".h") || s.EndsWith(".cs"));

3) The next step is string processing. The word frequency statistics function is easy to implement. You can use the container dictionary to store key-value pairs. Because the case is to be processed, two dictionaries are used here.

static Dictionary<string, int> wordtable = new Dictionary<string, int>();static Dictionary<string, string> word = new Dictionary<string, string>();

4) The wordtable key is the lowercase form of the word, and the value is the frequency. The word key is the lowercase form of the word, and the value is the highest priority word form, (For example, word ["file"] = "file"; wordtable ["file"] = 1;), consider that the sorting is based on the value in the key to complete the simple mode;

5) for extend mode, the string processing method is to first obtain the format from the file, such as "word1 wrod2... A long string in the form of wordn. Then, it constantly matches the string in the form of "word1 word2" (or "word1 word2 word3") that meets the requirements and treats it as the key in the wordtable, other methods are the same as those in simple mode.

It took about seven hours to complete the entire process, mainly because many methods in C # language were unfamiliar and many technical problems were encountered during programming. Among them, in order to efficiently complete the matching, it took a lot of time to learn the regular expression, there are a lot of attempts, debugging from the simple mode to the extend mode process.

 

0x10 Performance Analysis and code optimization

1.word_frequency.exe D: \ test 316 ms

2.word_frequency.exe-E2 D: \ test 475 Ms

3.word_frequency.exe-E3 D: \ test 559 Ms

It can be seen that the program running performance mainly depends on the performance of Regular Expression matching. The change in the Three-test time mainly stems from the increase in the complexity of matching words, but there is no space for optimization, the method for improving the performance of a star is to use multiple threads and process multiple files at the same time, which can effectively reduce the running time and sacrifice some memory to improve the performance.

 

Summary after 0x11

1) never imagined that it was finally completed before deadlin. Although this job has completed all the functions, but from the program performance or their own code style, did not do well. C # is easy to use. multi-threaded implementation is powerless. Although it has no impact on a small number of files, it has a moderate performance in a large number of file test lines. There are many reusable code segments in the code, however, I chose CTRL C + V in shame, which leads to high code redundancy and hope to avoid it in the next project as much as possible.

2) Deficiencies in the project process:

. The knowledge of common classes and methods is seriously insufficient. It takes a long time to find a parameter;

In the process of code, I always want to use ready-made methods. Where are all the algorithm ideas ?!

. I still haven't written multiple threads...

3) the gains are not small. C # has increased by several percentage points in terms of proficiency, and finally learned the regular expressions that sound amazing. I also learned a lot about the general process of program testing.

　　　　　　

Individual project-word_frequency