C ++ is sometimes slower than Python

Recently, the Department is engaged in dynamic JVM languages, such as groovy. After enjoying the flexibility of dynamic languages, the performance is naturally offered to the primary key.

Then, my colleagues playing with Python mentioned the old things and found a piece of Python code from the Internet. Many Python people know it and many c ++ people know it and it runs very fast.

In order to make the article look better, I will compile a story, saying that a software company is about to recruit C ++ programmers and Python programmers who have just graduated from college. The interview questions are the same:

There is a 0.15 million-line document, many of which have the same content. Please write a piece of code, read this file, and output non-repeated row content to the new file as soon as possible, do not change the order. For example, there are five rows:

B1

B2

A2

B2

B1

The content of the new file must be:

B1

B2

A2

 

Shortly after Mr. P learned pythont, he wrote the following code:

Code:

1. Import time
2. If _ name _ = "_ main __":
3. Beg = Time. Time ()
4. Hashtable = {}
5. FI = file ("test.txt", "R ")
6. Fo = file ("test_dst.txt", "W ")
8. Count = 0;
9. For line in FI:
10. If not hashtable. has_key (line ):
11. Hashtable [Line] = 1
12. Fo. Write (line)
13. Count = count + 1
15. Fo. Close ()
16. Fi. Close ()
18. End = Time. Time ()
20. Print "Count =", count, "time =", (end-Beg) * 1000, "Ms"

This is a well-defined pythont program. It conforms to the python style: it cannot be seen whether it is a novice or a handwritten program. :)

OK, on my machine, I have prepared a 0.15 million text file, but there are only 40 thousand lines of files without repeating the line. The running time of the above Code is 281 milliseconds. The performance is faster than the C ++ code with the same logic for a while, so the Small P is in phase.

 

The answer is the c ++ interview. Well, students who do not understand hash may be abandoned. The use of vector and list will be very slow. Maybe map users can consider it, the reason is that there is no hash Data Structure in STL strictly. On the other hand, some C ++ programmers who have carefully studied the data structure at school and actually used STL should know almost all version STL implementations, all of them are added to implement hash. Furthermore, in terms of performance, using map is still very slow. You should know the complexity of find and the multiples of hash tables.

Fortunately, although he has learned C ++ for a while in the story, he understands hash and also writes a piece of well-regulated code:

Code:

1. # Include <iostream>
2. # Include <string>
3. # Include <fstream>
4. # Include <sstream>
5. # Include <ctime> // For clock ()
6. # Include <EXT/hash_map>
8. Using namespace STD;
9. Using namespace _ gnu_cxx;
11. Struct string_hash
12. {
13. Size_t operator () (const string & Str) const
14. {
15. Return _ stl_hash_string (Str. c_str ());
16. }
17. };
19. Struct string_compare
20. {
21. Bool operator () (string const & str1, string const & str2)
22. {
23. Return str1 = str2;
24. }
25. };
27. Int main ()
28. {
29. Clock_t beg = clock ();
31. Hash_map <string, Int, string_hash, string_compare> hastable;
33. Ifstream fi ("test.txt ");
34. Ofstream fo ("test_dst.txt ");
36. String line;
37. Size_t COUNT = 0;
39. For (Getline (FI, line );! Fi. EOF (); Getline (FI, line ))
40. {
41. If (hastable. Find (line) = hastable. End ())
42. {
43. Hastable [Line] = 1;
44. Fo <line <Endl;
45. ++ Count;
46. }
47. }
49. Fo. Close ();
50. Fi. Close ();
52. Clock_t end = clock ();
53. Cout <"Count =" <count <", time =" <(end-Beg) <"Ms" <Endl;
55. Return 0;
56. }

The written code is a little longer than the Python version, but the logic is the same. The score should be the same as that of the small p version. But now we are struggling. Do we want to reduce the size of C! Why? As a beginner and a programmer with the same question, as a C ++ programmer, he writes this question, and the performance is worse than that of Python! We compiled his code into a speed-optimized version. It ran 10 times in a row with the same test conditions. The slowest version took more than one second, and the fastest was 600 milliseconds. Far less than the 281 ms of Python version. If optimization is to be implemented, we know that with psyco, the code can be increased by nearly 100 milliseconds without modifying the code line ......

I want to reject Mr. C and say what's better? Some people say that he doesn't know the advantages of using APIs. He can directly call the file ing and other technologies of Windows APIs, you can also choose not to use the _ stl_hash_string algorithm that generates the hash value from the string provided by STL (complain that small C is not doing well in Mathematics ?); You can also blame him for being lazy and using the C ++ extension library hash container. In fact, it only takes about 1 thousand lines to write a hash table with a higher query performance ...... (See here)

Yes, this is the c ++ language. I really don't like some kind of "Philosophy". Experts can always use various cool techniques to write something quite unusual, it can be an admirable distance from beginners ......

This is not what I like! If the C ++ language is learned, it is enough to turn a beginner into an expert (and then he is deemed to have entered the door, I really don't want everyone to be proficient in this algorithm, proficient in countless libraries, and proficient in the countless APIs of windows or Linux. Haha, I am not a real educator either, this is a bit heavy.

The real fact is that I, a programmer who mainly uses C ++, and my colleague who plays Python (he basically does not program at work ), after work, I had a drink and talked about the performance PK. You said, I took out hundreds or even thousands of lines of code, go to the master's simple and elegant 20 lines of code. Well, I am confused. In addition, the code can be executed after the next carriage return in Linux. Do you mean that I have to write thousands of lines of code? How many lines of Windows API calls are stored in Wei suo?

If you lose, you will lose, but you still need to optimize it within the acceptable range. I am not completely ignorant of Python. I also want to give some insider information about dynamic languages. Haha. The acceptable scope is as follows: 1. Do not call the API of a specific operating system, 2. Do it in the C ++ environment.

Code:

1. Int main ()
2. {
3. Clock_t beg = clock ();
5. // Optimization 1: Set a sufficient number for hastable.
6. Hash_map <string, Int, string_hash, string_compare> hastable (40050 );
8. Ifstream fi ("test.txt ");
9. Istringstream SSI;
11. Fi. seekg (0, ifstream: End );
12. Size_t filesize = Fi. tellg ();
13. Fi. seekg (0 );
15. // Optimization 2: read all the content of the file to the memory at a time
16. Char * Buf = new char [filesize];
17. Fi. Read (BUF, filesize );
19. // Optimization 3: directly constructs a "memory stream" from the memory where data is stored to avoid replication.
20. SSI. rdbuf ()-> pubsetbuf (BUF, Fi. gcount ());
21. Fi. Close ();
23. String line;
24. Size_t COUNT = 0;
26. Ostringstream SSO;
28. // Optimization 4: read each row from the memory stream instead of the file stream
29. For (Getline (SSI, line );! SSI. EOF (); Getline (SSI, line ))
30. {
31. If (hastable. Find (line) = hastable. End ())
32. {
33. Hastable [Line] = 1;
34. // Optimization 5: first output to the memory stream, not to the file stream
35. SSO <line <Endl;
36. ++ Count;
37. }
38. }
39. Delete [] Buf;
41. Ofstream fo ("test_dst.txt ");
42. Fo <SSO. STR () <Endl;
43. Fo. Close ();
45. Clock_t end = clock ();
46. Cout <"Count =" <count <", time =" <(end-Beg) <"Ms" <Endl;
48. Return 0;
49. }

The file size is not considered to exceed 2 GB. I think this "interview" is acceptable. Another thing to repeat is that clock_t is much larger in Linux. You can use clock_per_sec (?) Macro solution, which is omitted here.

1. Number of hash tables

To create a hash table, you must first reserve more than 40 thousand entries (similar to the number of lines with different content in the file under test. (Maybe your hash does not have this function)

2. Read all the content of the file at a time

One-time reading of files, which can be justified. It is estimated that the C language implementation of Py will also play with this.

3. directly convert the memory to "stream"

After reading the data into the memory, you can use the "pubsetbuf" function to directly convert it into a stream (stringstream). This is also inevitable and can avoid a large memory application, but more importantly, memory replication. I believe that for most languages, or even static Java languages, this is an inevitable Implementation Method for such requirements.

4. Read from the memory stream, not from the file stream

This is also the goal of step 1.

5. First output to the memory, and then output to a file.

This should be the first thought, and the effect may be the best.

----------------------------------

In this case, it is slower than Python: the fastest time is 296 seconds. The gap between several 1‰ seconds ...... My colleagues are about to sacrifice psyco.

 

What should I do? It's okay. I can say that winning or losing is not the purpose. It's important to learn from each other and understand the mechanisms between different languages, my c ++ level is very general ...... Some people will have more beautiful and elegant optimizations.

-Supplement :-----------Final optimization: Upgrade the Compiler----------------------------------------------------------

Some netizens in the comments below said that he only optimized the read/write Io and used the hash_map under tr1 (Technical Report 1, which will be officially added soon), which is faster. Thanks to his reminder, we may try to make it simple (I am not going to change the code in this small game), but the final tactics of effective performance optimization are as follows: my Python colleague psyco is not terrible, so we can enjoy the compiler upgrade when we sit down.

I remembered that the compiler I used was a very old mingw32-gcc3.4.5, paid effort, downloaded the new version from the Internet installed (4.4.0, it seems not the latest GCC ).

A line of code is not changed. It is upgraded from 296 to 187. Now, C ++ is better than the Python code of the unoptimized version, which can be compared with the version of psyco.

--------------------------------------------------------------------

The most important thing is the performance and elegance. Sometimes the latter is more important. Otherwise, I will not be playing with python for so many years. The performance is worse. But it is really elegant. Until I found out that, unless you are in a hurry, you can still use C ++ very elegantly. Elegance is not a personal task. It is wonderful to have a team of people who keep elegance.

To put it another simple principle: Do not think about performance optimization at the beginning. Therefore, we need small C.

 

------------ Welcome to the publication of "Vernacular C ++ -------------------------

Busy with house ...... I haven't been here for a long time. It's okay for all of you ~~~ If you want to communicate with me, click the following link to become a friend:
Http://student.csdn.net/invite.php? U= 112600 & C = f635b3cf130f350c