Analysis of Dictionary & lt; TKey, TValue & gt; In. NET

In. NET, Dictionary <TKey, Tvalue> is a very common key-value data structure, that is, a hash table .. NET also has a type called Hashtable, both of which are hash tables. Both types can implement the key-Value Pair storage function. The difference is that one is generic, the other is non-and the internal implementation is somewhat different. Today, I will study Dictionary <TKey, TValue> in. NET and some related issues. If any error occurs in this article, I hope to point it out. What is the definition of Hash table In the hash table Wikipedia? In computing, a hash table (also Hash map) is a data structure used to implement an associative array, a structure that can map keys to values. it is a data structure that directly accesses the memory storage location through keywords. This is a data structure that all data structures have in textbooks. We will not do much research here. However, there are several concepts to mention, because it is very useful for us to understand the internal implementation of Dictionary. More hash table content: Wikipedia, Hashtable blog Collision (Collision) and processing because the hash algorithm we use in the data structure is not a perfect hash algorithm, at the same time, we will limit the memory space we use to store. Therefore, collision cannot be avoided. Therefore, it is very important to consider how to deal with collision when designing a hash table. There are many ways to handle collisions, such as the Open Adressing method and the Separate chaining method ). The Dictionary uses a method called Separate chaining with linked lists. The figure on the Wikipedia below can clearly identify what this method looks like. The example of the separated link method uses two arrays. The buckets array saves only one address, which points to an instance (entry) in the entries array ). When the hash value conflicts, you need to add a new instance to the end of the linked list of the Instance currently pointed. The loading Factor exists because when the content in the array increases in the open addressing method, the probability of conflict increases, in the open addressing method, the conflict solution adopts the detection method, which will cause great performance loss. This figure in wikipedia compares the relationship between the decentralized link and the linear Probing Method with the CPU cache miss when different loading factors exist. Cpu cach miss-load factor in this method adopted by Dictionary, the loading factor is not an important factor and will not have a great impact on performance, therefore, Dictionary uses 1 by default and does not think it is necessary to provide any interface to set this value. How to Implement the inside of a Dictionary: int [] buckets and Entry [] entriesIEqualityComparer <TKey> comparer. these two are the two arrays mentioned above, the so-called Separate chaining with linked lists. When adding a new value to a Dictionary, you need to calculate the Hashcode Of the key. When a conflict arises, you also need to determine whether the two values are equal. This comparer is used for this purpose. Why not directly call the GetHashCode and Equal methods of key overloading? This will be discussed below. Insert uses an example to describe what a Dictionary did during insertion. Dictionary <int, string = ""> dict = new Dictionary <int, string = ""> (); dict. add (0, "zero"); dict. add (12, "twelve"); dict. add (15, "Ten"); dict. add (4, "four"); the following "figure" shows the changes of the two arrays in the insert operation. With the source code, you can find out what happened. --------- | Buckets | entries | ------- | 0 | --> | hashcode = 0, key = 0, next =-1, value = "zero" | ------- |-1 | empty | ------- |-1 | empty | ------- | --------- | buckets | entries | ------- | 1 | --> | hashcode = 0, key = 0, next =-1, value = "zero" | ------- |-1 | --> | hashcode = 12, key = 12, next = 0, value = "twelve" | ------- |-1 | Empty | ------- | --------- | buckets | entries | ------- | 2 | --> | hashcode = 0, key = 0, next =-1, value = "zero" | ------- |-1 | --> | hashcode = 12, key = 12, next = 0, value = "twelve" | ------- |-1 | --> | hashcode = 15, key = 15, next = 1, value = "shortten" | ------- | --------- | buckets | entries | ------- | 0 | --> | hashcode = 0, key = 0, next =-1, Value = "zero" | ------- | 2 | --> | hashcode = 12, key = 12, next =-1, value = "twelve" | ------- |-1 | --> | hashcode = 15, key = 15, next =-1, value = "shortten" | ------- |-1 | --> | hashcode = 4, key = 4, next =-1, value = "four" | ------- | 3 | empty | ------- | 1 | empty | ------- |-1 | empty | ------- | during the last insert operation in the expansion example above, dictionary performs a scale-out. It expands the Dictionary size from the original 3 to 7. It can be seen that the elements in entries do not change much during resizing, but there are some changes in next, because their hash value % length is no longer mapped to the same value after resizing, you do not need to share a value. I think there are two points worth mentioning. How can we get the next size for expansion? In the above example, why is it 7? Expansion is a change of two arrays. For the first problem, because the array length in Dictionary is limited, the location in a bucket array is obtained through key. GetHashCode () % length and then the next of the entry is changed. Therefore, we need to minimize the number of conflicts brought about by modulo, so the prime number can ensure that the modulo can be scattered in the array as much as possible. When the Dictionary function is expanded, the current capacity is * 2 first, and then a prime number closest to the value is found in a prime number table. This prime number table is long like this: public static readonly int [] primes = {3, 7, 11, 17, 23, 29, 37, 47, 59, 71, 89,107,131,163,197,239,293,353,431,521,631,761,919,110 3, 1327,159 7, 1931,233 3, 2801,337 1, 4049,486 1, 5839,701 3, 8419,101 03, 12143,145 91, 17519,210 23, 25229,302 93, 36353,436 27, 52361,628 51, 75431,905 23, 108631,130 363, 156437,187 751, 225307,270 371, 324449,389 357, 467237,560 689, 672 827,807 403, 968897,116 2687, 1395263,167 4319, 2009191,241 1033, 2893249,347 1899, 4166287,499 9559, 5999471,719 9369}. For the second question, we mentioned it in the previous article, this method does not need to re-hash the content stored in the hash table during resizing. We only need to re-perform the modulo operation on the hash value corresponding to the distributed elements in the bucket, and then place it on the new location. This operation is extremely fast. Search for key. GetHashCode () % length --> In the traversal chain table, find the equal key and delete the same query. Several Notes: performance problems we usually get used to when using Dictionary as in Code 1. This method is okay when we use the built-in type as the key, but if we need to treat a custom value type (struct) as the key, we need to pay attention to it. There is a problem that can be easily ignored, which may cause a large amount of unnecessary performance overhead when using a Dictionary. When we need to define some custom structures and put these instances in the set, we often use value types instead of defining them as a class, if these types only have data, the performance of the value type is much better than that of the class. (Choosing Between Class and Struct) Let's first conduct an experiment to compare the performance gap Between the value type and the Class as the key. The experiment code is as follows. In this code, I inserted 1000 to 10000 pieces of data to obtain the required time. Public class/struct CustomKey {public int Field1; public int Field2; public override int GetHashCode () {return Field1.GetHashCode () ^ Field2.GetHashCode ();} public override bool Equals (object obj) {CustomKey key = (CustomKey) obj; return this. field1 = key. field1 & this. field2 = key. field2 ;}} Dictionary <mykey, int = ""> dict = new Dictionary <mykey, int = ""> (); int tryCount = 50; double totalTime = 0.0; for (int count = 1000; count <10000; count + = 1000) {for (int j = 0; j <tryCount; j ++) {Stopwatch watcher = Stopwatch. startNew (); for (int I = 0; I <count; I ++) {MyKey key = new MyKey () {Field1 = I * 2, field2 = I * 2 + 1}; dict. add (key, I);} watcher. stop (); dict. clear (); totalTime + = watcher. elapsedMilliseconds;} Console. writeLine ("{0}, {1}", count, totalTime/tryCount);} The result is like this: Cl Ass vs struct WTF? Why is it different from my expectation? Shouldn't it be faster than the value type? Orz... here we will mention the IEqualityComparer <TKey> comparer and Dictioanry mentioned above all use this instance for internal comparisons. But we didn't specify it, so it uses EqualityComparer <TKey>. Default. Let's take a look at the source code to see how the Default came from, in CreateComparer, we can see that if our type is not byte, IEquatable <T> is not implemented, it is not Nullable <T>, or enum, an ObjectEqualityComparer () is created by default (). However, the Equal and GetHashCode methods in ObjectEqualityComparer <T> () do not seem to have any problems. What exactly is the problem? Performance problems related to value types can be immediately considered as the performance loss caused by packing and unpacking. Is there such an operation? Let's take a look at the following two sections of the Pipeline Code. ObjectEqualityComparer. Equals (T x, T y) IL code // Methods. method public hidebysig virtual instance bool Equals (! T x ,! T y) cel managed {// Method begins at RVA 0x62a39 // Code size 50 (0x32). maxstack 8 IL_0000: ldarg.1 IL_0001: box! T IL_0006: brfalse. s IL_0026 IL_0008: ldarg.2 IL_0009: box! T IL_000e: brfalse. s IL_0024 IL_0010: ldarga. s x IL_0012: ldarg.2 IL_0013: box! T IL_0018: constrained .! T IL_001e: callvirt instance bool System. Object: Equals (object) IL_0023: ret IL_0024: ldc. i4.0 IL_0025: ret IL_0026: ldarg.2 IL_0027: box! T IL_002c: brfalse. s IL_0030 IL_002e: ldc. i4.0 IL_002f: ret IL_0030: ldc. i4.1 IL_0031: ret} // end of method ObjectEqualityComparer '1: Equals ObjectEqualityComparer. the IL code of Equals (T x, T y. method public hidebysig virtual instance int32 GetHashCode (! T obj) cel managed {. custom instance void System. runtime. targetedPatchingOptOutAttribute ::. ctor (string) = (01 00 3b 50 65 72 66 6f 72 6d 61 6e 63 65 20 63 72 69 74 69 63 61 6c 20 74 6f 20 69 6e 6c 69 6e 65 20 61 63 72 6f 73 73 20 4e 47 65 6e 20 69 6d 61 67 65 20 62 6f 75 6e 64 61 72 69 65 73 00 00) // Method begins at RVA 0x62a6c // Code size 24 (0x18 ). maxstack 8 IL_0000: ldarg.1 IL_0001: box! T IL_0006: brtrue. s IL_000a IL_0008: ldc. i4.0 IL_0009: ret IL_000a: ldarga. s obj IL_000c: constrained .! T IL_0012: callvirt instance int32 System. object: GetHashCode () IL_0017: ret} // end of method ObjectEqualityComparer '1 :: getHashCode from the above two sections of code, we can see that many box (see highlighted rows) operations exist in the default Implementation of ObjectEqualityComparer. It is used to pack the value type into the reference type. This operation is time-consuming because it needs to create an object and copy the value in the value type to the newly created object. (The CustomKey. Equal method also has an unbox operation ). How can this problem be solved? I don't think it is enough to avoid packing, so we can create a Comparer ourselves. Public class MykeyComparer: IEqualityComparer {# region IEqualityComparer Members public bool Equals (CustomKey x, CustomKey y) {return x. field1 = y. field1 & x. field2 = y. field2;} public int GetHashCode (CustomKey obj) {return obj. field1.GetHashCode () ^ obj. field2.GetHashCode () ;}# endregion} then we will slightly modify the experiment code (Dictionary <CustomKey, int> dict = new Dictionary <CustomKey, int> (new MykeyComparer ());) in the test One. The results show that the performance has improved a lot.