List and Dictionary generic search efficiency analysis, listdictionary

List and Dictionary generic search efficiency analysis, listdictionary

The efficiency of List and Dictionary generic queries is significantly different. The background is to develop a matching program and recommend books (BookID) to users (UserID, there is a rule that the same book cannot be recommended to the same user within seven days.

The implementation of rules that the same book cannot be recommended to the same user within seven days is a process of continuous program optimization. The first version of the program directly retrieves the database and queries whether there are any records within seven days based on the BookID + UserID, if yes, no allocation is made. However, as the data volume increases, the program runs longer and longer, and optimization begins. The first optimization is to extract all the data from the past seven days, put it in the List <T>, and then search in the memory. It is found that the efficiency is only slightly improved, but not obvious. The second optimization adopted Dictionary <TKey, TValue>. Unexpected results were not very good, and the program efficiency was improved several times.

The following is a pseudo-code, which simplifies the program code. It only shows the differences between List and Dictionary efficiency and does not have practical significance.

/// <Summary> /// Efficiency Test of the Collection class /// </summary> public class SetEfficiencyTest {static List <TestModel> todayList = InitTodayData (); static List <TestModel> historyList = InitHisoryData (); public static void Run () {CodeTimer. time ("ListTest", 1, ListTest); CodeTimer. time ("DictionaryTest", 1, DictionaryTest);} public static void ListTest () {List <TestModel> resultList = todayList. findAll (re => {if (histor YList. exists (m => m. userID = re. userID & m. bookID = re. bookID) {return false;} return true;});} public static void DictionaryTest () {Dictionary <int, List <string> bDic = new Dictionary <int, list <string> (); foreach (TestModel obj in historyList) {if (! BDic. containsKey (obj. userID) {bDic. add (obj. userID, new List <string> ();} bDic [obj. userID]. add (obj. bookID);} List <TestModel> resultList = todayList. findAll (re => {if (bDic. containsKey (re. userID) & bDic [re. userID]. contains (re. bookID) {return false;} return true;}) ;}/// <summary> // initialize data (today) /// </summary> /// <returns> </returns> public static List <TestModel> InitTodayData () {List <TestModel> list = new List <TestModel> (); for (int I = 0; I <10000; I ++) {list. add (new TestModel () {UserID = I, BookID = I. toString ()});} return list;} // <summary> // initialize the data (History) /// </summary> /// <returns> </returns> public static List <TestModel> InitHisoryData () {List <TestModel> list = new List <TestModel> (); Random r = new Random (); int loopTimes = 60000; for (int I = 0; I <loopTimes; I ++) {list. add (new TestModel () {UserID = r. next (0, loopTimes), BookID = I. toString ()});} return list ;} /// <summary> /// test object /// </summary> public class TestModel {// <summary> // user ID /// </summary> public int UserID {get; set ;}/// <summary> /// book ID /// </summary> public string BookID {get; set ;}}}View Code

The output is as follows:

Unexpectedly, there are so many differences in efficiency between the two. Next, we will study the reasons for the huge difference between the two.

Implementation of the List <T>. Exists () function:

Public bool Exists (Predicate <T> match) {return this. FindIndex (match )! =-1;} public int FindIndex (Predicate <T> match) {return this. findIndex (0, this. _ size, match);} public int FindIndex (int startIndex, int count, Predicate <T> match) {if (startIndex> this. _ size) {ThrowHelper. throwArgumentOutOfRangeException (ExceptionArgument. startIndex, ExceptionResource. argumentOutOfRange_Index);} if (count <0 | startIndex> this. _ size-count) {ThrowHelper. throwArgumentOutOfRangeException (ExceptionArgument. count, predictionresource. argumentOutOfRange_Count);} if (match = null) {ThrowHelper. throwArgumentNullException (ExceptionArgument. match);} int num = startIndex + count; for (int I = startIndex; I <num; I ++) {if (match (this. _ items [I]) {return I;} return-1 ;}View Code

List <T>. Exists essentially searches for the data through a loop, and each call will repeat the loop, so the efficiency is very low. Obviously, this is not desirable.

Implementation of the Dictionary <TKey, TValue>. ContainsKey () function:

Public bool ContainsKey (TKey key) {return this. findEntry (key)> = 0;} // System. collections. generic. dictionary <TKey, TValue> private int FindEntry (TKey key) {if (key = null) {ThrowHelper. throwArgumentNullException (ExceptionArgument. key);} if (this. buckets! = Null) {int num = this. comparer. gethashcodes (key) & 2147483647; for (int I = this. buckets [num % this. buckets. length]; I> = 0; I = this. entries [I]. next) {if (this. entries [I]. hashCode = num & this. comparer. equals (this. entries [I]. key, key) {return I ;}} return-1 ;}View Code

Dictionary <TKey, TValue>. ContainsKey () is implemented through Hash search internally, so the efficiency is much higher than List.

Finally, we provide the following suggestions on MSDN:

1. if you need to add, delete, and search projects very quickly, and do not care about the order of items in the set, you should first consider using System. collections. generic. dictionary <TKey, TValue> (or you are using.. NET Framework 1.x, you can consider Hashtable ). All three basic operations (add, delete, and include) can be performed quickly, even if the collection contains millions of projects.

2. If you rarely need to delete or add a large number of entries in the usage mode, but it is important to keep the sequence of the set, you can still select List <T>. Although the search speed may be slow (because the basic array needs to be traversed when searching for the target project), it can ensure that the Set maintains a specific order.

3. You can select Queue <T> to implement the first-in-first-out (FIFO) sequence or Stack <T> to implement the last-in-first-out (LIFO) sequence. Although Queue <T> and Stack <T> both support enumeration of all items in the set, the former only supports insertion at the end and deletion from the beginning, while the latter only supports insertion and deletion from the beginning.

4. If you need to keep the sequence while implementing fast insertion, use the new sequence list <T> set to help you improve performance. Unlike List <T>, List <T> is implemented as a dynamically allocated object chain. Compared with List <T>, you only need to update two connections and add new projects to insert objects in the middle of the set. From the performance perspective, the disadvantage of the link list is that the garbage collector increases its activity because it must traverse the entire list to ensure that no object is released. In addition, because of the overhead related to each node and the location of each node in the memory, performance problems may occur in the large link list. Although the actual operation of inserting a project into the shortlist <T> is much faster than inserting a project in List <T>, however, you still need to traverse the list and find the correct position to insert a new value.

Reference: CLR full introduction: Best practice set, search efficiency of List and hashtable