Encapsulate multiple threads to process massive data operations (1)

Cause:

Encapsulate multiple threads to process massive data operations (2)

Recently I was writing a Data ImportProgram, You need to extract n more data from several old data tables, and then process it and add it to the corresponding table of the new database. The single-step operation is too slow. Isn't that exactly the purpose of multithreading?

I have to write a set of similarCodeThe ideographic code is as follows:

 
// Obtain a batch of old data dataset dsuser = olddbaccess from the old database. getoldusers (minid); // divide the data in dataset into N parts and place them in the able [] sectionusers = new datatable [threadcn]; // declare threadcn autoresetevent, let them issue the execute semaphores autoresetevent [] evts = new autoresetevent [threadcn] After the thread completes execution; // initialize the evts value // put the data and autoresetevent in one data and submit it to threadpool for processing. The specific processing method is omitted.
 

Since there are n kinds of old data, and each processing method is different, so I wrote n processing steps similar to the above, which is too tired, so I want to refactor, extract the same part of the above operation steps. With the asynchelper static class, this static class has several public static methods to perform the same process in the above steps for processing large amounts of data by multiple threads.

The signature of the first method should be like this.

/// <Summary> /// execute multi-threaded operation tasks /// </Summary> /// <Param name = "datacollection"> multi-threaded operation data set </param> /// <Param name = "threadcn"> How many threads are used for processing </param> /// <Param name = "processitemmethod"> processing a single data in a dataset </param> Public static void doasync (ilist datacollection, int threadcn, waitcallback processitemmethod)
 

A large amount of data defines a class of the ilist type as a parameter to pass. The number of threads required to process the data is specified using threadcn. It should be noted that waithandle. the waitall method determines that threadcn must be smaller than 64. The last parameter is a method for processing a large amount of data. The method signature has been obtained, and I have done several repeated operations on it. Writing this method should not be a problem. Similarly, we also need to divide the ilist data into threadcn portions, and then hand each portion to threadpool for processing. This is very simple.

/// <Summary> /// execute multi-threaded operation tasks /// </Summary> /// <Param name = "datacollection"> multi-threaded operation data set </param> /// <Param name = "threadcn"> How many threads are used for processing </param> /// <Param name = "processitemmethod"> processing a single data in a dataset </param> Public static void doasync (ilist datacollection, int threadcn, waitcallback processitemmethod) {If (datacollection = NULL) throw new argumentnullexception ("datacollection"); If (threadcn> = 64 | threadcn <2) {Throw new argumentoutofrangeexception ("threadcn", "The threadcn parameter must be between 2 and 64");} If (threadcn> datacollection. count) threadcn = datacollection. count; ilist [] colls = new arraylist [threadcn]; autoresetevent [] evts = new autoresetevent [threadcn]; for (INT I = 0; I <threadcn; I ++) {colls [I] = new arraylist (); evts [I] = new autoresetevent (false) ;}for (INT I = 0; I <datacollection. count; I ++) {object OBJ = datacollection [I]; int threadindex = I % threadcn; colls [threadindex]. add (OBJ) ;}for (INT I = 0; I <threadcn; I ++) {threadpool. queueuserworkitem (doprivate, new object [] {colls [I], processitemmethod, evts [I]});} waithandle. waitall (evts);} Private Static void doprivate (Object Data) {object [] datas = data as object []; ilist datalist = datas [0] As ilist; waitcallback method = datas [1]; autoresetevent EVT = datas [2] As autoresetevent; foreach (Object item in datalist) {method (item);} EVT. set ();}

This is easy to implement, but since we have to consider more, our thread is like one by one investigators. The task assigned to them this time is to capture an enemy and come back to ask the enemy, the task requires only one enemy. That is to say, if a scout catches one enemy, it will send a message to other comrades in arms to tell them that they are not busy and the task has been completed. What should we do? There are always more methods than problems.

The waithandle class has the waitany static method. Isn't the scout example a waitany? The main thread needs to notify all threads after receiving a signal from the completion of a thread. "The task is finished, let's go home "? If you are interested, you can give your own solutions. My solutions will be released tomorrow. Tomorrow, we will also solve the problem of getting the return values of multiple executed operations.