Paralleldbset. Add. add

Paralleldbset. Add. add
Problems Found

The requirement is very simple. It is generally to write data to the database in batches, So we plan to write data in Parallel, hoping to use the multi-core computer features to speed up program execution. I thought it was wonderful, so I quickly compiled a code string similar to the following:

                using (var db = new SmsEntities())                {                    Parallel.For(0, 1000, (i) =>                    {                        db.MemberCard.Add(new MemberCard()                        {                            CardNo = "NO_" + i.ToString(),                            Banlance = 0,                            CreateTime = DateTime.Now,                            Name = "Test_" + i.ToString(),                            Status = 1                        });                    });                    db.SaveChanges();                }

Unexpectedly, the following error was returned:

The amazing thing is that when I refresh again, the exceptions are different, so I refresh them several times. The following are the exceptions I have encountered:

1. You have not set the object reference to the instance of the object.

2. You have added an item with the same key.

3. The set has been modified. enumeration may not be performed.

4. An EdmType cannot be mapped to the CLR class multiple times. EdmType "SmsModel. MemberCard" is mapped more than once.

Among them, 1 and 2 are the most common, and all exceptions occur when they are added ~ There is no way. I can't find the reason for one-to-one breakpoint debugging. For the sake of progress, I switched to another solution, that is, the AddRange method of DbSet. First, a List of entities is accumulated in Parallel, and then added to DbSet at one time. The Code evolves:

            List<MemberCard> list = new List<MemberCard>();            using (var db = new SmsEntities())            {                var result = Parallel.For(0, 1000, (i) =>                  {                      list.Add(new MemberCard()                      {                          CardNo = "NO_" + i.ToString(),                          Banlance = 0,                          CreateTime = DateTime.Now,                          Name = "Test_" + i.ToString(),                          Status = 1                      });                  });                if (result.IsCompleted)                {                    db.MemberCard.AddRange(list);                    db.SaveChanges();                }            }

Then compile and test. If there is no problem, put it first.

 

Analyze problems

The company was still struggling with this issue the next day, so we opened the page and entered 1000 million data records (the number of cycles in the real project was manually entered). click the button to submit, an exception like eating melons... :

I thought yesterday's test was good (in fact, I entered 10 yesterday, and I felt guilty about my face...). I couldn't help it. I was shocked when I reached the breakpoint:

Obviously, there are only 1000 pieces of data in 971 cycles, and there is null in the results. After multiple debugging, it is found that this is a random phenomenon, and Count is random and null, sometimes there is sometimes no. It is preliminarily determined that this is a resource allocation exception caused by multithreading. So, let's take a look at the List introduction on MSDN. The last line "thread security" says:

Everything seems clear, so I plan to verify the result and add the lock. The test result is:

There is no null in the list, and it is confirmed that the exception is caused by multi-thread security. So I thought about whether the problem was the same yesterday. Then I searched for the DbContext class and DbSet class on MSDN. They all said this:

Then the dbcontext is locked and tested. This test finally works perfectly as expected. But what I don't know is how the first few exceptions are generated. Although the number of objects in the List is not enough, there are null objects, but no exceptions are directly reported. Now I only know that it is a thread problem, and I am not clear about the details. If you know it, please give me some advice.

Find the solution and verify the conclusion

I also thought about using Partitioner partitions, but I think that although the partition is a single thread, there are multiple threads between the partitions. If the partition is too small, it will lose the meaning of Parallel, you have to find another way out. Fortunately, the Framework also provides some thread-safe generic sets (such as ConcurrentBag and ConcurrentQueue) for us, but it uses locks in essence, therefore, we compared the performance of single-thread list, multi-thread list lock, multi-thread ConcurrentBag, and multi-thread ConcurrentQueue. The results are as follows:

1000 cycles:

10000 cycles:

100000 cycles:

 

• The conclusion is that when the number of executions is too large, the thread security type will be slower, and there is no advantage for the thread security type when the number of executions is small.

 

Solve the problem

Finally, after careful testing, verification, and consideration of the actual needs of the project (almost impossible at a time 10000), we can simplify the process, return to the original, and use the simplest and straightforward writing method single-thread loop to complete the process. Although I was still back to the beginning, I found unexpected problems in the process, found the cause, tested and verified, and finally got the optimal solution. In that sentence, you are more powerful than before!