How to bulk process big data

1. Through the data status (not processed, processed, processing completed)

2. Split-page layering

3. Lock select for update via data (multithreaded security)

4. Business

Instance:

User form, redemption eligibility form, redemption eligibility statistics

1. Tens of thousands of users

2. The user obtains several redemption qualifications through the sign-in

3. Scheduled tasks at a fixed time to count the total number of non-redeemed per user last year

4. The user can be redeemed manually, at this time not redeemed and exchange, also need to update the statistics table

Specific implementation:

By select non-processing the user and lock, and then update the processing status to (in process)

Loop through the N data.

for{

Lock statistics

UPDATE STATISTICS

Update user tables for processing completion (these three steps in the same transaction)

}

Issue 1: Timed tasks Multithreading how each user does not repeat

Lock N users with select and then update those users into eligibility statistics. (These two steps are in the same transaction)

Then process the N users (in another transaction)

2: What happens if the user table status is still handled if an exception occurs at the end of processing? How do you differentiate between processing or handling exceptions?

Manually updating all processes before a point in time is not processed, such as 10 days ago. At this point even if two threads are processed at the same time, because the tab is locked, it will be processed successively, there is no data security problem.

How to bulk process big data