RAID1 source code analysis

RAID1 source code analysis

The overall step for correct writing process is that raid1 receives the upper-layer write bio, applies for an r1_bio structure, and points all bios [] to this bio. Assume that there are N disks in the disk array. Then, clone N upper-layer bio structures, point each bios [] to a cloned bio structure, and then set accordingly.

If the Write Behind mode is not available, put all these bios [] (shared Page Structure) into the queue pending_list and set the memory bitmap. Then, the daemon extracts bio from the pending_list chain, synchronizes the memory bitmap to the disk, and immediately sends bio to the disk at a time, returns the write success, updates the bitmap status, and then asynchronously refreshes the disk. 4.

For the Write Behind mode, you also need to copy the received top bio page structure to the bios [] corresponding to the WriteMostly disk (one copy for each WriteMostly disk ), then, put all these bios [] into the queue pending_list and set the memory bitmap. Then, the daemon extracts bio from the pending_list chain, synchronizes the memory bitmap to the disk, and immediately delivers bio at a time. If only the WriteMostly disk is not completely written successfully (that is, the non-WriteMostly disk is successfully written), The write operation is successful and the result is returned. After all the WriteMostly disks are actually written, the copied Page Structure and rw.bio will be released. Update the bitmap status at the same time, and then asynchronously fl the disk. 1 and 2.

The overall function call relationship, process switching relationship, and major process are shown in figure 3.

Figure 1 Write process without the Write Behind Mode

Figure 2 Write process in Write Behind Mode

Figure 3 Overall raid1 read process framework

The write process mainly involves the following functions:

Request function make_request

Write Request raid1d

Callback Function raidincluend_write_request

Write error handling raid1d

The specific analysis and writing process is as follows.

1) The request function make_request

After the write request is encapsulated into bio, The md_make_request of the md device sends the request, and the md sends the request to the specific device raid1, corresponding to the make_request function of raid1, the following describes the process of this part from make_request of raid1. The overall process 4 is shown.

Figure 4 Overall framework of the write process of the make request Function

The code is analyzed as follows:

1. Call md_write_start and wait until the disk array is updated quickly.

1.1 if it is not a write operation, the system returns the result directly.

1.2 If the array is in the temporary read-only status, set it to the read/write status, set the MD_RECOVERY_NEEDED bit of the mddev array, and wake up the daemon and synchronization daemon threads.

Note:

Set_bit (MD_RECOVERY_NEEDED, & mddev-> recovery); indicates that resync or recovery may be required;

Resync synchronizes data on each sub-device. The rediscovery process restores data.

1.3 if the array is in safe mode, it is set to insecure mode.

1.4 If the in_sync of the mddev array is 1, set in_sync to 0, indicating that the array is about to start writing. Wake up the daemon.

Set_bit (MD_CHANGE_CLEAN, & mddev-> flags); that is, clear the MD_SB_CLEAN mark in the superblock.

1.5 synchronize the in_sync flag to the array superblock on the disk.

2. If barrier is set for access requests and the MD device (raid1 here) does not support barrier setting, terminate bio and return immediately. The-EOPNOTSUPP information is fed back to the upper layer.

Note: The barrier here refers to the barrier attribute of bio.

3. Wait until the barrier on the device is eliminated.

Note: This is a set of barriers that raid1 itself performs for synchronization.

4. Apply for an r1_bio structure (this structure is mainly used to manage the bio of RAID 1). An array of bios arrays in this structure points to the bio of each disk.

5. traverse all disks in the disk array.

5.1 if the disk exists but is Blocked (Blocked), jump out of the loop and wait for the blocking to be eliminated. re-enter the beginning of the loop. (Usually set and clear the ioctl by the user)

5.2 if the disk exists and the disk is not broken (! Faulty [fp6]) to increase the output IO count of the disk.

5.2.1 if the disk is Faulty (Faulty), reduce the output IO count of the disk. Set bio of the disk in the bio [] array of rw.bio to NULL.

5.2.2 direct the disk in the rw.bio array to user bio. Targets is used to indicate available disks.

5.3 if it is another case (it must be an error), set bio of the disk in the rw.bio array to NULL.

6. If the number of available disks in the disk array targets is smaller than conf-> raid_disks, some disks are broken. Set the disk array to the R1BIO_Degraded status.

7. If delayed writing is set, copy and save the user bio data in behind_pages by calling the alloc_behind_pages function. Set the disk array to the R1BIO_BehindIO status.

8. Set the number of incomplete requests for r1_bio and the number of incomplete requests for delayed write to 0.

9. Based on the BIO_RW_BARRIER flag in bio, determine whether to set the barrier flag in bio. That is, to determine whether set_bit (R1BIO_Barrier [fp7], & rw.bio-> state) is required ).

Note: Determine whether to set the barrie mark in raid-bio Based on the mark in user bio;

If the disk to be attached does not support the barrier operation, you can process it in raidreceivend_write_request. The specific process is to re-try it in the daemon process.

10. initialize a bio_list chain b1.

11. traverse all disks in the disk array.

11.1 For each disk, clone a user bio to the bios of the corresponding element of the rinibio array, and set related fields and the callback function raid1_end_write_request.

11.2 if delayed write is set, bio_vec of each element in the array bios in rw.bio points to the saved delayed write copy behind_pages. If the WriteMostly mode is set, a number of incomplete write requests are added to the disk array.

11.3 r0000bio-> remaining records the number of unsubmitted requests. Each disk is added to 1.

11.4 mount the cloned bio to bio_list chain b1.

12. Call bitmap_startwrite to notify bitmap to set data blocks.

13. Add the cloned b1 (same bio in multiple copies) to the pending_bio_list chain of RAID 1.

14. If the user IO is sync io, wake up the daemon raid1d and switch the process to raid1d. The Daemon continues to process the r1_bio request by operating the pending_bio_list chain.

2) Write requests to send raid1d

Pending_bio_list all bio items are submitted together, And rbio in retry_list is processed one by one.

If the pending_bio_list queue is not empty (there are waiting access requests), submit these requests one by one. Before submitting a write request, You Need To fl the memory bitmap to the disk (to avoid data loss or errors in the memory when power loss occurs) to ensure that bitmap is written before data writing. Until all requests of the pending_bio_list linked list are submitted.

The normal process goes down and write requests are sent here. 5.

Figure 5 Write requests issued by the daemon

3) callback function raid1_end_write_request

The overall process is shown in Step 6.

First, we do not consider the error process. Assume there are five disks, three of which are WriteMostly disks. When Write Behind is set, behind remaining = 3, remaining = 5.

If one WriteMostly disk and one non-WriteMostly disk are returned. Then there are two WriteMostly disks and one non-WriteMostly disk. In this case, behind remaining = 2 and remaining = 3. If a non-WriteMostly disk is returned, the judgment statement behind remaining> = remaining-1 does not need to be subtracted. Therefore, this condition is true. Set R1BIO_Returned and endio to notify the upper-layer Write Request has ended. At this time, only the WriteMostly disk is left, which achieves the effect of delayed writing. However, at this time, related structures such as rinibio and behind pages have not been released. After the WriteMostly disk is returned, save_put_page () and bitmap_endwrite () are released to release the behind pages and rw.bio structures.

If all the WriteMostly disks are returned and no non-WriteMostly disks are returned, behind remaining

If you do not set Write Behind, it is easy to understand. Refer to the flowchart and the following code for reading and analyzing.

Figure 6 process of the raid1_end_write_request Function

The following describes the specific code process:

1. Select the disk number mirror for the bio to be called back and complete.

2. If the request requires setting barrier, but the mounted device does not support barrier, set the disk array to R1BIO_BarrierRetry. Go to Step 8.

Note: In this case, the raid 1 device supports barrier bio, but the underlying device does not. The barrier here is different from the barrier at the beginning of the make request. The-EOPNOTSUPP value here, is the value passed in by the lower-level callback after the delivery. The-EOPNOTSUPP passed in bio_endio in make_request calls back-EOPNOTSUPP to the upper layer of raid1. One is the information returned to the received lower-layer device, and the other is the information returned to the upper-layer device.

3. Set r1_bio-> bios [mirror] pointer to NULL. (The original region has not been released. Use the to_put pointer to find it)

4. If the status is not "valid" (not uptodate), set the disk to an error. And downgrade the disk array.

5. If the status is "valid", set the disk array to R1BIO_Uptodate.

6. Record the location of the operation ended on the disk.

7. If there is a delay in writing.

7.1 If the disk is WriteMostly, the number of uncompleted write requests delayed is-1.

7.2 if only the WriteMostly disk requests are left and the r1_bio status is R1BIO_Uptodate, the write operation is considered successful and endio is returned.

7.3 reduce the io delivery count of the disk.

8. Reduce a remaining and check whether all requests have been completed (the remaining is 0 ). If all requests in rw.bio are completed, proceed to the following process. This indicates that the request is complete and the related structure can be released.

8.1 if the R1BIO_BarrierRetry status (previously set), add the r1_bio to the retry queue. Jump to the retry process.

8.2 release delayed write pages.

8.3 set the bitmap attr attribute to CLEAN.

8.4 security mode.

8.5 end io.

9. If the count is 0, release to_put bio.

After a write request is sent to the disk, clear the corresponding bit in the bitmap Memory Page and click the Download button next to the bitmap file. These are done through the daemon process, which does not need to wait for the write bitmap disk file to complete, so it is asynchronous. (Completed by bitmap_daemon_work) bitmap does not need to be synchronized here, because it can ensure data correctness. Even if the write fails, it can bring up to extra synchronization without causing data harm.

4) write error handling raid1d

If the receiving upper-layer bio fails because the barrier attribute is set and the sub-device does not support the barrier (this happens only in write operations), the barrier attribute of rw.bio is cleared, resubmit this r1_bio.

The specific process for the daemon process to handle this write error is 7.

Figure 7 daemon processes write errors caused by barrier bio

The code process is as follows:

1. Clear the R1BIO_BarrierRetry and R1BIO_Barrier status bits of r1_bio.

2. Increase the number of r1_bio-> remaining requests in the disk array, and the number is the number of disks in the disk array.

3. For each disk in the disk array, clone master_bio and initialize it. (Each page of the original failed bio must be copied to the new bio one by one, because the write behind device may exist ).

4. issue this new bio.

For more details, please continue to read the highlights on the next page:

1 2 Next Page

[Content navigation]
Page 1: Write Process	Page 1: synchronization process