LINUX REISERFS 6 disk RAID5 mail server recovery Experience

New network, Enterprise mail server, stored in the 146gx6 RAID5, there are millions of enterprise users of the mail, data area, only one district, file system for ReiserFS, normal work, RAID suddenly offline, the administrator to the computer room inspection, found two disk alarm, After one of the forced online found that the volume could not mount, so forcibly fsck and Rebuld tree, lasted 4 days, after completion still cannot mount. Unfortunately, most companies are unable to provide viable solutions to data recovery companies. The new network in the multi-party comparison and evaluation, choose let us finish.

Analysis

This raid problem is in fact very common, usually because the light of the two disk is not at the same time off the line, and coincidentally, forced online early offline hard drive, resulting in the data area fresh and old mixed together, file system structure is inconsistent. When it is forced online, a new inspection stripe will be generated during the reading and writing process, which will affect some of the data, but the severity of the disaster will be much smaller if read-write is low or not, and the most serious problem in this case is rebuild tree, The equivalent of trying to keep a mixed file system continuous. Such a result would lead to a complete error in all the structure of the filesystem, which is usually irreparable. In addition, the user's file directory structure is very complex, the total number of files roughly estimated billion, but also a slim chance.

[Solution]

1, should attempt to the file system structure area separately proposed to carry on the analysis, such workload will be much smaller, also gives the possibility for the repeated search analysis. However, the file system area of ReiserFS is relatively scattered and irregular, it is necessary to extract and analyze by the autonomous program, in this case, the size of the 1-level node is up to 6G, and the file structure is complex. (Users are also due to the EXT3 faced with such a structural collapse to choose ReiserFS, visible its structural complexity)

2, the file system area for consistency testing, equivalent to manual fsck, correction error Place, in this case, many file system node area due to the test relationship, so that the key attribute byte changed. Unified initialization of all node states through the program, complete node consistency processing

3. There are two ways to accomplish the above two steps, one is in the Linux system again fsck, this example is not good, (because of the limited functionality of the Linux fsck, in the parent node a little error, the child nodes will be all into the lost+found, unable to restore the original directory structure), the second is through read-only mode, Using autonomous programs to extract data under Windows, you need to ignore many errors, modify the program, use this method, all data can be extracted.

PostScript

Recently this two hard drives offline, do not know which block first away, which block after the example of a lot. It is hoped that the raid user will be cautious when the two hard drives are offline, and that if logs can be traced, the log is OK. If forced online error, should immediately stop operation, must not do fsck and other operations.

In addition, the commonplace thing, the Linux fsck risk is very large (actual Windows will also have), please try to see the prompts before, if the error information is abnormal, you should choose another way.