A bitter lesson of linux Server Management

A bitter lesson of linux Server Management-general Linux technology-Linux technology and application information. The following is a detailed description. The system uses fedroa8. When the data center installs the system, the partitions are managed by lvm by default.

Later, the data center switched the power off, and it was estimated that there was something wrong with the file system. All the disks were read-only. Then I want to check the disk and run the fsck command. The result shows that the check failed, and the file system was detached again. All commands are useless. We had to restart the IDC, and the system would not be able to get up. I suspect that the system did not respond when I restarted the system at the data center. A few more times, the system was interrupted when checking the hard disk.

To save the data, I had to add a new hard disk, mount the system to the new hard disk, and mount the old hard disk. I slowly recovered the data.

First, we encountered the issue of lvm partition mounting, which cannot be directly mounted like other partitions. I checked it online:

First use vgscan to find the virtual volume, and then use vgchange-ay to activate the virtual volume. The virtual volume device is available under/dev. The default value is/dev/VolGroup00/LogVol00/dev/VolGroup00/LogVol01.

Mount with mount/dev/VolGroup00/LogVol00/root/vg0. the error "mount: you must specify the filesystem type" is returned.

I checked it with fdisk and found that the virtual volume was identified, but the partition table was damaged and the system could not identify the partition format. You have to use fsck to fix the issue. Don't stop prompting, start watching, then get bored and add a fsck-y. Good. All the way smoothly. check done. Mount the virtual volume and enter the directory. It's dumb. There is only one lost + found folder in the partition. Nothing else. Enter the lost + found, ls, and wait for half a day to show the result. It turns out that all files have been moved to this file, and all files have been renamed to start with #, a string of numbers ending with tens of thousands of files. To cry.

In desperation, I tried grep and found that some folder names have changed but the structure is still there. In a hurry, I wrote a shell script to restore data. Fortunately, the main data is basically restored.

I have summarized the following lessons:

1. fsck cannot be used in disorder. First, remove the file system umount and then check. It is best to start fsck in single-user mode.

2. Do not use lvm when necessary. I really don't know why fedora uses lvm by default. For the sake of experimental technology? This ghost makes the hard disk one by one. It must be completely damaged, and it also affects the disk I/O performance. You can see that the boot folder is also put under lvm. When the system cannot start up, you want to manually load the kernel address, and you do not know how to express the kernel address.

3. Partitioning and backup are too important. The datadir of the database must be manually specified to the partition where data is stored, so that data will not be lost when the system is reinstalled. The configuration files under the etc directory must be backed up. Otherwise, after the system is reinstalled, it is acceptable to reconfigure them.

I usually use linux on my own and have little experience in server management, which makes old birds laugh. I have never been wise.