Overview of Linux log file systems _unix Linux

A file system is a system software that manages and organizes data stored on disk drives, guaranteeing data integrity, which guarantees the consistency of the data written to the disk and the content that is subsequently read. In addition to saving data stored as files, a filesystem also stores and manages some important information about the file and filesystem itself (for example, date time, owner, access rights, file size and storage location, and so on). This information is often referred to as metadata (metadata).

Because the general file system works asynchronously to avoid disk-access bottlenecks, it can cause data loss if a disk operation is interrupted abruptly. For example, if this happens: what happens when you process a document on a Linux ext2 file system and suddenly the machine crashes?

There are several possibilities:

* When you save the file, the system crashes. This is the best situation and you will not lose any information. You only need to restart your computer and continue working.

* The system crashes before you save the file. You will lose all of your work, but older versions of the document will still exist.

* System crashes when the saved document is being written to disk. This is the worst case scenario: The new file overwrites the old version of the file. This leaves only one part of the new part of the old file on the disk. If the file is a binary file, the file cannot be opened because the file format is different from what the application expects.

In the last case, this could be worse if the system crash occurs when the drive is writing metadata. This is when the file system is corrupted, and you may lose the entire directory or the entire disk partition's data.

The Linux standard file system (EXT2FS) will attempt to recover corrupted meta data information by calling the file scanning Tool at reboot. Because the ext2 file system holds backups of redundant critical metadata information, it is generally unlikely that data will be completely lost. The system accountant calculates the location of the corrupted data, either by recovering the redundant metadata information, or by directly deleting files that are corrupted or damaged by the metadata information.

Obviously, the larger the file system to be detected, the longer the detection process takes. For partitions with a size of dozens of G, it can take a long time to detect. As Linux starts to be used for more and more important applications in large servers, it is becoming more and more intolerable for long time machines. This requires more sophisticated and sophisticated file systems to replace Ext2.

Therefore, the log-file system (journalling filesystems) is present to meet this requirement.
What is a log-file system

This is only a simple description of the log-file system. If you need more in-depth information, refer to the article log file system, or the log-file system introduction.

Most modern file systems use log technology from the database system that is developed to improve crash resilience. Disk transactions are written sequentially to a specific location in the log area (or log area) of the disk before they are actually written to the disk.

The information written to the log area is not exactly the same, depending on the technology implemented by the log file system. Some implementation techniques only write file system metadata, while others record all writes to the log.

Now, if the crash occurs before the contents of the log are written, the original data is still on disk, and only the latest updates are lost. If the crash occurs in a real write operation (that is, the log content has been updated), the log contents of the log file system show what has been done. So when the system reboots, it can easily recover the corrupted updates based on the log content.

In either case, the full data is obtained and no corrupted partitions are present. Because the recovery process is based on the log, the entire process can take only a few seconds.

It should be noted that using the log file system does not mean that you do not need to use the file Scanning Tool fsck at all. Hardware and software errors for randomly occurring file systems are not recoverable by log, and must be assisted by the Fsck tool.

The current log file system in Linux environment

Three log file systems are discussed in the following sections: The first is ext3, developed by the Linux kernel, Stephen Tweedie. Ext3 is implemented by adding log functionality to the Ext2 file system, which is currently the default file system for redhat7.2, Namesys developed reiserfs log file system that can be downloaded and is currently Mandrake8.1 using this log file system. SGI published the XFS log-file system in March 2001. can be downloaded in oss.sgi.com/projects/xfs/. The following three log file systems will be tested with different tools for detection and performance testing.

Install Ext3

For EXT3 file System technical issues please refer to Dr. Stephen Tweedie's papers and interviews. The ext3 log file system comes directly from its ancestor ext2 file system. It has a key feature that is completely backward compatible, and in fact it simply adds log functionality to the Ext2 log file system. Its biggest disadvantage is that there is no modern file system has the ability to improve file data processing speed and compression of high-performance.

Ext3 from 2.2.19 as a patch. If you want to add support for the kernel to the Ext3 file system, you will need to use patches to get patches, which require the following files:

* EXT3-0.0.7A.TAR.BZ2: Kernel patches

* E2FSPROGS-1.21-WIP-0601.TAR.BZ2 support EXT3 e2fsprogs Program Kit

Copy linux-2.2.19.tar.bz2 and ext3-0.0.7a.tar.bz2 to the/USR/SRC directory for decompression:

MV Linux Linux-old
TAR-IXVF linux-2.2.19.tar.bz2
TAR-IXVF ext3-0.0.7a.tar.bz2
CD Linux
Cat.. /ext3-0.0.7a/linux-2.2.19.kdb.diff | Patch-sp1
Cat.. /ext3-0.0.7a/linux-2.2.19.ext3.diff | Patch-sp1

First add SGI KDB kernel debugger patches to the kernel, and the second is ext3 file system patches. You need to configure the kernel and answer yes to the "Enable Second extended FS Development Code" section of the file system. Then compile.

After the kernel compiles and installs, the E2fsprogs software kit needs to be installed:

TAR-IXVF e2fsprogs-1.21-wip-0601.tar.bz2
CD e2fsprogs-1.21
./configure
Make
Make check
Make install

The job to do is to create a ext3 file system on the partition and reboot with the new kernel, when you have two options to create a new log file system or to upgrade an existing Ext2 file system to the Ext3 log file system.

For situations where you need to create a new ext3 file system, you can create a new ext3 file system by using the MKE2FS command in the installed E2fsprogs package plus-F parameters:

Mke2fs-j/dev/xxx

Here/dev/xxx is the new partition that you want to create the Ext3 file system. The-j parameter represents the creation of the ext3 rather than the ext2 file system. You can use the parameter "-jsize=" to specify the size of the log area you want (n units of M).

To upgrade an existing ext2, use TUNE2FS on it:

Tune2fs-j/dev/xxx

You can upgrade the file system you are loading and the file system that is not loaded. If the current file system is being loaded, the file. Journal is created in the directory where the file system mount point resides. If you are upgrading a file system that was not loaded at that time, use the implied system Inode to log the logs, at which point all the contents of the file system will be preserved and not corrupted.

You can use the following command to load the Ext3 file system:

Mount-t Ext3/dev/xxx/mount_dir

Since Ext3 is actually a ext2 file system with a log function, a ext3 file system can be loaded in ext2 way.

Install XFS file system

If you need to learn about the XFS file system from a technical aspect, refer to SGI's XFS file system and SGI information page. You can also refer to the FAQ.

XFS is a SGI development of the Linux environment of the log file system, it is a mature technology, initially used in the IRIX system file system. XFS follows the GPL copyright statement. The current XFS file system is the latest version of 1.02. To download a kernel that supports patches on the kernel XFS file system or directly downloads the RPM package, we'll patch you through how to use XFS for the 2.4.14 kernel. Download the following content first

patch-2.4.14-xfs-1.0.2.bz2
patch-2.4.14-xfs-1.0.2-kdb.bz2

Copy the Linux kernel linux-2.4.2.tar.bz2 to the/USR/SRC directory, modify the old kernel directory name, and then unpack the new kernel:

MV Linux Linux-old
TAR-IXF inux-2.4.2.tar.bz2

Copy each patch to the kernel source directory (for example:/usr/src/linux) and Patch:

Zcat patch-2.4.14-xfs-1.0.2.bz2 | Patch-p1
Zcat patch-2.4.14-xfs-1.0.2-kdb.bz2 | Patch-p1

Then configure the kernel to open the kernel options for the File System section: "XFS filesystem Support" (CONFIG_XFS_FS) and "Page Buffer Support" (CONFIG_PAGE_BUF). You will also need to upgrade the following system tools to the following or higher versions:

modutils-2.4.0
autoconf-2.13
e2fsprogs-devel-1.18

Install the new kernel and restart the server.

Then download the XFS tool. This package includes the following command to process the file system, using the following command to install the package:

TAR-ZXF xfsprogs-1.2.0.src.tar.gz
CD xfsprogs-1.2.0
Make Configure
Make
Make install

After you install these commands, you can create a new XFS file system:

Mkfs-t xfs/dev/xxx

If XXX is an existing file system, then you need to use the "-F" parameter to create a new partition, but remember that this will destroy all the data for that partition.

Mkfs-t xfs-f/dev/xxx

Once created, you can use the following command to load a new file system:

Mount-t Xfs/dev/xxx/mount_dir

Installing the ReiserFS file system

If you want to learn more about the ReiserFS file system from a technical aspect, refer to the Namesys and FAQ.

The ReiserFS file system starts with 2.4.1-pre4 as the officially supported file system of the Linux kernel. In order to use the ReiserFS file system you first need to install the File System Support tool on the system (for example, the Mkreiserfs tool that creates the ReiserFS file system). The latest version of the ReiserFS file system can be added to the 2.2.x or 2.4.x kernel in a patched way. Here we take 2.2.19 as an example:

The first step, first under the kernel source, and under the ReiserFS file System 2.2.19 Patch, the current patch is the latest version of LINUX-2.2.19-REISERFS-3.5.34-PATCH.BZ2. You should also download the tool package: reiserfsprogs-3.x.0j.tar.gz.

Then extract the kernel source and patch package into the/USR/SRC:

TAR-IXF linux-2.2.19.tar.bz2
Bzcat linux-2.2.19-reiserfs-3.5.34-patch.bz2 | Patch-p0

The compiler kernel supports reiserfs and installs the kernel. Then install the file system Tools software:

Cd/usr/src/linux/fs/reiserfs/utils
Make
Make install

Install the new kernel and reboot. You can now create a new ReiserFS file system and load:

Mkreiserfs/dev/xxxx
Mount-t Reiserfs/dev/xxx/mount_dir

File System performance Test

The computer environment used for the test environment is as follows: Pentium III-16 Mb RAM-2 Gb HD, and the operating system is RedHat6.2. All file systems work properly, so benchmark analysis is performed to compare them. First I unplug the system directly to simulate the system power off, to test the log file system recovery process. All file systems are successfully scanned for file scans, and after a few seconds the system scans and then starts the system normally.

The next step is to test with a bonnie++ performance test program that accesses a file for database types, creates, reads, and deletes small files, which are the most common operations for Squid, inn, or mail server programs in maildir format (qmail). The Performance Test command is:

bonnie++-d/work1-s10-r4-u0

It tested 10Mb (-S10) on the file system loaded in the/WORK1 directory. Therefore, you must create the appropriate type of file system and load it into the directory/work1 before you perform the test. The other parameters specify the m number of the memory size (-R4) and run the test program as root, and the test results are as follows:

Each test has two sets of data: File system speed (K/SEC) and CPU Utilization (%CPU). The higher the speed, the better the file system. And for CPU rate, the smaller the number of performance the better. You can see that the ReiserFS file system has the best performance in file operations (sequential create and random create parts) and is 10 times times more than other file systems. Other aspects (sequential output and sequential Input) are comparable to other file system performance. There is no particular difference between the other file systems. XFS performance is close to the Ext2 file system, and the Ext3 file system is slightly slower than ext2 (because it takes some extra time to log logs). Finally, the MONGO of the obtained performance test program is used and modified to test three kinds of log file systems. This adds a command to load XFS and ext3 file systems in the mongo.pl program, formats them, and then starts the performance test analysis. The script format divides the area/dev/xxxx, loads it, and runs a specified number of processes at each stage: creating, copying, symbolic connection processing, reading, displaying file status information, renaming and deleting files. Also, the program calculates the number of segments (fragmentation) after the creation and copying stages.

FRAGM = Number_of_fragments/number_of_files

You can get the same test comparison results in the result file:

Log-original result
LOG.TBL-The output of the comparison program
Log_table-Tabular results

The following commands are tested:

mongo.pl EXT3/DEV/HDA3/WORK1 LOGEXT3 1

If you want to test other file systems, you need to modify the ext3 in the parameters of the above command to ReiserFS or XFS. The other parameters are the partitions to load, the load path, the file name of the test result, and the number of processes that were started.

The following table is the test result. The data unit is seconds. Lower values perform better. The first table test uses a data block size of 100 bytes, the second table is 1000 bytes, and the last one is 10000 bytes

From the table above you can see that ext3 has a better performance in state deletion and renaming, while the ReiserFS file system performs better in file creation and copy performance. You can also see that reiserfs, as its technical documentation mentions, has a pretty good performance in small file processing.

Conclusion

At present Linux has at least two robust and reliable log file systems available (XFS and ReiserFS), which are widely used. For example, Mandrake8.1 supports the ReiserFS file system by default.

As you can see from the results of the performance test, ReiserFS is the best choice.