First-recognized Log File System in centos-ext3

First-recognized Log File System in centos-ext3

Outline

1. Log File System

2. Advantages of ext3

3. Three log modes of ext3

4. Select log Mode

1. Log File System

Generally, when a file is written to a system running, the metadata of the file (such as permission, owner, Creation Time, and access time) is not written ), if the system shuts down abnormally after the file content is written and before the file metadata is written, the file system uninstalls the file during the write process abnormally, and the file system is in an inconsistent state. When the system restarts, Linux runs the fsck program, scans the entire file system, ensures that all file blocks are correctly allocated or used, finds the damaged directory item, and tries to fix it. However, The fsck does not guarantee repair damage. In this case, inconsistent metadata in the file fills up the space of the lost file, and the file items in the directory items may be lost, resulting in the loss of the file.
To minimize File System inconsistencies and shorten the startup time of the operating system, the file system needs to track records that cause system changes. These records are stored in a separate location from the file system, usually we call it "log ". Once these logs are securely written, the log file system can apply them to clear records that cause system changes and make them a set that causes changes to the file system, place them in the transaction processing of the database, and maintain the normal operation of valid data in the state, not in conflict with the performance of the entire system. When any system crashes or needs to be restarted, the data is restored according to the information records in the log file. Because log files have regular checkpoints, they are usually very neat. The file system design mainly considers efficiency and performance issues.
Linux supports many log file systems, including FAT, VFAT, HPFS (OS/2), NTFS (Windows NT), UFS, XFS, JFS, ReiserFS, ext2, and ext3.

2. Advantages of ext3

Why do you need to migrate data from ext2 to ext3? There are four main reasons: availability, data integrity, speed, and ease of migration.

• Availability

After an abnormal host (power failure or system crash), The ext2 file system can be loaded and used only after the e2fsck is used for consistency verification. The time for running e2fsck depends on the size of the ext2 file system. It takes a long time to verify larger file systems (dozens of GB. If the number of files in the file system is large, the verification time is longer. It may take an hour or longer to verify a file system of several hundred GB. This greatly limits availability. In contrast, ext3 does not require file system verification unless a hardware fault occurs. This is because the data is always written to the disk in the same way as the file system. After an abnormal shutdown, the recovery time of the ext3 file system does not depend on the file system size or number of files, but on the size of the "log" required for maintaining consistency. With the default log settings, the restoration takes only one second (depending on the hardware speed ).

• Data Integrity

When an ext3 file system is used to shut down abnormally, the complete data performance is reliably guaranteed. You can select the type and level of data protection. You can choose to ensure the consistency of the file system, but allow the data on the file system to be damaged during abnormal shutdown; this can increase the speed (but not all) in some situations ). You can also choose to keep the data reliability consistent with the file system; this means that you will not see any data spam in the newly written files after the server goes on. The default setting is used to ensure data reliability consistent with that of the file system.

• Speed

Although ext3 writes more data than ext2, ext3 is often faster than ext2 (high data stream ). This is because the log function of ext3 optimizes the rotation of the Hard Disk Head. You can select one of the three log modes to optimize the speed and selectively sacrifice some data integrity. The first mode is data = writeback, which is used to ensure data integrity and allow the old data to exist in files after the server. This mode can increase the speed in some cases. (In most log file systems, this mode is set by default. This mode provides limited data integrity for the ext2 file system, and is more to avoid long file system verification during system startup, data = orderd (default mode) to ensure data reliability is consistent with that of the file system. This means that you will not see any junk data in newly written files on the machine. The third mode, data = journal, requires larger logs to ensure moderate speed in most cases. It takes a longer time to recover the instance after the host is on. However, in some databases, the operation speed is faster. We recommend that you use the default mode. To change the mode, add the data = mode option to the file system in the/etc/fstab file. For more information, see the man page online manual for the mount command (execute man mount ).

• Easy to migrate

You can easily migrate from ext2 to ext3 without reformatting the hard disk and enjoy the advantages of a reliable log file system. Yes, you do not need to perform a long, boring, and potentially incorrect "backup-Reformat-Restore" operation to experience the advantages of ext3. There are two migration methods:

1. If you upgrade your system, the Red Hat Linux installer will assist in migration. The job you need to do is to click the select button for each file system.

2. You can use the tune2fs program to add the log function to the existing ext2 file system. If the file system has been mounted during the conversion process, a file will appear in the root directory ". journal "; if the file system is not loaded, the file will not appear in the file system. To convert a file system, you only need to run tune2fs-j/dev/hda1 (or any device name of the file system you want to convert ), at the same time, change the ext2 in the/etc/fstab file to ext3. If you want to convert your own root file system, you must use the initrd boot to start. Follow the mkinitrd manual description to run the program and check that initrd is loaded in your LILO or GRUB configuration. (if not, the system can still be started, but the root file system will be loaded as ext2, instead of ext3, you can use the command cat/proc/mounts to confirm this .) For more information, see the man page online manual for the tune2fs command (run man tune2fs ).

3. Three log modes of ext3

Ext3 supports multiple log modes, including changing the metadata of the file system or the data of the file system (including changes to the file itself, the following three log modes are activated when the/etc/fstab file is booted:

• Data = journal log Mode
  Logs contain all the data and metadata that change the file system. It is the slowest of the three ext3 log modes, but it minimizes the possibility of errors. Using the "data = journal" mode requires that ext3 write each change twice to the file system and write once to the log, which reduces the overall performance of the file system. All new data is first written into the log before being located. After an accident occurs, the log can be replayed to bring the data and metadata back to the consistent state. Because metadata and data updates are recorded in ext3, these logs work when a system restarts.

• Data = ordered log mode (default)
  Only the metadata of the file system is changed, and the overflow file data must be added to the disk. This is the default ext3 log mode. This mode reduces the redundancy between writing to the file system and writing logs, so the speed is fast. Although changes in file data are not recorded in logs, they must be done, in addition, the daemon program of ext3 is executed before the related file system metadata changes, that is, the file system data must be modified before the metadata is recorded, which will slightly reduce the system performance (speed ), however, the file data in the file system can be synchronized with the metadata of the corresponding file system.

• Data = writeback log Mode
  Only the metadata of the file system is recorded, but according to the standard file system, the write program still records the changes of the file data on the disk to maintain the consistency of the file system. This is the fastest ext3 log mode. Because it only records changes in metadata, and does not need to wait for updates related to file data, such as file size and directory information, updates to file data and changes in recorded metadata can not be synchronized, that is, ext3 supports asynchronous logs. The defect is that when the system is shut down, the updated data cannot be written to the disk, which cannot be solved yet.
  There are differences between different log modes, but the setting method is as convenient. You can use the ext3 File System to specify the log mode, which is completed at/etc/fstab startup. For example, if you select data = writeback log mode, you can make the following settings:
  /Dev/hda5/opt ext3 data = writeback 1 0
  In general, the data = ordered log mode is the default mode of the ext3 file system.

To specify the log mode, use the following method:

1. Add an appropriate string to the/etc/fstab option field, for example, data = journal.

#/Dev/sda3/varext3defaults, data = writeback12

2. Specify the-o data = journal command line option when calling mount.

# Mount-odata = journal/dev/sdb1/mnt

If you want to view how to query logs of a file system, run the dmesg command:

# Dmesg | grep-B1 "mountedfilesystem" kjournaldstarting. Commitinterval5secondsEXT3-fs: Small. -- EXT3FSonsda1, internaljournalEXT3-fs: Small. -- EXT3FSonsdb1, internaljournalEXT3-fs: Small. -- EXT3FSonsdb1, internaljournalEXT3-fs: small.

4. Select log Mode

• Speed

In some typical cases, using data = writeback can significantly increase the speed, but it also reduces the protection of data consistency. In these cases, the protection of data consistency is basically the same as that of the ext2 file system, the system constantly maintains the integrity of the file system (this is the log mode used by other log file systems ). This includes frequent shared write operations and frequent creation and deletion of a large number of small files, such as sending a large number of small email messages. If you switch from ext2 to ext3 and find that the application performance is greatly reduced, the data = writeback option may help you improve performance. Even if you do not get expensive data consistency protection measures, you can still enjoy the benefits of ext3 (file systems are always consistent ). Red Hat is still working to improve the performance of ext3 in some aspects, so some aspects of ext3 performance can be improved in the future. This also means that even if you have selected data = writeback, you need to re-test the future version with the default value of data = journal to determine whether the change of the new version is related to your work.

• Data Integrity

In most cases, you write data at the end of the file. In some cases (such as databases), users write data between existing files. You can even overwrite an existing file by truncating the file and then writing data to the end of the file. In data = ordered mode, if the system crashes while writing a file, the data block may be partially rewritten, but the write process is not complete, therefore, the system has incomplete data blocks that do not belong to any files. In the data = ordered mode, the only condition of unordered data blocks after a crash is that a program is overwriting a file during the crash. In this case, the write order cannot be absolutely guaranteed unless the program uses fsync () and O_SYNC to force write operations in a specific order.

The ext3 file system also involves how to cache data to the hard disk. It is implemented through the kupdate process for regular refreshing. By default, it is checked once every 5 seconds, and dirty data over 30 seconds is flushed to the hard disk.

In as 3.0, you can modify/proc/sys/vm/bdflush. In as 4.0, you can modify/proc/sys/vm/dirty_writeback_centisecs and/proc/sys/vm/dirty_expire_centisecs.

Because the default ordered mode is used, in this mode, if an IO writes data files before writing log files. If crashed occurs in the system after the data file is written and before the log file is written, the data will be lost, which is absolutely not allowed in the database, whether it is Oracle or MySQL. SoFor database writes, each write operation will first be written to pagecache, and then notify kernelthread to fl the buffers to the hard disk, then write the metadata to the log, and finally return the write success operation. In this way, writing to a database is obviously not as fast as writing to a slave device.

So when using Ext3 to run the database,Setting the log mode to the journal mode should improve the performance (no tests have been conducted, so should be theoretically analyzed ). Because in journal mode, a write operation is performed on the database. First, changes in data and file systems are directly written to the log (directly written without the cache, with better performance), and then the data is written to the cache, then, the kupdate process refreshes the data to the hard disk. In contrast, for DB, its performance should be faster than the previous one.

In addition, we also mention the sync_binlog parameter in MySQL. If this parameter is set to 1, the binlog file will be flushed to the hard disk at the same time, just like the Oracle write IO. If this parameter is disabled, it is handed over to the OS for management, that is, it is checked every 5 seconds, and old data 30 seconds ago is found to be flushed to the hard disk. The innodb_flush_log_at_trx_commit parameter also involves hard disk flushing.

As an enhanced version of ext2, ext3 is almost identical to ext2's superblock, inode, group descriptor, and other data structures. Therefore, ext3 is backward compatible with ext2. You can use:

1#tune2fs–j/dev/sd1

Directly convert the ext2 File System to the ext3 file system without detaching the partition.

If we suddenly lose power when editing files, or the system is locked and forced to restart, what will happen? In this case, part of the file is lost, and the content of the entire file is messy. In addition, the file system crashes directly. This would be a terrible thing. When linux is shut down normally, we will see a Print message for uninstalling the file system. Otherwise, the file system may be inconsistent, this inconsistency will be found when the file system is mounted during the system restart phase, and then it will try to fix it. Unfortunately, as the storage device capacity increases, the time it takes for such repairs is becoming increasingly intolerable.

The biggest feature of Ext3 is the addition of the log function on the basis of ext2. Therefore, ext3 file systems are often called log file systems. However, the log file system is not only ext3, there are also JFS, reiserFS and XFS, and NTFS that we often see on Windows.

The log feature of Ext3 mainly relies on an intermediate Device named "log Block Device layer" in its lower layer, called JBD (Journaling Block Device layer ). JBD is not part of the file system specification. It has nothing to do with the ext3 file system specification, and JBD is the basis for implementing the file system transaction processing function. In short, JBD is designed to achieve the Special Purpose of logging on any block device (the more abstract the transaction is, the more compact the transaction is ....)

Those who have experience in database development or data O & M are certainly not unfamiliar with transactions. Here we will not end the concept, nor stick to the academic definition, as long as you knowThe main role of a transaction is to ensure the atomicity of the operation. How can this sentence be understood? For example, in the financial system, you need to transfer X yuan from account A to account B. This service must ensure that X yuan is successfully allocated from account A, and then X yuan is successfully added to account B. Only when the two operations are successful at the same time can the task be transferred successfully. If any operation fails, the business must be terminated. If the transfer from account A is successful and an error occurs when writing data to account B, the X yuan transferred from account A must be returned to account. In more extreme cases, the data of account A crashes for various reasons, so the transaction mechanism of the database must ensure that the X yuan of account A will not be lost. This is the atomicity of database business operations. In the log file system, the atomicity of file data operations is ensured by JBD. Ext3 implements its logging function through the APIs of hooking in JBD. Although the JBD layer itself does not have much code, it is a very complicated software part. Here we will not miss it, and we will have the opportunity to play with it later.
Of course, the log file system must record the log, and the log also occupies the storage space. Therefore, the log file system opens a special block area on the storage medium for storing log information:

We use an image to briefly describe the layout of the bottom layer of ext3: