Linux Operations Learning Notes-Document System Knowledge System Summary

File System Knowledge Summary

What do I need to do to store the data for a newly purchased hard drive?

First, the hard disk installed RAID, after the raid to partition, after the completion of the format to create a file system, and finally storage data.

internal and external structure of the hard drive:

Physical shape:

Interface type:

The IDE (Integrated drive Electronics) is an "electronic integrated driver".

SCSI (Small computer system Interface) small computer system interface.

SATA (Serial Advanced technology Attachment) serial drive.

The SAS (Serial attached SCSI) serial-attached SCSI is a new generation of SCSI technology that is backwards compatible with SATA.

Purchase Disk Considerations:

1. Spindle Rotation: 5400/7200/10000/15000/m
2. Interface type: Sas/sata/scsi/ide
3. Read and write more sensitive heads

Selection:

1. On-line business, with SAS disk
2. Offline business, with SATA disk, tape library
3. High-concurrency, small-capacity business with SSD disk

Note: Never use SATA disks for data storage or database services with high concurrent service online.

Enterprise production Environment Mainstream disk related information comparison:

Unit capacity vs. Performance and Price: SSD > SAS > SATA >scsi

Unit price Purchase disk capacity: SATA > SAS > SSD

The prevalence of enterprise production scenarios: SAS > SATA > SSD

SSD Solid state drives a hard disk made of solid state electronic memory chip arrays, consisting of a control unit and a memory unit (Flash chip, DRAM chip). SSD in the interface specification and definition, function and use of the same as the ordinary hard disk, in the product shape and size is also fully consistent with the ordinary hard disk.

Solid-State Drive benefits	Advantage Description
Start fast	No click to speed up the process of rotation
Fast write speed	DRAM-based solid-state disk writing speed is extremely fast
Fragmentation does not affect read time	A relatively fixed read time. Disk fragmentation does not affect read time because the addressing time is independent of the data store location
No noise	Due to the absence of mechanical motors and fans, the operating noise value is 0 decibels. Some high-end or large-capacity products are equipped with fans, which can still generate noise
Low calorific value	Low-volume, flash-based solid-state disks consume less energy and heat at work, but high-end or high-capacity products consume more energy
No mechanical failure	There is no mechanical moving parts inside, no mechanical failure, no fear of collisions, shocks, vibrations. This will not affect normal use even with high-speed movement even with rollover tilt, and minimizes the possibility of data loss when the laptop sends accidental drops or hard objects collisions
Greater operating temperature range	Ordinary hard drives can only operate in the range of 5-55 ℃, while most SSDs work in 10~70℃, and some industrial grades can also operate in-40~85℃, even at a greater temperature range.
Small size and light weight	Low-capacity solid-state disks are smaller and lighter than the same-capacity disks. However, this advantage gradually decreases with the increase of capacity, until 256GB, the SSD is still lighter than the same capacity of the ordinary hard disk
Anti-Vibration	Solid-state disks are much more resilient than traditional disks, allowing data to be stored more securely
SSD drawbacks	Defect description
High cost	Per unit capacity price is 5~10 times for traditional disks (based on Flash), even 200~300 times (based on DRAM)
Low capacity	Currently, the maximum capacity of solid state disks is much lower than traditional disks. American company Foremay launched the ec188m series solid-state disk 2TB. The capacity of traditional disks is still growing, and there are now 4TB of
be susceptible to external influences	Because they are not shielded from the Faraday cage like traditional disks, solid-state disks are more susceptible to some external influences. such as power outages (especially for DRAM-based solid-state disks), magnetic interference, electrostatic, etc.
Limited Write Life	The Write lifetime is limited (flash-based), with a typical flash write life of 10,000 to 100,000, specially crafted to 1 million to 5 million, while some parts of the filesystem, such as the file allocation table, will still be written more than this limit during the lifetime of the entire computer. A specially crafted file system or firmware can share the location of the write, so that the overall life of the solid-state disk reaches more than 20 years
Data is hard to recover	Data corruption is difficult to recover, once damaged on the hardware, if it is a traditional disk or tape storage, data recovery may also be able to salvage some of the data, but if it is solid-state storage, once the chip is damaged, want to break into a few pieces or the current breakdown of the chip to retrieve data that is almost impossible, Of course, this deficiency can also be compensated for by the storage space, mainly with raid
Short battery life	Based on actual testing, laptops using solid-state disks run at idle or low load, with battery life shorter than a 2.5-inch HDD using 5400RPM
Higher energy consumption	DRAM-based solid-state disks consume more energy at any time than traditional disks, especially when powered off, or data is lost

Note: In operation and maintenance work should reasonably plan the tiered storage mechanism, according to the business and data usage frequency combined with the use of Ssd+sas (15000 rpm) +sata (10000 rpm).

disk-related terminology : spindle, platter, track, sector, No. 0-disc, head, drive arm, drive shaft, start-stop area, seek, Sector (identifier (head, track, Sector code) + data)

Calculate disk Size Company: Disk size = 512 bytes * Number of Sectors * Number of tracks * Number of heads

Mechanical disk read and write disk Data principle Summary:

1, the disk is based on the cylinder to read and write data, that is, read the same disk surface of a certain track, after reading, if the data is not read, the head will not switch to other tracks, but choose to switch the head, read the next disk surface of the same radius of the track, until all the disk surface of the same radius after the completion If the data has not been read and written, the other different radii of the track will be switched, the process of switching tracks is called Pathfinding.

2, the switch between the different heads of the electronic switch, and the switch between the different tracks need to do radial movement of the head, this radial movement requires stepping motor adjustment, this action when the mechanical switch.

The head seek is a mechanical motion, and the switching head is an electronic switch.

RAID : Inexpensive redundant disk array

The most immediate benefits of doing raid:

1. Improve the security of your data
2. Improve data read and write performance
3. Provides a larger single logical disk data capacity

Different levels of RAID pros and cons comparison:

RAID level	Key Benefits	Key drawbacks	Practical Application Scenarios
RAID 0	Fastest read and write speeds	Without any redundancy	MySQL Slave, cluster of nodes RS
RAID 1	100% Redundancy, mirroring	Read and write performance in general, high cost	Individual, data-critical, and not-downtime business, monitoring, system disk
RAID 5	Have a certain performance and redundancy, can be bad piece of disk, read good performance	Write Performance is low	General business can be used
RAID 10	Fast read/write speed, 100% redundancy	High cost	High performance and redundancy requirements for business, database master and storage master nodes

Note:raid 0 can not be bad, RAID 1 allows bad 50% of the disk, RAID 5 allows bad 1 disk, RAID 6 (dedicated hardware storage) allows bad 2 blocks.

RAID 10 and 01

Disk partitioning

Contents of a 16-byte partitioned table

Number of bytes	Description
1Bytes	State: Partition status, 0 = inactive, 0x80= active
1Bytes	Starthead: Partition starting magnetic Number one
2Bytes	STARTSC: Partition start sector and cylinder number. Low-byte low 6-bit is the sector area code, the height of 2 bits is the 9th, 10 bits of the cylinder number, and the high byte is the low 8 bits of the cylinder number.
1Bytes	Type: partition type, such as 0x0b=fat32,0x83=linux, 00 indicates that this item is not used
1Bytes	Endhead, partition over magnetic number one
2Bytes	ENDSC: Partition end sector and cylinder number, defined as previous
4Bytes	Relative: Partition relative sector address under linear Addressing (absolute address for base partition)
4Bytes	Sectors: Partition size (total number of sectors)

the nature of partitioning: modifies the 64-byte partition table information after the first 446 bytes of the 0 Head 0 Track 1 sector.

Partition tool: Fdisk (suitable for less than 2T disk partition), parted (good at more than 2T disk partition, also can be less than 2T disk partition), preferred Fdisk, more than 2T to choose parted.

Note:

1. The GUID disk partition table (guidpartition table, abbreviation: GPT) is the standard for the structural layout of the partitioned table of an entity hard disk. It is part of the Extensible Firmware Interface (EFI) standard (BIOS used by Intel to replace a personal computer) and is used instead of a sector in the BIOS system to store logical block address and size information for the master Boot Record (MBR) partition table. Because the MBR partition table does not support partitions larger than 2.2TB (2.2x10^12 bytes) for disks that have a 512-byte sector.

2. The FDISK command does not support GPT partitioning, parted supports GPT partitioning methods.

3. The parted operation is real-time.

File system

File system new file operation flow

1, first determine whether the user to add files to the directory has W and x permissions, if any, can be added;

2, according to the inode bitmap find unused inode number, and the new file permissions/attributes written;

3, according to block bitmap found no used block number, and the actual data written to block, and update the inode point to the data storage block;

4. Synchronize the inode with the block data just written to the Inode bitmap and block bitmap, and update the contents of the Superblock.

In normal circumstances, the above-mentioned new operations can certainly be completed successfully, but if the unexpected situation, will produce Superblock, Inode bitmap, block bitmap These intermediate data and the actual data storage area to produce inconsistent situation.

The system reads the data flow:

1. Find File name

2. Find Inode to view access rights

3. Find Block

4, repeated cycle 1-3 steps

format (create file system): A file system is a way for the operating system to explicitly organize files on storage devices (storage of data, hierarchical organization, access and acquisition, and so on).

Format (create file System) Tool:mkfs

View system supported file system formats: ls/lib/modules/' Uname-r '/kernel/fs

Creating a file system generates a certain number of inode and block:

Inode: Property information that holds the file (including not limited to file size, owner (user), genus Group (user group), file permissions, file type, modification time, and the ability to refer to a pointer to a file entity (Inode-block correspondence), etc. But the inode does not contain the file name alone.

Block: Store actual data (image, video, normal file data).

File properties:

Inode Summary:

1. A certain number of inode and block will be generated when the disk is partitioned and formatted as a EXT4 file system

2. The inode becomes an index node, which is used to store the file's attribute information as well as the index of the file (the entity that points to the file).

3, EXT3/4 file system block storage is the actual contents of the file.

4, Inode is a piece of storage space on disk, C6 non-boot area inode default size 256 bytes, C5 is 128 bytes.

5, the Inode performance is a series of numbers, different files corresponding to the Inode (a string of numbers) in the file system is unique.

6, the Inode node number of the same file, each other as a hard-link file, can be considered a different entry of a file.

7, EXT3/4 file system, a file is created to occupy at least one inode and block.

8, EXT3/4 file system, the normal situation of a file occupancy and can only occupy one inode (the relationship between people and identity cards).

9, block is used to store the actual data, the size of each block is generally 1k,2l,4k several. Where the boot partition is 1 k, the other common partitions are more than 4K (C6).

10, if a file is large (high-definition large 4G), need to occupy multiple blocks, if the file is small (0.01K), at least one block, and the block's remaining space is wasted, that can no longer store other data.

11. View the size and total of the inode:

Dumpe2fs/dev/vda1|egrep-i "Block size| Inode size "

Dumpe2fs/dev/vda1|egrep-i "Block count| Inode Count "

12. View the total number of inode and usage

Command Df-i

13. View the Inode information method of the file

Ls-li or Stat/etc/hosts

14. How to generate and specify Inode size

Format command: Mkfs.ext4-b 2048-i 256/dev/sdb

Block Summary:

1. The disk to read data is read by block.

2. A file may occupy more than one block. But every block you read consumes a disk I/O.

3. If you want to improve disk IO performance, try to read as much data as possible at once.

4, a block can only hold the content of one file, no matter how small the content. If the block defaults to 4K size, then storing a 1K file, the remaining 3K will not be able to store other files, can only be wasted.

5, block is not bigger the better. Block too large for storing small files will waste disk space, for example: 1000K file, block size 4K, occupy 250 blocks, if the block default is 1 K, you need to occupy 1000 blocks. Who is more efficient to access? The consumption of Io is 250 and 1000 times respectively.

6, according to business needs, determine the default block size, if it is large file (larger than 16K) general settings block larger, small files (less than 1K) general settings smaller.

7, block is too large, for example, 4 K, files are 0.1K, a lot of wasted disk space, but high access performance.

8, block is too small, such as 1K, files are 1000K, consumes a lot of disk IO.

9, block size setting is also the format of the partition when determined, command mkfs.ext4-b 2048-i 256/dev/sdb

10, the enterprise files will be relatively large (generally more than 4K), block setting larger will improve the efficiency of disk access.

11, EXT3/4 File System (C5/6), generally set to 4K.

Linux Operations Learning Notes-Document System Knowledge System Summary