linuxday12--disk storage and file system

　　

First, Drive introduction

Mechanical HDD (HDD): Hard disk drive, which is the traditional ordinary hard disk, mainly by: platters, head, disc hinge and control motor, head controller, data converter, interface, caching and other components. All the discs in the mechanical hard disk are mounted on a rotating shaft, each disc is parallel, there is a head on the storage surface of each disk, the distance between the head and the disk is smaller than the diameter of the head hair, all the heads are connected to a head controller, and the head controller is responsible for the movement of the heads. The head can move along the radius of the disc, plus the disk rotates at a high speed of thousands of RPM, the head can be positioned at the specified position of the disk for data read and write operations. The data is written to the disk by the magnetic flux through the magnetic head by the electromagnetic flow, or it can be read in the opposite way. Hard drive for precision equipment, air entering the hard drive must be filtered

SSD: Solid state drives, a hard disk made of solid state electronic memory chips, consisting of a control unit and a memory unit (Flash chip, DRAM chip). SSD in the interface specification and definition, function and use of the same as the ordinary hard disk, in the product shape and size is also consistent with the ordinary hard disk

The SSD transfer rate performance is twice times higher than that of HDD,SSD in shock-proof, transmission rate, power, weight, and noise.

has an absolute advantage over price, capacity, and service life compared to SSD,HDD

1. Hard disk storage terminology

Sector sector: the sector on each track on the hard disk is identified with 6bit, with a maximum of 64 sectors per track, 512byte per sector

Track tracks: number of tracks equals number of heads, theoretically 256

Head head: Identifies up to 256 heads with 8bit markings on the hard disk

Cylinder cylinder: 10bit identification on hard disk cylinders, up to 1024 cylinders

Disk capacity calculation method: Sector size *64*256*1024=8589934592=8g

As a result, older hard drives cannot exceed 8G capacity

This way, you can waste a lot of disk space (as with a floppy disk). In order to solve this problem, further increase the capacity of hard disk, people to use the same density structure to produce hard disk. That is, the sector of the outer ring track is more than the inner ring track. With this structure, the hard disk no longer has the actual 3D parameters, and the addressing method is also changed to linear addressing, that is, the sector is addressed in units.

Second, hard disk partition

Disk partitioning uses the partition editor (partition editor) to divide a few logical parts on disk, and once the platters are divided into partitions, different directories and files can be stored in separate partitions. The more partitions there are, the more different places, the more granular the nature of the files can be, and the more segmented nature of the files stored in different places to manage the file, but too many partitions become troublesome. Space management, access licensing, and directory search are based on the file system installed on the partition. When the ability to change size is dependent on the file system installed on the partition, the size of the partition needs to be carefully considered.

1, hard disk for what to partition

1. Optimizing I/O performance
2. Implement disk space quota limits
3. Improved repair speed
4. Isolating systems and programs
5. Install multiple OS
6. Using different file systems

2. partition table type (MBR and GPT) mbr:master boot record master boot records

Also called the Master boot sector, which is the first sector that must be read when the computer accesses the hard disk, the three-dimensional address on the hard disk is (cylinder, head, sector) = (0,0,1). When delving into the internal structure of the main boot sector, it is sometimes referred to as the "Master Boot Record" (MBR), followed by 4 16-byte "Disk partition Table" (DPT), and a 2-byte end flag (55AA) for the initial 446-byte content. Therefore, when using the term "Master boot Record" (MBR), it is necessary to determine whether it refers to the entire primary boot sector or the first 446 bytes of the primary boot sector, depending on the situation.

MBR partition with 4 bytes storage partition total number of sectors, the maximum can represent 2 of the number of 32 sectors, per sector 512 bytes calculation (2^32*512byte=2199023255552byte=2t), so the MBR does not recognize more than 2T after the space

0 Track 0 sector : 512bytes

446bytes:boot Loader

64bytes: Partition table, 16bytes identifies a partition, supports 4 partitions, or 3 primary partitions and one extended partition

2bytes: partition marker bit 55AA

MBR partition structure:

Gpt:globals Unique Identifiers

is the standard for the structural layout of the partitioned table of an entity hard disk. It is part of the Extensible Firmware Interface (EFI) standard (BIOS used by Intel to replace a personal computer) and is used to replace a master boot record (MBR) partition table that stores logical block addresses and size information in a 32bits BIOS system. For disks with a 512-byte sector, the MBR partition table does not support partitions larger than 2.2TB (2.2x1012 bytes)^{[1], however, some hard disk manufacturers (such as Seagate and Western data) Note this limitation and upgrade their larger capacity disks to 4KB sectors. This means that the maximum effective capacity of the MBR is increased to + TiB. This seemingly "right" solution, while temporarily reducing the need to improve disk allocation tables, also gives the market some confusion about how best to partition disk partitions when booting from the BIOS on devices with larger blocks. GPT allocates 64bits to the logical block address, thus making the maximum partition size possible in 264-1 sectors. For each disk with a sector size of 512 bytes, that means there can be 9.4ZB (9.4x1021 bytes) or 8 Zib 512 bytes (9,444,732,965,739,290,426,880 bytes or 18, 446,744,073,709,551,615 (264-1) Sectors x512 (29) bytes per sector)}

GPT supports 128 partitions, using 64-bit representations of sectors, and theoretically supports 8Z disks if each sector is 512Byte.

Use a 128-bit UUID (universally Unique Identifier) to represent disks and partitioned GPT partition tables
Automatic backup of two parts in head and tail with CRC check bit
UEFI (Unified extension Firmware interface) hardware supports GPT, enabling the operating system to boot

GPT Partition structure:

3. Partition Management in Linux

Identification of hard disk and partition tools in Linux

In the early system of the identification of the hard disk interface, such as the IED interface of the hard disk in/dev/to Hd[a-z], the new system is identified by sd[a-z], do not distinguish between the interface

A partition is identified by a number, such as a hard disk is SDA, the first partition of this hard disk is sda1, the second is sda2, and so on.

All files in Linux, the device is identified as a file, the device has the main device number and the secondary device number, the main device number represents different types of devices, the second device number represents the same device of different devices

Tools for managing partitions in Linux

fdisk /dev/disk

-L: List partition information

-C: Using non-cylindrical mode

Interactive:

P: Print the current partition table

M: Help

N: Add a new partition

D: Delete partition

W: Save exit

L: List partition types

T: Modify the partition type

Q Do not save and exit

gdisk: used to manage GPT partitions with similar usage to Fdisk

parted /dev/disk

Mklabel {Gpt|msdos} sets the partition table type of the hard disk and can also enter interactive mode

-L: List partition information

Interactive

partprobe Sync partition table, but bug on Red Hat series 6.x

Partx

-A/DEV/SDA Synchronization partition table (6.x), used when additional partitions appear out of sync

-D--nr m-n/DEV/SDA Synchronous partition table (6.x), used when deleting a partition that is not synchronized

lsblk View the partition table in which memory is now in effect

Third, the file system

A file system is a method and data structure used by the operating system to explicitly store files on a storage device (common disk, also a NAND flash-based SSD) or partition, that is, the method of organizing files on a storage device. The software organization that manages and stores the file information in the operating system is called the file management system, referred to as file system. The file system consists of three parts: the interface of the file system, the software collection, objects and attributes that manipulate and manage the object. From a system perspective, a file system is a system that organizes and distributes the space of a file storage device, which is responsible for storing files and protecting and retrieving the files deposited. Specifically, it is responsible for creating files for users, depositing, reading, modifying, dumping files, control file access, when the user is no longer used when the revocation of files.

Linux file systems: ext2 (Extended file system), ext3, ext4, xfs(SGI), Btrfs (Oracle), ReiserFS, JFS (AIX), Swap
Disc:iso9660
Windows:fat32, ExFAT,NTFS
Unix:ffs (FAST), UFS (Unix), JFS2
Network File system:NFS, CIFS
Clustered file system: GFS2, OCFS2 (Oracle)
Distributed File systems: Fastdfs,ceph, Moosefs, MogileFS, Glusterfs, Lustre
RAW: Unprocessed or unformatted file system

Virtual file system for Linux: VFS

Because the underlying file system type is too many, resulting in program invocation complexity, there is a VFS to act as a unified interface, acting as a middleman

The relationship between Super block and Inode table

1. File System Management Tools

mkfs. {ext4|ext3|xfs| ...} format the specified device partition

-T fstype: Format specified device partition

-L: Set the volume label

-B: Block size

-T: Partition type

-M #: reserved%

-I: Create a node number for the number of bytes

-N: Specify the node number

Findfs label=<label>| Uuid=<uuid>: Finding partitions

blkid: Viewing partition property Information

-U UUID

-L LABEL

fsck: file system Check, Fs_type must be the same as the file type on the partition, the partition must be in the non-mounted state when repairing

Fsck. Fs_type

Fsck-t Fs_type

-P: Auto Fix error

-r: Interactive fix error

-Y: Auto Answer Yes

-F: Forced repair

xfs_info: View information for XFS file systems

2. Create ext File System

MKE2FS: Ext Series File system dedicated management tool

-T{EXT2|EXT3}

-b{1024|2048|4096}

-L LABEL

-j equivalent to-t

-I create an inode for the data space

-n Specifies the inode number in the partition

The size of the referee space that is rapidly occupied by an inode

-M

-O feature Enable feature

blkid: Block device properties Information view

Blkid[option] ... [DEVICE]

-U UUID: Find the corresponding device based on the specified UUID

-L Label: finds the corresponding device according to the specified label

E2label: Manage label for ext series File system

E2label DEVICE [LABEL]

Findfs: Finding partitions

Findfs[options] Label=<label>

Findfs[options] Uuid=<uuid>

tune2fs: Reset the values of the EXT series file system adjustable parameters

-L: View the specified file system super block information; Super Block

-L ' LABEL ': Modify volume label

-M #: Fixed percentage of space reserved for administrators

-j: Upgrade ext2 to Ext3

-O: File System Properties enabled or disabled, –O ^has_journal

-O: Adjust default mount options for the file system, –o ^acl

-U uuid: Modifying the UUID number

dumpe2fs:

Block group management, 32768 blocks

-H: View Super block information, do not display grouping information

Mount for file system labeled "No clean"

Note: Be sure not to fix it in a mounted state

fsck: File System Check

Fsck. Fs_type

Fsck-t Fs_type

-P: Auto Fix error

-r: Interactive fix error

Fs_type must be the same as the file type on the partition

e2fsck: Ext series file-specific detection and repair tool

-Y: Auto Answer Yes

-F: Forced repair

Only one device can be mounted on a folder (only one is valid)

One device can mount multiple folders

Empty Folders act as Mount points

Fedre Ubanti

3. Device Mount Tool

Mount

Usage: Mount device mount point

Label= Volume label name Mount point

Uuid= ' xxxxxx ' mount point

-R Read-only mode mount

-W Read-write mount, default

-N Do not update/etc/mtab

-t specifies the file system, generally not, the system can automatically identify

-A automatically mounts the device in the/etc/fstab

-B | --bind Directory mount to Directory

-O: Options

ACL: Enable ACL

Noacl: Close ACL

Remount: Re-mount

RO: Read-only

Async: Async Mode

Sync: Synchronous mode

Loop: Can mount files, loop device

Atime/noatime: Include Directories and files

Diratime/nodiratime: Access timestamp for directory

Auto/noauto: Whether automatic mount is supported, whether the-a option is supported

Exec/noexec: Whether to support running the application on the file system

Dev/nodev: Whether to support the use of device files on this file system

Suid/nosuid: Whether Suid and Sgid permissions are supported

User/nouser: Whether to allow ordinary users to mount this device,/etc/fstab use

Defaults: Equivalent to RW, suid, dev, exec, auto, nouser, async

umount {Device name | mount point} uninstall device

Fuser

-V/DEV/SDA1 viewing devices who is using

-KM/DEV/SDA1 kill the process using the device

lsof /dev/sda1 View device that process is in use

findmnt Find if a folder is in a mounted state

Losetup -A view the currently mounted loop device

Losetup /dev/loop# loopfile binding file to loop device

4. Memory and Swap management tools

free to view memory and swap information

-m displays in units of M.

-G is displayed in units of

-H is displayed in a way that humans can read

mkswap /dev/sdb1-l swap_sdb1 formatting

Swapon dev| Loopfile activating a swap partition

-S view current swap information

-A activates the swap configured in the Fstab table

Swapoff Disabling a swap partition

5. Other block equipment Operation Tools

eject Eject optical drive

-T bounces into the optical drive

Cp/dev/sr0/data/centos7.iso making the CD into an ISO file

DD If=/dev/sr0 Of=/data/centos7.iso to make the CD into an ISO file

mkisofs -r-o etc.iso/etc/Packages the directory into an ISO file, but does not boot

mkdvdiso.sh Source/destination/dvd.iso making a bootable CD

Wodim –v–eject centos.iso burning Discs

lsusb View UBS

6, DF, Du and DD

DF displays device information for mounting

-H: Displayed in a way that humans can read

-H: Displays the partition size in 1000 units

-T: Display file system

-I: Show node number

-P: Display in standard format

du Display directory size

-H is displayed in a way that humans can read

-S View Directory only

--max-depth # Viewing sub-directory depth

DD convert and copy files

Of=file output

If=file Read-In

Bs=size per byte size

Ibs=size per read-in size

Obs=size each read out size

Cbs=size

Skip=blocks Source skipped

Seek=blocks Target skipped.

count=#

Conv Convert the file with the parameters specified below

ASCII conversion EBCDIC to ASCII

EBCDIC convert ASCII to EBCDIC

LCase converting uppercase characters to lowercase characters

UCase converting lowercase characters to uppercase characters

Nocreat do not create output file

NoError does not stop when an error occurs

Notrunc Not truncated output file

Sync fills each input block into IBS bytes, and the less part is padded with empty (NUL) characters

Copy memory data to hard disk

DD If=/dev/mem Of=/root/mem.bin bs=1024

Copy the in-memory data to the Mem.bin file in the root directory

Copy ISO image from disc

DD If=/dev/cdrom Of=/root/cd.iso

Copy the disc data to the root folder and save it as a Cd.iso file

Destroying disk data

DD If=/dev/urandom OF=/DEV/SDA1

Using random data to populate the hard disk, in some necessary cases can be used to destroy the data, after doing this, the/dev/sda1 will not mount, the creation and copying operations cannot be performed

Test hard Drive Write speed

DD If=/dev/zero of=/root/1gb.file bs=1024 count=1000000

Test Drive Read Speed

DD If=/root/1gb.file bs=64k | DD Of=/dev/null

Repairing the hard drive

DD IF=/DEV/SDA OF=/DEV/SDA

When the hard disk is not used for a long time (for example, 1, 2 years), a degaussing point is generated on the disk. When the heads read these areas, they encounter difficulties and can cause I/O errors. When this condition affects the first sector of the hard disk, it may cause the hard disk to retire. The command above may bring the data back to the dead, and the process is safe and efficient.

Four, RAID disk array

Redundant array of independent hard disks (raid,redundant array of independent Disks), referred to as disk arrays. The basic idea is to combine relatively inexpensive hard drives to become a hard disk in the array group, so that performance reaches even more expensive, large-capacity hard drives. Depending on the version selected, RAID has the benefit of one or more of the following aspects than a single hard drive: Enhanced data integration, enhanced fault tolerance, increased throughput or capacity. In addition, the disk array looks like a separate hard disk or logical storage unit for a computer. Divided into raid-0,raid-1,raid-1e,raid-5,raid-6,raid-7,raid-10,raid-50,raid-60.

1. Standard raidRAID 0

RAID 0 is also known as a stripe set. It makes more than two disks in parallel and becomes a large-capacity disk. When storing data, fragmentation is stored on these disks, since both reads and writes can be processed in parallel, so at all levels, the speed of RAID 0 is the fastest. But RAID 0 is neither redundant nor fault-tolerant, and if one disk (physical) is damaged, all data is lost and the risk level is comparable to JBOD.

RAID 1

More than two groups of n disks mirror each other, in some multi-threaded operating systems can have a good reading speed, the theoretical reading speed is equal to the number of hard disk multiples, the same as RAID 0. In addition, there is a slight decrease in write speed. As long as a disk is normal to maintain operation, the highest reliability. The principle is that the data is stored on the primary hard drive and the same data is written on the mirrored drive. When the primary hard disk (physical) is damaged, the mirrored hard disk replaces the work of the primary hard disk. Data security for RAID 1 is best at all RAID levels because there is a mirrored hard disk for data backup. However, no matter how many disks are used for RAID 1, the capacity of only one disk is the lowest level of disk utilization in all raid.

If you use two different size disks to build RAID 1, the free space for the smaller disk, the larger disk space can also be partitioned into a zone to use, will not cause waste.

RAID 5

RAID Level 5 is a storage solution that combines storage performance, data security, and storage costs. It uses disk Striping (hard disk partitioning) technology. RAID 5 requires at least three hard drives, RAID 5 does not back up the stored data, but instead stores the data and the corresponding parity information on each disk that makes up the RAID5, and the parity information and the corresponding data are stored on separate disks. When one of RAID5 's disk data is damaged, it can recover the corrupted data using the remaining data and the corresponding parity information. RAID 5 can be understood as a compromise between RAID 0 and RAID 1. RAID 5 provides data security for the system, but with a lower level of protection than mirroring and higher disk space utilization than mirroring. RAID 5 has a similar data read speed as RAID 0, just because a bit more parity information, the speed of writing data is relatively slow to write to a single hard disk, if the use of "write-back cache" can improve performance. And because multiple data corresponds to one parity message, RAID 5 has a higher disk space utilization than RAID 1 and is relatively inexpensive to store.

RAID 6

RAID 6 adds a second independent parity information block compared to RAID 5. Two independent parity systems use different algorithms and the reliability of the data is very high, and any two disks fail at the same time without compromising data integrity. RAID 6 requires more disk space to be assigned to parity information and additional checksum calculations, with more IO operations and computations compared to RAID 5, and its "write performance" strongly depends on the implementation, so RAID6 is not typically implemented by software, but is more likely to be implemented in hardware/firmware mode.

A maximum of two disks in the same array are allowed to be corrupted. After the new disk is replaced, the data is recalculated and written to the new disk. According to design theory, RAID 6 must have more than four disks to take effect.

RAID 6 is the most common type of disk array in the functionality of the hardware disk array card.

2.Hybrid RAIDJBOD

JBOD (Just a Bunch of Disks) in the classification, JBOD is not the grade of raid.

RAID 10/01

RAID 10 is the first mirror-and-partition data, then divides all the hard drives into two groups, considered as the lowest combination of RAID 0, and then the two groups respectively as RAID 1 operation.

Raid 01 is the reverse of a RAID 10 program, which partitions and then mirrors the data into two groups of hard disks. It divides all the hard drives into two groups, becoming the lowest combination of RAID 1, while the two groups of hard disks are considered RAID 0 respectively.

When RAID 10 has one hard drive damaged, the remaining drives will continue to function. Raid 01 only needs one hard drive to be damaged, all hard drives in the same group of RAID 0 will stop working, leaving only the other groups ' hard drives running with less reliability. If you build raid 01 with six hard disks, and then use three RAID 0 for the mirror, then the bad one will have three hard drives offline. Therefore, RAID 10 is much more common than raid 01.

3. Modular Tools

Mdadm [Mode] <raiddevice> [options] <component-devices>

Supported RAID levels: LINEAR, RAID0, RAID1, RAID4, RAID5, RAID6, RAID10

Llinux in partition type: FD

Example: Mdadm-c/dev/md#-a yes-l 5-c 32-n 3-x1/dev/sd# using software to create a soft RAID5

-C: Create

-A: Check

-l:raid level

-c:thunk size, default 512K

-N: Hard disk Member

-X: Spare Hard drive

mdadm-d/dev/md# View RAID status

mdadm-ds/dev/md# >/etc/mdadm.conf Generate configuration file, next boot automatically

Mdadm-s/dev/md# Disable RAID devices

Mdadm-a/dev/md# Activate RAID device

mdadm/dev/md#-f/dev/sd# analog HDD Damage

mdadm/dev/md#-r/dev/sd# Manual removal of the hard drive

mdadm/dev/md#-a/dev/sd# Manually add a new hard drive

-D: View, Cat/proc/mdstat can also view raid information

-ds: Generate configuration file, save to/etc/mdadm.conf

-S: Deactivate

-A: Active

-F: Mark hard disk damage

-R: Remove the hard drive

-A: Adding a hard drive

Mdadm-g/dev/md#-n4-a/dev/sd# Extended raid

-G: Extended raid

Mdadm--zero-superblock/dev/sd# Remove the superblock removed from the hard drive to avoid the effects of re-use

V. Logical Volume Manager LVM1, LVM

Logical Volume Manager ( English:Logical Volume Manager, abbreviated to LVM), also translated as logical File Manager, logical Sector Manager, Logical Disk Manager, is the logical volume management provided by the Linux core (Logical Volume Management) function. It creates a logical layer on top of the hard disk partition to facilitate system management of the hard disk partition system.

Partition type in Linux: 8e

PV (physical Volume) Physical volume

VG (Volume Group) volume group

LV (Logical Volume) Logical Partitioning

PE (physical Extent) physical range. The smallest unit in the VG, similar to the chunk size of a raid.

Device Paths in Linux:

/dev/mapper/vg0-lv0

/dev/vg0/lv0

/dev/dm-0

2. LVM Management tools:

PVs: View PV Information

Pvscan: View PV Information

pvdisplay: View PV Information

pvcreate: Creating PV

VGS: View VG Information

vgscan: View VG Information

vgdisplay: View VG Information

vgcreate -S 16M vgname/dev/sd# Create VG

-S: Specify PE size

LVS: view LV Information

lvscan: view LV Information

lvdisplay: view LV Information

lvcreate -n lvname {-L #G |-l #|} vgname Create LV

-N: Specify the name of the LV

-L # (G| M): Specify the LV size

-L #%free | -L # (PE): Specify the LV size

Vgrename Oldvgname newvgname Rename VG

lvrename /dev/vgname/oldlvname/dev/vgname/newlvname renaming LV

3. LVM Snapshot:

lvcreate -n lvname-snapshot-s-L 1g-p r/dev/vgname/lvname Creating a Snapshot

-S: Snapshot feature

-P R: Snapshot read-only property

lvconvert --merge/dev/vgname/lvname-snapshot Recovery Snapshot

lvremove /dev/lvname-snapshot Deleting a snapshot

linuxday12--disk storage and file system