Linux Learning notes: storage Management

All the hardware devices in the Linux system are represented and used in the way of files, we refer to these files as device files, there are a large number of device files in the/dev directory under Linux, and they are divided into character device files and block device files according to different device files.
The access to the character device file is carried out in a character stream, one at a time. Common with printers, terminals (TTY), plotters and tape devices, and so on, character device files are sometimes referred to as "raw" device files.
Block device files are accessed as chunks of data, and the most common device is disk. When the system accesses data through a block device file, it first reads or writes the data from the buffer in memory. Instead of transmitting data directly to the physical disk. This approach effectively improves the I/O performance of the disk.
Here are some of the operations and commands we've learned about storage management under Linux:

Raid:
Ide
Scsi


Features of RAID:
Improved IO capability:
Use multiple disks for parallel reading and writing;
Disk stripe, chunk
Increased Durability:
Rely on disk redundancy for
The data will not be affected even if the disk fails or is damaged;

How RAID is implemented:
Hardware implementation:
A hardware RAID controller or adapter that organizes the required disks into RAID and then installs the OS;
BIOS Program

Software Simulation Implementation
The operating system simulates the raid implementation by means of a system call;

Operation Level of RAID:
RAID0:
Boost IO performance, stripe volume, strip
At least two disks, no redundant fault tolerance, no storage space wasted, requiring each disk to provide the same size of storage space.
RAID1: Mirrored volume, mirror
Provides high availability
Requires two disks
First, the data in the main disk, and then the data into the slave disk;
Low write efficiency, slightly improved read efficiency, disk space overall utilization of 50%, redundant fault tolerance
...
RAID4:
Use multiple disks for XOR operation, get the checksum value, and use a special disk to store the checksum, even if a disk is damaged, the data will not be lost; the test disk IO pressure is huge, it is easy to form a performance bottleneck;
RAID5:
Multiple disks are checked for cyclic redundancy, and officer are randomly assigned to different disk bands; Read and write IO performance is significantly improved without performance bottlenecks
Disk utilization = (n-1)/n * 100%
Redundant fault-tolerant,
Requires at least 3 disks
RAID6:
Multiple disks for two rounds of cyclic redundancy check, officer randomly assigned to two different disk bands;
Significantly improved read/write IO performance without performance bottlenecks
Running up to two disks fails or is still not data streaming
The cost is to increase the time to calculate the calibration value;
RAID7:

RAID Mix level:
RAID01:
Do RAID0 first, then do RAID1
RAID10:
Do RAID1 first, then do RAID0
RAID50:
RAID7: Can be understood as a storage computer, with the corresponding operating system has the appropriate management tools, can run independently.

JBOD: Just a set of disks;
The storage space of multiple disks is connected together, and the data is stored sequentially;

To implement soft raid on CentOS:
The kernel provides a MD module (Multi DISKS,MULTI devices)
User space needs Mdadm tools to set and modify the parameters of the MD kernel module;

Mdadm: Modelling Tools
Mdadm [mode] <raiddevice> [option ...] <component device>
Patterns include:
Create pattern:-C
-N #: Create a RAID device with a # block disk
-L #: Indicates the level of RAID
-A {Yes|no}: Allows the system or does not allow the system to automatically create MD device files;
-C chunk_size: Specify CHUNK size
-X #: Specifies the number of free disks in the array
Assembly Mode:-A
Management mode:-A,-R,-F
Miscellaneous:
-D--scan
Display details of a RAID device
mdadm-d--scan >/etc/mdadm.conf
This configuration file is used for the re-assembly of RAID devices
-S: Device to stop raid

LVM2
Logical Volume Manager, Logical Volume manager, Version 2
Ibm

Use pure software to organize one or more underlying block devices, and redefine them as a solution to a logical fast device;;

Use the DM module in the kernel to achieve:
Dm:device Mapper, Device mapping table
The DM module can organize one or more underlying block devices into a logical block device;
User space in the corresponding command to the DM module issued a system call, you can complete the post-logical fast device management;

Logical block devices are stored uniformly in/dev/dn-#

Steps for LVM Management using the DM mechanism:
1. Create and represent physical volumes, PV
2. Create a volume group based on PV, logical block device, create the volume group while specifying the size of the PE;
Note: Once the PE size is specified, changes are not allowed;
3. Create a logical volume in a volume group that has already been created
4. Creating a file system in a logical volume (Advanced format)
5. Mounting

Management operations for physical volumes:
Pvcreate: Creating a physical volume
Pvdisplay: Displaying details of physical volumes
PVS: Display simple information for physical volumes
Pvremove: Deleting physical volumes
Pvmove: Moves all the PE in a physical volume to another physical volume;

Management operations for volume groups:
Vgcreate: Creating a volume Group
-S #{kkmmgg}: Specify the size of the PE, if this option is omitted, the default PE is 4M;
Vgremove: Deleting a volume group
Vgextend: Expand the volume group capacity and add the new PV to the volume group;
Vgreduce: Reduce the volume group capacity, remove PV from the volume group, before doing this, you should use Pvmove to ensure that the removed PV is not occupied by the PE
Vgdisplay: Displays the details of the volume group
VGS: Display a short message for a volume group


Management operations for logical volumes:
Lvcreate: Creating a logical volume
-L Lv_size

Lvremove: Removing logical volumes
Lvdisplay: Displaying detailed information for a logical volume
LVS: Displaying a brief message for a logical volume
Lvchange: Modifying the status of the LV
-ay: Activating a logical volume
-an: Disabling logical volumes
Lvextend: Extend the space of the logical volume,
Note: Be sure to extend the physical boundaries of the logical volumes before extending the logical boundaries of the logical volumes;
When using the Ext series file system, the RESIZE2FS command expands the logical boundary;
Lvreduce: Reducing the space for logical volumes
Note: The logical boundary of the logical volume is reduced before the physical boundary of the logical volume is reduced;
When using the Ext series file system, the RESIZE2FS command reduces the logical boundary;

For more convenient use of logical volume, for/dev/dm-#设备创建了两个符号链接文件;

/dev/mapper/vg_name-lv_name--- /dm-#
/dev/vg_name/lv_name--- /dm-#


Snapshots of logical volumes:
A snapshot is itself a logical volume, and another access path to the destination logical volume.

A snapshot is a special logical volume that is an exact copy of a logical volume that exists when a snapshot is generated
Snapshots are the most appropriate choice for temporary copies of existing datasets and other operations that require backup or replication.
Snapshots consume space only when they are different from the original logical volume.
A certain amount of space is allocated to a snapshot when it is generated, but only if the original logical volume or snapshot has changed to use the space
When the original logical volume changes, the old data is copied to the snapshot.
The snapshot contains only the changed data from the original logical volume or the data changed in the snapshot since the snapshot was generated
You can also use Lvextend to extend the snapshot volume.

The snapshot is to record the system information at that time, as if there were any data changes in the future, the original data will be moved to the snapshot area, the unchanged area is shared by the snapshot area and the file system.

Since the snapshot area and the original LV share a lot of PE chunks, so the snapshot to be snapshot with the LV must be on the same VG! When the system recovers, the number of files cannot be higher than the actual capacity of the snapshot area.

Snapshot logical Volumes
Lvcreate-l snapshot_size-s-P r-n SNAPSHOT_NAME/PATH/TO/ORIGIN_LVM
-L Size: Specifies the size of the snapshot logical volume
-S: Create a snapshot Logical volume
-P R: Create processed Logical volume is read-only permission
-N snapshot_name: Specifies the name of the snapshot logical volume

Three commands:
DF Command:
Df-report file system disk space usage
DF [OPTION] ... [FILE] ...
-H,--human-readble: unit conversion
-L,--local: Displays only the local file system and does not display the network file system
-I,--inodes: Displays inode usage status
-T,--print-type: Display file system type
-P,--portability: Easier to read with POSIX output format


Du command:
Du-estimate File Space usage
Du [OPTION] ... [FILE] ...
-S, sumary,: Displays the file size for the entire directory rollup
-H, human-readble: unit conversion

DD command: Convert and copy a file
DD [OPERAND] ...
DD OPTION

Common operand:
DD IF=/PATH/FROM/SRC of=/path/to/dest
Bs=bytes:block size, copying unit sizes
Count=n: How many BS do you copy?
Of=file: Writes to the named file instead of to the standard output
If=file: Read from the named file instead of the standard input
Ibs=bytes: Read a size byte at a time
Obs=bytes: Write a size byte at a time
Skip=blocks: Ignores BLOCKS of IBS-sized blocks from the beginning
Seek=blocks: Ignores blocks of BLOCKS obs size from the beginning
Conv=conversion[,conversion ...] : Converts the file with the specified parameters.
Conversion parameters:
ASCII: Convert EBCDIC to ASCII.
EBCDIC: Convert ASCII to EBCDIC.
Block: Converts each row to a cbs-length record, with a blank padding for the insufficient portion.
Unblock: The length of each line is CBS, and the insufficient part is filled with spaces.
LCase: Converts uppercase characters to lowercase characters.
UCase: Converts lowercase characters to uppercase characters.
Swab: Swaps each byte of the input.
NOERROR: Continue reading on read error.
Notrunc: Does not truncate the output file.
Sync: Fills each input block into IBS bytes, with insufficient portions filled with empty (null) characters

Disk Copy:
~]# DD IF=/DEV/SDA OF=/DEV/SDB
Backup MBR:
~]# DD IF=/DEV/SDA Of=/tmp/mbr.bak bs=512 count=1
Destroy the bootloader in the MBR:
~]# DD If=/dev/zero OF=/DEV/SDA bs=64 count=1 seek=446
There are binary files filea,size>2k. Now want to start reading from the 64th byte position, the size to be read is 128Byts. And Fileb, want to read the above 128Bytes write to the beginning of the 32nd byte position, replace 128Bytes, how can I do?
~]# DD If=filea of=fileb bs=1 count=128 skip=63 seek=31 conv=notrunc
Backup:
Back up the local/DEV/SDX full disk to the/DEV/SDY
~]# DD If=/dev/sdx Of=/dev/sdy back up the local/DEV/SDX full disk to/dev/sdy
To back up the/DEV/SDX full data to the image file of the specified path
~]# DD IF=/DEV/SDX of=/path/to/image
Backup/DEV/SDX full data and compress with gzip tool to save to specified path
~]# DD IF=/DEV/SDX | Gzip >/path/to/image.gz

Recovery:
Restore the backup file to the specified disk
~]# DD If=/path/to/image OF=/DEV/SDX
Restores a compressed backup file to a specified disk
~]# gzip-dc/path/to/image.gz | DD OF=/DEV/SDX

Copy memory data to hard disk
Copy the in-memory data to the Mem.bin file in the root directory
~]# DD If=/dev/mem Of=/root/mem.bin bs=1024

Copy ISO image from disc
Copy the disc data to the root folder and save it as a Cd.iso file
~]# DD If=/dev/cdrom Of=/root/cd.iso

Destroying disk data
The use of random data to populate the hard disk, in some necessary occasions can be used to destroy data. After you do this, the/dev/sda1 cannot be mounted, and the Create and copy operations cannot be performed.
~]# DD If=/dev/urandom of=/dev/sda1

Get the most appropriate block size
The best block size for the system can be determined by comparing the command execution time shown in the output of the DD instruction.
~]# DD If=/dev/zero bs=1024 count=1000000 of=/root/1gb.file
~]# DD If=/dev/zero bs=2048 count=500000 of=/root/1gb.file
~]# DD If=/dev/zero bs=4096 count=250000 of=/root/1gb.file
~]# DD If=/dev/zero bs=8192 count=125000 of=/root/1gb.file

Test drive Read and write speed
The read/write speed of the test drive can be calculated by the execution time of the last two command outputs.
~]# DD If=/root/1gb.file bs=64k | DD Of=/dev/null
~]# DD If=/dev/zero of=/root/1gb.file bs=1024 count=1000000

Repairing the hard drive
When the hard disk is not used for a long time (for example, 1, 2 years), a degaussing point is generated on the disk. When the heads read these areas, they encounter difficulties and can cause I/O errors. When this condition affects the first sector of the hard disk, it may cause the hard disk to retire. The above command may bring the data back to the dead. And the process is safe and efficient.
~]# DD IF=/DEV/SDA OF=/DEV/SDA

Linux Learning notes: storage Management