Multi-path and large-capacity hard drive mounting in CentOS

I. application environment and requirements blade servers connect to HP storage through optical fiber switches, forming a 2x2 link. The operating system is CentOS6.464-bit mounted with a storage capacity of TB. Based on this application environment, you need to solve two problems: to ensure link stability and transmission performance, you can use multi-path technology; the attached storage hard disk exceeds 2 TB, and the MBR partition format is not supported. you need to use GPT partition 1, application environment and requirements.

1. The blade server connects to HP storage through an optical fiber switch, forming a 2x2 link.
2. CentOS 6.4 64-bit operating system
3. The attached storage capacity is 2.5 TB.

Based on this application environment, two problems need to be solved:

1. Multi-path technology can be used to ensure link stability and transmission performance;
2. The size of the attached storage disk exceeds 2 TB, and the MBR partition format is not supported. you need to use the GPT partition format.

Because CentOS 6.4 already comes with an HP storage driver, it will automatically identify the mounted storage hard disk. Otherwise, you needFirst install the storage driver.

2. what is multi-path?

From: http://rootking.blog.51cto.com/2619611/476212

 

A common computer host is attached to a bus by a hard disk. this is a one-to-one relationship. In a SAN environment composed of optical fibers or an IPSAN environment composed of iSCSI, the host and storage are connected through an optical fiber switch or multiple network adapters and IP addresses, it constitutes a many-to-many relationship. That is to say, you can select multiple paths from the host to the storage. IO between the host and storage can be selected from multiple paths. How many different paths can each host go through to the corresponding storage? if they are used at the same time, how can I/O traffic be allocated? What should I do if one of the paths breaks down? From the operating system perspective, the operating system considers each path as a physical disk, but it actually only leads to a different path to the same physical disk, in this way, the user is confused. The multi-path software came into being to solve the above problems.

The main function of multi-path is to work with the storage device to implement the following functions:

1. failover and recovery

2. load balancing of IO traffic

3. disk virtualization

Because the multi-path software needs to be used together with the storage, different vendors provide different versions based on different operating systems. In addition, some vendors, software and hardware are not sold together. if you want to use multi-path software, you may need to purchase license from the vendor. For example, if EMC is based on multi-path software in linux, it needs to separately purchase license. Fortunately, the 2.6 kernel of RedHat and Suse comes with free multi-path software packages and can be used for free. it is also a common package that supports devices of most storage vendors, even some vendors that are not famous can support and run well by slightly modifying the configuration file.

The intuitive feeling is to executefdisk -lCommand to display hard disks like/dev/sda1,/dev/sdb1,/dev/sdc1,/dev/sdd1. Because there are a total of four combined paths, the Linux system considers each hop link as a hard disk attached.

III. Introduction to multipath in Linux

In CentOS 6.4, multipath is installed by default:

[root@localhost ~]# rpm -qa|grep mapperdevice-mapper-multipath-0.4.9-64.el6.x86_64device-mapper-event-libs-1.02.77-9.el6.x86_64device-mapper-multipath-libs-0.4.9-64.el6.x86_64device-mapper-persistent-data-0.1.4-1.el6.x86_64device-mapper-libs-1.02.77-9.el6.x86_64device-mapper-event-1.02.77-9.el6.x86_64device-mapper-1.02.77-9.el6.x86_64

Where (from: http://rootking.blog.51cto.com/2619611/476212 ):

1. Device-mapper-multipath: multipath-tools. It mainly provides tools such as multipathd and multipath, and configuration files such as multipath. conf. These tools use the ioctr interface of device mapper to create and configure the multipath device (call the user space library of device-mapper, and the created multi-path device will be in/dev/mapper ).

2. Device-mapper: mainly includes two parts: kernel and user.

   The kernel is mainly composed of the device mapper core (dm. ko) and some target driver (md-multipath.ko ). The core completes device ING, while the target processes the I/o from the mappered device based on the ing relationship and its own characteristics. At the same time, an interface is provided in the core section. you can use ioctr to communicate with the kernel section to guide kernel driver behavior, such as how to create mappered device and the divece attributes. The user space of linux device mapper mainly includes the device-mapper package. This includes the dmsetup tool and some libraries that help you create and configure mappered devices. These libraries are mainly abstracted and encapsulated interfaces for communication with ioctr to facilitate the creation and configuration of mappered devices. These libraries must be called in the multipath-tool program.

3. Dm-multipath.ko and dm. ko: dm. ko are device mapper drivers. It is the basis for multipath implementation. Dm-multipath is actually a target driver of dm.

4. Scsi_id: Included in the udev package. you can configure this program in multipath. conf to obtain the serial number of the scsi device. By serial number, you can determine that multiple paths correspond to the same device. This is the key to multi-path implementation. Scsi_id is an inquery command that sends EVPD page80 or page83 to the device through the sg driver to query the ID of the scsi device. However, some devices do not support the EVPD inquery command, so they cannot be used to generate multipath devices. However, scsi_id can be rewritten to provide a virtual identifier for the device that cannot provide the scsi device identifier and output it to the standard output. When creating a multipath device, the multipath program calls scsi_id to obtain the scsi id of the device from its standard output. When rewriting, you need to modify the scsi_id program's return value to 0. In the multipath program, the system checks the result to determine whether the scsi id has been successfully obtained.

4. configure multipath

The basic configuration script is as follows:

[root@localhost ~]# cat /etc/multipath.confdefaults {      polling_interval       30      failback               immediate      no_path_retry          queue      rr_min_io              100      path_checker           tur      user_friendly_names    yes      }      # SVC            device {                 vendor                  "IBM"                 product                 "2145"                 path_grouping_policy    group_by_prio                 prio_callout            "/sbin/mpath_prio_alua /dev/%n"                                }

Multipath Basic Operation Commands

#/Etc/init. d/multipathd start # enable the mulitipath service # multipath-F # Delete an existing path # multipath-v2 # format path # multipath-ll # View multiple paths

If the configuration is correct, more devices, such as mpathbp1, will be added to the/dev/mapper/directory.fdisk -lCommand to view the disk created by the multi-path software, such as:/dev/mapper/mpathbp1

5. format the hard disk

Runfdisk -lYou can see that the storage has been identified and the multi-path configuration is correct. The information is as follows:

[root@localhost ~]# fdisk -l......Disk /dev/mapper/mpathb: 2684.4 GB, 2684354560000 bytes255 heads, 63 sectors/track, 326354 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x00000000             Device Boot      Start         End      Blocks   Id  System/dev/mapper/mpathbp1               1      267350  2147483647+  ee  GPT......

The above information shows that the partition format of GPT is used, and the hard disk needs to be formatted. If not, perform the following steps:

1. create a partition

[Root @ localhost ~] # Pvcreate/dev/mapper/mpathb [root @ localhost ~] # Parted/dev/mapper/mpathbGNU Parted 2.1 Using/dev/mapper/mpathbp1Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) mklabel gpt # set the partition type to gpt (parted) mkpart extended 0% 100% # extend the partition and use the entire hard disk (parted) quit # exit Information: you may need to update/etc/fstab.

2. Format and mount the hard disk

[root@localhost ~]# mkfs.ext4 /dev/mapper/mpathbp1[root@localhost ~]# mount /dev/mapper/mpathbp1 /test

After the mounting is successful, you can use it.

3. dynamic Mount partition

After a new partition is created in the system, because mount fails after the system is restarted, you need to write the partition information to the/etc/fstab file for permanent mounting.

[root@localhost ~]# vi /etc/fstab/dev/mapper/mpathbp1    /test                ext4    defaults        1 2

Save and exit. after restart,/dev/mapper/mpathbp1 will be automatically mounted to the/test directory.