One, what is MultipathThe normal computer host is a hard disk attached to a bus, here is a one-to-one relationship. In the case of a SAN environment with optical fiber, or an ipsan environment made up of iSCSI, the host and storage are connected by a fibre switch or multiple network cards and IP, which makes the relationship much more numerous. In other words, host-to-storage can have multiple paths to choose from. The IO between host-to-storage is selectable by multiple paths. Each host to the corresponding storage can go through a few different paths, if it is used at the same time, how I/O traffic allocation? One of the paths broke down, how to deal with it? Also in the operating system perspective, each path, the operating system will be considered to be an actual physical disk, but in fact only to the same physical disk of the different paths, which is in use, the user is confused. Multi-path software is to solve the above problems arose. The main function of multipath is to work with storage devices to achieve the following functions:
1. Failover and recovery of faults
Load Balancing of 2.IO traffic
3. Virtualization of Disks
Because Multipath software is needed and stored together with the use of different vendors based on different operating systems, the different versions are available. And some vendors, software and hardware are not sold together, if you want to use multi-path software, you may also need to buy license to the manufacturer. For example, EMC's multi-path software under Linux requires a separate purchase of license. Fortunately The 2.6 kernel of Redhat and SuSE comes with free multipathing packages and is free to use, as well as a more generic package that supports most storage vendors ' devices, even if some are not famous, and can be supported and run nicely by making minor modifications to the configuration files.
second, Linux under Multipath Introduction, need the following toolkit:In CentOS 5, the minimum installation of the system Multipath has been installed to see if Multipath is installed as follows:
1
, Device-mapper-multipath:That is multipath-tools. Mainly provide MULTIPATHD and multipath tools and multipath.conf and other configuration files. These tools create and configure multipath devices through the Ioctr interface of the device mapper (call the Device-mapper User space Library. The multipath device that is created is in/dev/mapper).
2
, Device-mapper:It mainly consists of two parts: kernel part and user part. The kernel is mainly composed of the device mapper core (Dm.ko) and some target driver (Md-multipath.ko). The core completes the mapping of the device, and target handles the I/O from the mappered device, based on the mapping relationship and its own characteristics. At the same time, in the core section, provides an interface, the user through the ioctr can communicate with the kernel, to guide the kernel-driven behavior, such as how to create mappered device, these divece properties and so on. The user space portion of the Linux device Mapper mainly includes the Device-mapper package. These include the Dmsetup tool and some libraries that help you create and configure mappered device. These libraries are primarily abstract and encapsulate interfaces to ioctr communication to facilitate the creation and configuration of mappered device. These libraries need to be called in the Multipath-tool program.
3
, Dm-multipath.ko and Dm.ko: Dm.ko is a device mapper driver. It is the foundation of realizing Multipath. Dm-multipath is actually a target driver for DM.
4
, scsi_id：included in the Udev package, you can configure the program in multipath.conf to get the serial number of the SCSI device. By ordinal, you can tell that multiple paths correspond to the same device。 This is the key to multi-path implementation. SCSI_ID is to query the identity of the SCSI device by sending the inquery command EVPD page80 or page83 to the device via the SG driver. However, some devices do not support EVPD's inquery commands, so they cannot be used to generate multipath devices. However, scsi_id can be rewritten to provide a virtual identifier for a device that cannot supply a SCSI device identity and output to standard output. When the Multipath program creates the Multipath device, it calls SCSI_ID and obtains the SCSI ID of the device from its standard output. When overwriting, the return value of the SCSI_ID program needs to be modified to 0. Because in the Multipath program, the straight is checked to determine if the SCSI ID has been successfully obtained./sbin/scsi_id-g-u-s/BLOCK/SDA back to SCSI Wwid
third, the basic configuration process of multipath in CentOS 5:
1
, installing and loading multipathing packages
# yum–y Install device-mapper device-mapper-multipath# chkconfig–level 2345 multipathd on #设置成开机自启动multipathd # lsmod |GR EP Dm_multipath #来检查安装是否正常
If the module does not load successfully use the following to initialize the DM, or restart the system
---Use the following commands to initialize and start DM for the first time:
# modprobe Dm-multipath
# modprobe Dm-round-robin
# service MULTIPATHD Start
# MULTIPATH–V2
2
, configure multipath:Multipath configuration file is/etc/multipath.conf, if you need to multipath normal operation only need the following configuration: (If you need more detailed configuration, see the following introduction of this article) blacklist {devnode "^SDA"} Defaults {user_friendly_names yespath_grouping_policy multibusfailback immediateno_path_retry fail}# vi/etc/ Multipath.conf 3,
multipath Basic Operations Command
#/ETC/INIT.D/MULTIPATHD Start #开启mulitipath服务
# multipath-f #删除现有路径
# Multipath-v2 #格式化路径
# multipath-ll #查看多路径如果配置正确的话就会在/dev/mapper/Mpath0, mpath1 and other devices. Use the Fdisk-l command to see the disks created by the Multipath software, as in/dev/dm-[0-3] 4,
basic operation of multipath disk
To operate a disk that is generated by Multipath software, directly manipulate the disk in the/dev/mapper/directory. It's a good idea to run the pvcreate command before partitioning the Multipath software-generated disk: # pvcreate/dev/mapper/mpath0# fdisk/ Dev/mapper/mpath0 using FDISK to partition the disk generated by the Multipath software will be an error, this error is ignored. When Fdisk partitions the disk generated by the Multipath software, the resulting disk partition is not immediately added to the/dev/directory, and we will restart the driver for Ipsan or Fcsan. If you are using Iscsi-initiator to connect Ipsan's restart iSCSI service, you can see that the resulting disk partition has a # service iSCSI restart# Ls-l/dev/mapper/ such as MPATH0P1 and MPATH1P1 are our partitions on the multipath disk # mkfs.ext3/dev/mapper/mpath0p1 #对mpath1p1分区格式化成ext3文件系统 # mount/dev/ mapper/mpath0p1/ipsan/#挂载mpath1p1分区
iv. High configuration of MultipathThe above is done with the multipath default configuration to complete the multipath configuration, such as the name of the mapping device, Multipath load Balancing method is the default settings. Is there a way to configure multipath according to our own definition?
1
, multipath.conf
Configuration of FilesThe next task is to edit the/etc/multipath.conf configuration file multipath.conf mainly includes the configuration of the Blacklist, multipaths, and devices three parts
blacklist
ConfigurationBlacklist {devnode "^SDA"}
Multipaths
Partial ConfigurationConfiguration of the multipaths and devices two parts. multipaths {multipath {wwid **************** #此值multipath-v3 can see alias Iscsi-dm0 #映射后的别名, can take Path_grouping_policy Multibus #路径组策略path_checker tur #决定路径状态的方法path_selector "Round-robin 0" #选择那条路径进行下一个IO操作的方法}}
Devices
Partial ConfigurationDevices {device {vendor ' iscsi-enterprise ' #厂商名称product ' Virtual disk ' #产品型号path_grouping_policy Multibus # The default path Group Policy getuid_callout "/sbin/scsi_id-g-u-s/block/%n" #获得唯一设备号使用的默认程序prio_callout "/sbin/acs_prio_alua%d" #获取有限级 The value uses the default program Path_checker readsector0 #决定路径状态的方法path_selector "Round-robin 0" #选择那条路径进行下一个IO操作的方法failback immediate #故障 The restored mode no_path_retry queue #在disable queue before the system attempts to use the number of failed paths Rr_min_io #在当前的用户组中, the number of IO requests before switching to another path}}
The following is a complete configuration fileblacklist {devnode "^SDA"}defaults {user_friendly_names no}multipaths {multipath {wwid 14945540000000000a67854c6270b4359c66c272e2f356321alias iscsi-dm0path_grouping_policy multibuspath_checker turpath_selector "Round-robin 0"}multipath {wwid 14945540000000000dcca2eda91d70b81edbcfce2357f99eealias iscsi-dm1path_grouping_policy multibuspath_ checker turpath_selector "Round-robin 0"}multipath {wwid 1494554000000000020f763489c165561101813333957ed96alias iscsi-dm2path_grouping_policy multibuspath_ checker turpath_selector "Round-robin 0"}multipath {wwid 14945540000000000919ca813020a195422ba3663e1f03cc3alias iscsi-dm3path_grouping_policy multibuspath_ checker turpath_selector "round-robin 0"}}devices {device {vendor " Iscsi-enterprise "product " Virtual disk "path_grouping_policy multibusgetuid_callout "/sbin/scsi_id -g -u -s /block/%n" path_checker readsector0path_selector " Round-robin 0 "}}
ways to get Wwid:(1) By default, each Multipath device name will be set using the configuration in/var/lib/multipath/bindings, and if each Wwid alias is set in/etc/multipath.conf, the alias will override this setting. (2) # MULTIPATH-V3 command Lookup
Test steps
1. Deploy a test environment where two Emulex light cards on the host are connected to the fibre switch and then connected to EMC CX storage via a fibre switch to form a San topology. and partition 1 100G LUNs to the host.
9 5.8 SUSE9 9 SP1 5.9.2 9 SP3 5.9.4 9 SP4 6.1
2. Host install the operating system (pre-test preparation), install the latest version of the Device-mapper-multipath package. Verify multi-path load balancing and path failover with the Device-mapper-multipath user tool: Use the FDISK command to see the 4 disk devices identified by the system, which are the device names of multiple paths, actually pointing to the same LUN on the storage, This indicates that the red flag operating system has correctly identified the LUNs partitioned by EMC CX storage and is preparing for the next multi-path management. The command and output are as follows:
#fdisk-L
disk/dev/sdf:103 GB, 107374182400 bytes heads, sectors/track, 102400 cylinders
Units = Cylinders of 2048 0* = 107374182400 bytes
Disk/dev/sdf doesn ' t contain a valid partition table
disk/dev/sdh:103 GB, 107374182400 bytes
Heads, Sectors/track, 102400 cylinders
Units = Cylinders of 2048 0* = 107374182400 bytes
DISK/DEV/SDH doesn ' t contain a valid partition table
disk/dev/sdj:103 GB, 107374182400 bytes heads, sectors/track, 102400 cylinders
Units = Cylinders of 2048 0* = 107374182400 bytes
Disk/dev/sdj doesn ' t contain a valid partition table
disk/dev/sdl:103 GB, 107374182400 bytes heads, sectors/track, 102400 cylinders
Units = Cylinders of 2048 0* = 107374182400 bytes
DISK/DEV/SDL doesn ' t contain a valid partition table
In fact, these 4 devices correspond to a LUN, just seen through a different path.
3. Start multi-path management software
# modprobe dm-multipath (load dm-multipath kernel module)
Description: This module is not loaded by default at system startup. If application deployment is required, it can be customized at system startup. #/ETC/INIT.D/MULTIPATHD Start (Start multipath daemon service) # MULTIPATH–V3 (Assembly multipath Device)
# MULTIPATH-LL (shows the current multipath topology) 3600601604b991100f4e5b5c83ef5da11
[Size=100 GB] [features= "1 Queue_if_no_path"] [hwhandler= "1 EMC"]
\_ Round-robin 0 [Active]
\_ 1:0:2:1 SDF 8:80 [Ready][active]
\_ 2:0:1:1 SDL 8:176 [Ready][active]
\_ round-robin 0 [enabled]
\_ 1:0:3:1 SDH 8:112 [Ready][active]
\_2:0:0:1 sdj 8:144 [Ready][active]
The devices here are divided into two groups, which are actually the devices seen through two controllers, one of which is [active], which indicates that this is the current active controller. The subsequent reading and writing of the device will be performed by the/DEV/SDF and/DEV/SDL under the controller. The device/DEV/SDH,/DEV/SDJ under the controller currently in the [enabled] state is enabled only if the [active] controller fails or if Tresspass is executed.
