SCSI lock details

What is a SCSI lock?

In a shared storage environment, multiple hosts may access the same storage device at the same time. If multiple hosts Write A Lun at the same time, then we can imagine that this Lun will not know which data to write first, which data to Write later. To prevent data corruption caused by this situation, the concept of SCSI lock is introduced. For example, when HostA reads and writes a Lun, it adds a SCSI lock to the Lun. In this case, HostB cannot access the Lun.

HostA HostB

\/

\/

Lun

1. SCSI lock type.

Usually there are two types of SCSI locks: SCSI-2Reservation and SCSI-3 Reservation, where SCSI-3Reservation is also called Persistent Reservation. The two types of locks cannot exist on one Lun.

The SCSI-2 Reservation only allows the device to be accessed by a locked Initiator, which generally refers to the HBA. For example, if fcs0 on HostA adds a SCSI-2 lock to the accessed LUN, then fcs1 on HostA cannot access the Lun. So SCSI-2 Reservation is sometimes called single-pathreservation.

SCSI-3 Reservation (PersistentReservation) is to use the PR Key to lock the device. Generally, a Host has a unique PR Key. Different hosts have different prkeys. Therefore, the general SCSI-3 Reservation is usually applied under the multi-path sharing environment.

2. Under what circumstances will the device be locked?

The device is usually locked when it is turned on. For example, varyonvg and dd, note that the device will be locked when the command dd is run and automatically unlocked after the command is run.

Note: varyonvg-c does not lock the device.

In addition, after vg varyon, only varyoffvg or varyonvg-B can unlock the vg-related devices. Directly Using the shutdown command does not perform the varyoffvg action, so it does not unlock.

3. Deal with SCSI-3 lock

1) view the sense data of SC _DISK_ERR * orFSCSI_ERR.

01-thisindicates the SCSI status field is valid

18-SCSI device is reserved by anotherHost

For example:

SENSE DATA

0600 0000 0000 0000 0000 0000 0000 00000000 0000 0000 0000 0118 0000 0000 0000

This is usually seen in an SC _DISK_ERR * orFSCSI_ERR * error (errpt-a output ).

2) process persistent reservation on ESS/DS8000/DS6000.

You can view persistentreservation in lquerypr-vl/dev/vpathX. If PR exists, the returned value is the PR Key, which can be compared with uname-.

Lquerypr-ch/dev/vpathX can be used to clear persistentreservation. Note: Use this command with Caution !!!

Use pcmquerypr in the sddpcm environment.

3) process persistent reservation on DS4000.

View and clear the SCSI-3reservation of the logical drive through Advanced-> Maintenance-> PersistentReservations in SM.

4. Deal with SCSI-2 lock

Sometimes we can't see the lock on the vpath through lquerypr, or the SM persistentreservation output also shows no lock, but hdisk/vpath cannot be accessed. Check if there is a SCSI-2reservation.

Note: The following command is not limited to SCSI-2 locks, and also applies to SCSI-3 locks.

1) DS4000: hlmTestLunShowSSIDView.

HlmTestLunShow 8

LunNumber: 0x8 LunInfo: 0x6642c5c State: 0x0

QuiescenceFlag: 0x0 Owner: 0x1 IsReady: 0x1

ReserveId: 0xe resv3rdId: 0 xffff

Value = 128 = 0x80

In the output, reseveId represents the SCSI-2 reservation, and resv3rdId represents the SCSI-3 reservation. If the value is 0xffff, there is no lock. If the value is 0xe, it is occupied by the host whose hostid is 0xe.

Unlock method:

Ü hlmTestReleaseReservId, SSID

Ü switch the Lun from one controller to another Controller

Ü on AIX, run the command/usr/sbin/cluster/events/utils/cl_flutereset/dev/hdiskXX of HACMP.

2) view DS6800: catreef "fb/volstatuslss"

Catreef "fb/volstatus 0x12"

Vol clr da State FB Status Known Format Status

-------------------------------------------------------------------------

1200 pprc good Ready formatted

1201 trad good Ready formatted

1202 pr good Ready formatted

1203 None GOOD Ready formatted

PPRC Means PPRC (suggest its probably a PPRC target)

TRAD Traditional sci2 reserve

PR SCSI3 Persistent reserve.

Unlock method:

Ü cmt-aRESET_LUN_RESERVATION-t volume

For example, cmt-aRESET_LUN_RESERVATION-t 0x1202

3) DS8000: cat "/dev/cpss0/fb/volstatusLss"View

Cat "/dev/cpss0/fb/volstatus0x98"

Vol clr da State FB Status Known Format Status

-----------------------------------------------------------------

981D pr good Ready formatted

Unlock method:

Ü cmt-aRESET_LUN_RESERVATION-tVolume

For example, cmt-aRESET_LUN_RESERVATION-t 0x981D

5. reserve_policy

Each Vendor's device and driver have their own properties, but most of them are similar. Here we use the reserve_policy of MPIO as an example:

No Reserve reservation policy

If you set MPIO devices with this reservepolicy, there is no reserve being made on MPIO devices. A device withoutreservation can be accessed by any initiators at any time. input/output can besent from all the paths of the MPIO device. this is the default reserve parameter yof SDDPCM. (Please pay attention to this)

Exclusive Host Access single-pathreservation policy

This is the scsi-2 reservation policy. ifyou set this reserve policy for MPIO devices, only the fail_over path selectionalgorithm can be selected for the devices. with this reservation policy, anMPIO device only has one path being opened, and a scsi-2 reservation is made bythis path on the device. input/output can only be sent through this path. whenthis path is broken, another path will be opened and scsi-2 reservation will bemade by the new path. all input and output will be routed to this path.

Persistent Reserve Exclusive Host Accessreservation policy

If you set an MPIO device with thispersistent reserve policy, a persistent reservation is made on this device witha persistent reserve (PR) key. any initiators who register with the same PR keycan access this device. normally, you shoshould pick a unique PR key for a server. different servers shoshould have different unique PR key. input and output isrouted to all paths of the MPIO device, because all paths of an MPIO device areregistered with the same PR key. in a nonconcurrent clustering environment, such as HACMP, this is the reserve policy that you should select.

Current HACMP clustering software supportsno_reserve policy with Enhanced Concurrent Mode volume group. HACMP support forpersistent reserve policies for supported storage MPIO devices is notavailable.

Persistent Reserve Shared Host Accessreservation policy

If you set an MPIO device with thispersistent reserve policy, a persistent reservation is made on this device witha persistent reserve (PR) key. however, any initiators that implementedpersistent registration can access this MPIO device, even if the initiators areregistered with different PR keys. in a concurrent clustering environment, suchas HACMP, this is the reserve policy that you should select for sharingresources among multiple servers.

Current HACMP clustering software supportsno_reserve policy with Enhanced Concurrent Mode volume group. HACMP support forpersistent reserve policies for supported storage MPIO devices is notavailable.

6. commands related to locks on AIX

1) varyonvg/varyoffvg

Varyonvg locks related hdisk, vpath, and other devices. In general, DS4000 hdisk, DS6000/DS8000/ESS vpath device will be added with a SCSI-3 lock.

However, early DS4000 microcodes (SM8.4 or earlier, unconfirmed) Use SCSI-2 locks.

For DS6000/DS8000/ESS, use the SCSI-2 lock if vpath is not used.

Of course, you can also specify the lock by modifying attributes such as dpo, hdisk, and vpath.

Varyoffvg can unlock devices related to VG.

Varyonvg-B will also unlock VG-related devices. Generally, this command is run on the host where VG is being used. It is used with the "-u" parameter and can be used for LVM operations in the HA environment.

Note: The "-B" parameter calls SC _FORCED_OPEN to open the hdisk lock in VG. However, for both SCSI and FC devices, this command unlocks all Luns on the targetaddress of the hdisk. For example, if hdisk0 and hdisk1 are both in fcs0, hdisk0 belongs to datavg, and hdisk1 belongs to testvg, varyonvg-B datavg is used, and hdisk0 and hdisk1 are unlocked.

In addition, you can use varyonvg-B to unlock the VG on an AIX host that shares the VG but does not have varyon in certain environments. For example, if HostA and HostB share hdisk0 and hdisk0 constitute a datavg, The datavg is currently varyon on HostA. In some specific environments, running "varyonvg-bdatavg" on HostB can remove the lock on hdisk0. Note: The varyonvg design is not used in this environment. Unpredictable situations may occur. Please use it with caution! One way to try this method is that HostA shares a non-IBM storage with HostB. This storage does not have its own tools to unlock and is not supported by HACMP. The datavg consisting of hdisk0 is varyon on the HostA. In this case, the HostA is abnormal and the HostB cannot take over normally. Therefore, the hdisk0 lock cannot be released. At this time, you can try to use varyonvg-B datavg on HostB to unlock it, but it may not be successful (depending on the support of the storage vendor ).

All in all, when a device is not used, please shut down after varyoffvg is normal. Use varyonvg-B with caution to unlock.

2) commands related to HACMP

Normally, during the HACMP switchover, the/usr/es/sbin/cluster/events/utils/cl_disk_available script is called to determine the device type and whether the device has a lock, then, call the relevant commands to unlock the instance.

/Usr/es/sbin/cluster/events/utils:

Cl_flutereset (for DS4000)

Cl_fscsilunreset (for SCSI-3)

Cl_iscsilunreset (for iSCSI)

Cl_pscsilunreset (for SCSI-2)

Cl_scdiskreset (for IBM 7135)

Cl_vpathreset (for sdd)

Note: The use of these commands alone is not necessarily applicable to all storage, and the use of these commands alone is not officially supported by IBM.

3) lquerypr/pcmquerypr/pcmgenprkey

These commands are described in detail in Multipath SubsystemDevice Driver User's Guide. In this case.

7. A situation to be aware of (Excerpted from MultipathSubsystem Device Driver User's Guide)

Understanding the persistent reserve issuewhen migrating from SDD to non-SDD volume groups after a system reboot

There is an issue with migrating from SDDto non-SDD volume groups after a system reboot. this issue only occurs if theSDD volume group was varied on prior to the system reboot and auto varyon wasnot set when the volume group was created. after the system reboot, the volumegroup will not be varied on.

The command to migrate from SDD to non-SDDvolume group (vp2hd) will succeed, but a subsequent command to vary on thevolume group will fail. this is because during the reboot, the persistentreserve on the physical volume of the volume group was not released, so whenyou vary on the volume group, the command will do a SCSI-2 reserve and failwith a reservation conflict.

There are two ways to avoid this issue.

1. Unmount the filesystems and vary offthe volume groups before rebooting the system.

2. execute lquerypr-vl/dev/vpathX on thephysical LUN before varying on volume groups after the system reboot. if theLUN is reserved by the current host, release the reserve by executing lquerypr-Vrh/dev/vpathX command. after successful execution, you will be able to varyon the volume group successfully.

To sum up, there are two reasons for this problem:

1. AIX to hdisk consists of VG varyon, hdisk plus SCSI-2 reservation; vpath consists of VG varyon, vpaht plus SCSI-3 reservation.

2. In the case of VG Varyon, direct shutdown of AIX will not be unlocked.

This is a typical issue.

Appendix:

1. varyonvg-B parameter description:

Breaks disk reservations on disks lockedas a result of a normal varyonvg command. Use this flag on a volume group thatis already varied on.

Notes:

L This flag unlocks all disks in a givenvolume group.

L The-B flag opens the disks in the volumegroup using SC _FORCED_OPEN flag. for SCSI and FC disks this forces open allluns on the target address that this disk resides on. volume Groups shouldtherefore not share target addresses when using this varyon option.

L The-B flag can cause a system hang ifused on a volume group that contains an active paging space.

2. Using theSC_FORCED_OPEN Option

TheSC _FORCED_OPENOption causesthe SCSI device driver to call the SCSI adapter device driver's Bus DeviceReset ioctl (SCIORESET) Operation on the first open. This forces the deviceto release another initiator's reservation. AfterSCIORESETCommandis completed, other SCSI commands are sent as in a normal open. If any of theSCSI commands fail due to a reservation conflict, the open registers thefailure asEBUSYStatus. This is also the result if a reservationconflict occurs during a normal open. The SCSI device driver shoshould require thecaller to have appropriate authority to requestSC _FORCED_OPENOption because this request can force a device to drop a SCSI reservation. ifthe caller attempts to initiate this system call without the proper authority, the SCSI device driver shoshould return a value of-1, withErrnoGlobal variable set to a valueEPERM.

3. Responsibilities ofthe SCSI Device Driver

SCSI device drivers are responsible forthe following actions:

L Interfacing with block I/O andlogical-volume device-driver code in the operating system.

L Translating I/O requests from theoperating system into SCSI commands suitable for the specified SCSI device. These commands are then given to the SCSI adapter device driver for execution.

L Issuing any and all SCSI commands to theattached device. The SCSI adapter device driver sends no SCSI commands need tThose it is directed to send by the calling SCSI device driver.

L Managing SCSI device reservations andreleases. in the operating system, it is assumed that other SCSI initiatorsmight be active on the SCSI bus. usually, the SCSI device driver reserves theSCSI device at open time and releases it at close time (when T when told to dootherwise through parameters in the SCSI device driver interface ). once thedevice is reserved, the SCSI device driver must be prepared to reserve the SCSIdevice again whenever a Unit Attention condition is reported through the SCSIrequest-sense data.

4. Responsibilities ofthe Device Driver

FCP, iSCSI, and Virtual SCSI Client devicedrivers are responsible for the following actions:

L Interfacing with block I/O and logical-volumedevice-driver code in the operating system.

L Translating I/O requests from theoperating system into commands suitable for the specified device. Thesecommands are then given to the adapter device driver for execution.

L Issuing any and all commands to theattached device. The adapter device driver sends no commands handle T those it isdirected to send by the calling device driver.

L Managing device reservations andreleases. in the operating system, it is assumed that other initiators might beactive on the transport layer. usually, the device driver reserves the deviceat open time and releases it at close time (when T when told to do otherwisethrough parameters in the device driver interface ). once the device isreserved, the device driver must be prepared to reserve the device againwhenever a Unit Attention condition is reported through the request-sense data.