windows2012 R2 cluster disk becomes raw Unable to format the resolution

Symptom: After a windows2012 R2 failover cluster is removed, after reinstalling the system, it is found that the disk has become raw and the formatting does not complete successfully after it has been brought online.

Analysis: Due to the abnormal removal of the cluster, these disks are still locked by the PR Key (SCSI-3 reservation) of the previous cluster, causing the disks to be occupied and inaccessible and formatted.

Processing:
The following cleanup reserved disk command is performed on the exception disk for one of the nodes, followed by the disk tag number in Disk Manager,-disk, in Https://technet.microsoft.com/en-us/library/ee461016.aspx,
command to clean up cluster information:
Clear-clusterdiskreservation-disk <disk_number>

The disk status is normal after you execute it sequentially.

Summary: When reloading a cluster, it is important to first exit the cluster node in turn and clear the cluster disk, and then remove the cluster to prevent the shared disk from continuing to be locked by the previous cluster's PR key.

Knowledge Supplement:
SCSI locks are the basic mechanism used by multiple hosts to operate LUNs. In a Windows storage environment, SCSI locks are used when multiple Windows hosts require access to a LUN, such as a Windows cluster environment.
Principle Interpretation:
In a shared storage environment, multiple hosts may access the same storage device at the same time, if more than one host at the same point in time to write a LUN, then it is conceivable that this LUN will not know which data to write first, which data is written. In order to prevent the data corruption caused by this situation, the concept of SCSI lock is introduced. With SCSI reservation mechanism for SCSI lock operation, most of the disks currently support the ' SCSI reservvation command '. If a host transmits a SCSI reservation command to the disk, the disk is locked for the other hosts. If there are other hosts that send read-write requests to the disk that is already locked, you will receive a ' Reservation conflict ' error message. If the host that retains the SCSI lock crashes, or another host sends a ' break reservation or reset target command to the disk, it is used to unlock the SCSI lock. Then, the second host sends an I/O request before it needs to resend the SCSI reservation command to the disk.

Classification of SCSI Locks:

There are two types of SCSI locks: SCSI-2 Reservation and SCSI-3 reservation. Only one type of SCSI lock can exist on a LUN.

SCSI-2 reservation only allows the device to be accessed by a SCSI lock initiator, which is the host's HBA. For example, HBA1 on Host 1 adds a SCSI-2 lock to the LUN that is accessed, even if the HBA2 of host 1 cannot access the LUN. So SCSI-2 reservation is also known as single Path reservation.

SCSI-3 Reservation (persistent reservation) uses a PR key to lock the disk. Usually a host will have a unique PR key, different host corresponding to the PR key is also different. So the general SCSI-3 reservation is usually applied under a multi-channel shared environment. Here SCSI-3 reservation also called persistent reservation.

SCSI locks in Windows cluster:

 Windows 2003集群中使用SCSI-2 reserve/release命令。作为非持久的reservation，所以集群中的一台节点会持有SCSI-2 Reservation的锁，然后每过3秒会重新刷新一次。如果故障转移发生，则切换节点主机会在相应的磁盘上放置SCSI-2 Reservation然后维护SCSI锁。如果所有节点主机上的集群服务都会关闭，Reservation也不会保留。 Windows 2008以上集群中使用SCSI-3 persistent reservation机制。如果磁盘从主机上没有正确移除，集群使用的磁盘（Cluster Disk）会保留着这些Reservation。锁对应的SCSI锁会一直存在于相应的磁盘之上，即使集群服务被关闭或者磁盘对于主机取消掩饰（unmasked）。所以，有些时候需要强行移除磁盘上的Reservation。      什么情况下设备会被加锁？

A lock is added to the general device when it is opened. such as VARYONVG, DD, and so on, it is important to note that for the DD command, when it is running, the device will be locked and automatically unlocked when the operation is completed.

Note: The varyonvg-c does not lock the device.

In addition, only VARYOFFVG or varyonvg-b will unlock the VG-related devices after the VG Varyon. Direct with the shutdown command does not do VARYOFFVG action, so it will not unlock.

How does the Cluster service keep a disk and return the disk to online status?

The Cluster service uses only the SCSI protocol to manage disks on the shared bus.

Note: This does not mean that all disks will be of type SCSI, specified as SCSI hardware interfaces, but instead, the storage unit must be able to correctly interpret and process SCSI protocols and commands.

The following command is the other SCSI protocol feature that will be used when disks in a clustered environment.

保留︰ 通过获得或保留所有权的 SCSI 设备的主机总线适配器发出此命令。保留设备的所有其他主机总线适配器，但最初保留其发起人之一会拒绝所有的命令。释放︰ 所属的主机总线适配器发出此命令，当磁盘资源处于脱机状态。它释放保留的另一台主机总线适配器的 SCSI 设备。重置︰ 在目标设备上，此命令中断预留。此命令可以重置 （对于整个总线） 或使用 storport 驱动程序目标重置总线上的特定设备的总线。

The following procedure describes how a server cluster starts and takes control of a shared disk. This scenario assumes that only one node is opened at a time:

When the computer starts, the cluster disk driver (Clusdisk.sys) reads the following local registry key to obtain a list of shared disk signatures that are managed in the cluster:
Hkey_local_machine\system\currentcontrolset\services\clusdisk\parameters \signatures
After the list is obtained, the Cluster service attempts to scan the shared SCSI bus on all devices to find a matching disk signature.

When the cluster disk driver is started on the first node in the cluster, all LUNs marked for the first time (the Lun: logical unit number, the unique identifier used on the SCSI bus to differentiate the device sharing the same bus), match the signing key to an offline volume. Note that this is not the same, but to take the cluster resources offline. The volume is marked offline to prevent multiple nodes from having volume write permissions at the same time. If the cluster shares a disk cluster, one of the disks is designated as the quorum disk for the Cluster service. When the quorum disk is the first resource online, the Cluster service attempts to form a cluster.

When the Cluster service on the Forming node starts, it first attempts to online specify the physical device as the quorum disk. Performs a disk quorum algorithm on the quorum disk that gets ownership. For a successful quorum, the Cluster service sends a request to ClusDisk to start sending periodic reservations to disk (to retain ownership). The Cluster service sends the request to ClusDisk to unblock the access to the quorum disk, and then mount the volume on the disk. Successfully mount the volume, complete the online process and the Cluster service, and then continue to use the form in the cluster process. Requests a disk that is passed from the cluster disk driver to the Microsoft storage driver stack and finally to the HBA-specific driver for communication. It may also pass to any multipath software running in the storage stack. For more information about the storage stack and driver model, click the following link:

After the storage controller/device driver reports that the device was successfully retained, the Cluster service ensures that the drive can be read and written. Once all the disks in these tests are crossed, the disk resource is marked online and the Cluster service continues to bring all other resources online.

Each node in the cluster renews its reservation for any Lun every three seconds. If a node in the cluster is lost (for example, if there is no communication on the private or public network) to communicate with other networks, the node begins the process known as quorum to determine ownership of the quorum disk. The node that wins the communication between the cluster nodes that owns the quorum disk resource remains in effect. The Cluster service and any resources, any node that cannot communicate, cannot maintain or gain ownership of the quorum disk will terminate the node's hosting will be moved to another node in the cluster.

目前拥有仲裁磁盘的节点是保护节点。Defender 假设，它抵御任何群集节点它无法与通信和未接收关闭通知。Defender 不断通过 SCSI 请求续订仲裁为其预留储备放在 LUN 上每三秒。所有其他节点 （节点不拥有仲裁磁盘并不能传达与拥有仲裁资源的节点） 将成为具有挑战性的节点。当挑战者检测到丢失的所有通信时，它会立即请求总线范围的 SCSI 重置中断任何现有的预留。7 秒后重置 SCSI 请求，挑战者尝试保留仲裁磁盘。如果 defender 节点联机并且运行正常，它将已经保留了仲裁磁盘那样通常每隔三秒。挑战者并检测到它不能预留仲裁，群集服务将终止。如果 defender 工作不正常，挑战者可以成功保留仲裁磁盘。10 秒钟后，挑战者将在线仲裁，并将群集中的所有资源的所有权。如果保护节点失去仲裁设备的所有权，然后保护节点上的群集服务立即终止。

When the disk resource required by the cluster node goes offline, it requests that the SCSI reserved drive be freed and then again will be unavailable to the operating system. As long as the cluster's disk resources are offline in the cluster, the volume that the resource points to (a disk with matching signatures) will not be able to access the operating system on any of the cluster nodes.

windows2012 R2 cluster disk becomes raw Unable to format the resolution