Solve the problem of deleting volume error

It's been a long time since the deletion volume error makes volume in http://www.aliyun.com/zixun/aggregation/16539.html ">error_deleting state, There was this problem when I deleted a volume yesterday, and here's a way to record the solution. Note that I am here for the back-end with iSCSI mode, specifically to me here is the TGT+LVM way.

reason

The reason I have encountered the deletion volume error is only one "device is busy", if you look at the log of the storage node where volume is located, you will see the following similar content

[-] Exception during message handling Traceback (most recent called last): File "/usr/lib/python2.6/site-packages/ cinder-2012.2.1-py2.6.egg/cinder/openstack/common/rpc/amqp.py ", line 276, in _process_data rval = Self.proxy.dispatch (Ctxt, version, method, **args) File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/openstack/common/rpc/dispatcher.py", Line 145, in Dispatch return GetAttr (Proxyobj, method) (Ctxt, **kwargs) File "/usr/lib/python2.6/site-packages/ cinder-2012.2.1-py2.6.egg/cinder/volume/manager.py ", line 206, in Delete_volume {' status ': ' error_deleting '}) File" usr/lib64/python2.6/contextlib.py ", line, in __exit__ self.gen.next () File/usr/lib/python2.6/site-packages/ cinder-2012.2.1-py2.6.egg/cinder/volume/manager.py ", line 195, in Delete_volume Self.driver.delete_volume (volume_ REF) File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/driver.py", line 203, in Delete_ Volume Self._delete_volume (volume, volume[' size ') File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/driver.py ", line, in _delete_volume run_as _root=true) File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/driver.py", line +, in _ Try_execute Self._execute (*command, **kwargs) File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/ cinder/utils.py ", line 187, in execute cmd= '. Join (CMD) processexecutionerror:unexpected the error while running command. Command:sudo cinder-rootwrap/etc/cinder/rootwrap.conf dmsetup remove-f/dev/mapper/ NOVA--VOLUMES-VOLUME--C1374407--D1B3--407F--BBCC--7416756071C1 Exit code:1 Stdout: ' Stderr: ' Device-mapper:remove IOCTL Failed:device or resource Busy\ncommand failed\n '

Why is the equipment busy? First, we tgtadm--LLD iSCSI--mode Target--op Show command at the storage node where volume is located to view the volume information we want to delete as follows

Target 7:iqn.2010-10.org.openstack:volume-c1374407-d1b3-407f-bbcc-7416756071c1 System information:driver: iSCSI State:ready i_t Nexus information:i_t nexus:24 initiator:iqn.1994-05.com.redhat:351f5adb9c85 IP address:10.61.2.5 LUN information: ...

Originally 11545.html "> We have a compute node to this volume link is not released. Here I want to say the approximate work of ISCSI+LVM, create a volume in the VG specified in the configuration item to create a size of LV size volume, and then through TGTADM this management command to create an iSCSI Target and the created LV as the back-end store for target. When mounting to an instance, the compute node where the instance is located calls the Iscsiadm command linked to the corresponding target on volume and then the underlying virtualization software assigns the target to the instance. When you uninstall volume, the underlying virtualization software removes the target from the instance and then calculates the node and then releases it. So if there's a link to a compute node that doesn't release target for some reason, it's going to happen. That was not the case at the time, but I found that if an instance of a mount with volume migrated from one node to another, the link from the source node to target would not release , that is, there are two nodes linked to target at the same time, and then there is the problem.

Solution

The process and the reason is clear, the solution is simple, release the link, reset the volume state, and then delete. First we log in compute node and release the link with the Iscsiadm command

[Root@stack5 ~]# iscsiadm-m node-t iqn.2010-10.org.openstack:volume-c1374407-d1b3-407f-bbcc-7416756071c1-u

Modify the database to reset the volume state

[root@stack5 ~]# mysql-h 10.61.2.12-u cinder-p cinder-e "Update volumes set status = ' available ' WHERE id = ' C1374407-d1b3-407f-bbcc-7416756071c1 ' "

Before this operation can be normal delete volume, but this time unexpectedly repay wrong, it seems to be a trick

Clear capabilities removing VOLUME:C1374407-D1B3-407F-BBCC-7416756071C1 [-] Exception during message handling Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/openstack/common/ rpc/amqp.py ", line 276, in _process_data rval = Self.proxy.dispatch (Ctxt, version, method, **args) File"/usr/lib/ python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/openstack/common/rpc/dispatcher.py ", line 145, in Dispatch return GetAttr (Proxyobj, method) (Ctxt, **kwargs) File "/usr/lib/python2.6/site-packages/ cinder-2012.2.1-py2.6.egg/cinder/volume/manager.py ", line 206, in Delete_volume {' status ': ' error_deleting '}) File" usr/lib64/python2.6/contextlib.py ", line, in __exit__ self.gen.next () File/usr/lib/python2.6/site-packages/ cinder-2012.2.1-py2.6.egg/cinder/volume/manager.py ", line 193, in Delete_volume self.driver.remove_export VOLUME_REF) File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/driver.py", Line 474, in Remove_export self.tgtadm.remove_iscsi_target (iscsi_target, 0, volume[' id ']) File "/usr/lib/python2.6/ site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/iscsi.py ", line, in Remove_iscsi_target raise exception. Iscsitargetremovefailed (volume_id=vol_id) iscsitargetremovefailed:failed to remove iSCSI target for volume C1374407-d1b3-407f-bbcc-7416756071c1. Workaround Two

This method is more violent, direct manual. Update database tags volume for deleted, the storage node with the Tgtadm command to delete the corresponding target, empty the corresponding LV data, with lvremove command to remove LV, finish work.

[Root@store2 ~]# mysql-h 10.61.2.12-u cinder-p cinder-e "Update volumes set deleted = 1, Deleted_at = Now () where id = ' C13 74407-d1b3-407f-bbcc-7416756071c1 ' [Root@store2 ~]# tgtadm--lld iSCSI--mode Target--op Delete--tid 7[root@store2 ~]# DD If=/dev/zero OF=/DEV/MAPPER/NOVA--VOLUMES-VOLUME--C1374407--D1B3--407F--BBCC--7416756071C1 bs=1M[root@store2 ~] # LVREMOVE/DEV/MAPPER/NOVA--VOLUMES-VOLUME--C1374407--D1B3--407F--BBCC--7416756071C1

Northern Polytechnical University | Cloud Computing Research Center | Jiang Yong