CloudStack's SSVM and CPVM cannot be deleted due to host downtime

Problem Background:

The physical machine Running SSVM and CPVM goes down. Check that the status of SSVM and CPVM is still Running, and the host still displays as a down physical machine. Therefore, the physical machine is successfully started, log on to the physical machine and run the virsh list -- all command to check whether SSVM and CPVM run successfully. If no, query all physical machines and no SSVM or cpvm vm is found, however, the CloudStack UI shows that SSVM and CPVM are always Running and run on the host. Of course, the IP address cannot be pinged.Deleting SSVM and CPVM does not work. Even stopping an instance fails, but the instance can be created smoothly. This is a big bug!

Log info:/var/log/cloudstack/management/management-server.log

2013-12-17 21:33:26,525 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-130:job-130) Executing org.apache.cloudstack.api.command.admin.systemvm.DestroySystemVmCmd for job-1302013-12-17 21:33:26,527 DEBUG [cloud.api.ApiServlet] (catalina-exec-9:null) ===END===  10.200.251.246 -- GET  command=destroySystemVm&id=94576696-a734-459b-b697-9ade8d616e68&response=json&sessionkey=yY8M0StWM6ohsnSO3nhPZGj7xKk%3D&_=13873339954952013-12-17 21:33:26,612 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-130:job-130) VM state transitted from :Running to Stopping with event: StopRequestedvm's original host id: 1 new host id: 1 host id before state transition: 12013-12-17 21:33:26,618 WARN  [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-130:job-130) Unable to stop vm, agent unavailable: com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Host with specified id is not in the right state: Disconnected2013-12-17 21:33:26,618 WARN  [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-130:job-130) Unable to stop vm VM[SecondaryStorageVm|s-1-VM]2013-12-17 21:33:26,628 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-130:job-130) VM state transitted from :Stopping to Running with event: OperationFailedvm's original host id: 1 new host id: 1 host id before state transition: 12013-12-17 21:33:26,628 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-130:job-130) Unable to stop the VM so we can't expunge it.2013-12-17 21:33:26,628 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-130:job-130) Unable to destroy the vm because it is not in the correct state: VM[SecondaryStorageVm|s-1-VM]2013-12-17 21:33:26,628 INFO  [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-130:job-130) Did not expunge VM[SecondaryStorageVm|s-1-VM]2013-12-17 21:33:26,640 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-130:job-130) Complete async job-130, jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Fail to destroy system vm2013-12-17 21:33:26,728 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 15-1464552034: Received:  { Ans: , MgmtId: 345051385634, via: 15, Ver: v1, Flags: 10, { GetHostStatsAnswer } }2013-12-17 21:33:27,100 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-13:null) Ping from 82013-12-17 21:33:27,235 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-9:null) Ping from 142013-12-17 21:33:27,454 DEBUG [agent.transport.Request] (AgentManager-Handler-8:null) Seq 8-1342917711: Processing:  { Ans: , MgmtId: 345051385634, via: 8, Ver: v1, Flags: 10, [{"Answer":{"result":false,"details":"timeout","wait":0}}] }2013-12-17 21:33:27,455 DEBUG [agent.transport.Request] (AgentManager-Handler-12:null) Seq 8-1342917712: Processing:  { Ans: , MgmtId: 345051385634, via: 8, Ver: v1, Flags: 10, [{"Answer":{"result":false,"details":"timeout","wait":0}}] }2013-12-17 21:33:27,455 DEBUG [agent.transport.Request] (AgentTaskPool-3:null) Seq 8-1342917711: Received:  { Ans: , MgmtId: 345051385634, via: 8, Ver: v1, Flags: 10, { Answer } }2013-12-17 21:33:27,455 DEBUG [cloud.ha.AbstractInvestigatorImpl] (AgentTaskPool-3:null) host (10.196.53.73) cannot be pinged, returning null ('I don't know')2013-12-17 21:33:27,455 DEBUG [cloud.ha.UserVmDomRInvestigator] (AgentTaskPool-3:null) sending ping from (9) to agent's host ip address (10.196.53.73)2013-12-17 21:33:27,455 DEBUG [agent.transport.Request] (AgentTaskPool-16:null) Seq 8-1342917712: Received:  { Ans: , MgmtId: 345051385634, via: 8, Ver: v1, Flags: 10, { Answer } }2013-12-17 21:33:27,455 DEBUG [cloud.ha.AbstractInvestigatorImpl] (AgentTaskPool-16:null) host (10.196.53.74) cannot be pinged, returning null ('I don't know')2013-12-17 21:33:27,455 DEBUG [cloud.ha.UserVmDomRInvestigator] (AgentTaskPool-16:null) sending ping from (9) to agent's host ip address (10.196.53.74)2013-12-17 21:33:27,460 DEBUG [agent.transport.Request] (AgentTaskPool-3:null) Seq 9-241192500: Sending  { Cmd , MgmtId: 345051385634, via: 9, Ver: v1, Flags: 100011, [{"PingTestCommand":{"_computingHostIp":"10.196.53.73","wait":20}}] }2013-12-17 21:33:27,461 DEBUG [agent.transport.Request] (AgentTaskPool-16:null) Seq 9-241192501: Sending  { Cmd , MgmtId: 345051385634, via: 9, Ver: v1, Flags: 100011, [{"PingTestCommand":{"_computingHostIp":"10.196.53.74","wait":20}}] }2013-12-17 21:33:27,585 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 16-1532317381: Received:  { Ans: , MgmtId: 345051385634, via: 16, Ver: v1, Flags: 10, { GetHostStatsAnswer } }2013-12-17 21:33:27,890 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-1:null) Ping from 11

Key Information:
Unable to destroy the vm because it is not in the correct state: VM [SecondaryStorageVm | s-1-VM] Database InformationMysql> SELECT * FROM host WHERE name like '% s-1-VM %' \ G // system VM information in host information **************** * *********** 1. row *********************** id: 21 name: s-1-VM uuid: 986db967-13a9-48ca-815b-c41d6951a3f3 status: DisconnectedType: SecondaryStorageVM private_ip_address: commandid: 255.255.255.0 private_mac_address: 06: 51: e0: 00: 00: 07 principal: commandid: 255.255.255.0 storage_mac_address: 06: 51: e0: 00: 00: 07storage_ip_address_2: NULLstorage_mac_address_2: NULL storage_netmask_2: NULL cluster_id: NULL public_ip_address: 10.196.53.76 public_netmask: 255.255.255.0 public_mac_add Ress: 06: e0: 2c: 00: 00: 0e proxy_port: NULL data_center_id: 1 pod_id: 1 cpus: NULL speed: NULL url: NoIqn fs_type: NULL hypervisor_type: NULL hypervisor_version: NULL ram: 0 resource: NULL version: 4.1.1 parent: NULL total_size: NULL capabilities: NULL guid: s-1-VM-NfsSecondaryStorageResource available: 1 setup: 0 dom0_memory: 0 last_ping: 1354828061 mgmt_server_id: 345051385634 disconnected: NULL cr Eated: 05:18:54 removed: NULL update_count: 2 resource_state: Enabled owner: NULL lastUpdated: NULL engine_state: Disabled1 row in set (0.00 sec) mysql> SELECT * FROM vm_instance WHERE name like '% s-1-VM %' \ G // system VM information in the VM instance, the instance and system VM status on the cloudstack interface are read from the state field in the table. * *************************** 1. row *********************** id: 22 name: s-1-VM uuid: 8bd3ab0c-a431-4dd2-85a7-013921427f6a instance_name: s-1-VM state: RunningVm_template_id: 3 guest_ OS _id: rule: 06: 51: e0: 00: 00: rule: 10.196.53.74 pod_id: 1 data_center_id: 1 host_id: 15 last_host_id: 15 proxy_id: 55 principal: 05:20:52 vnc_password: Login/+ login = ha_enabled: 0 limit_cpu_use: 0 update_count: 3 update_time: 05:18:59 created: 05:17:04 removed: NULL type: Login vm_type: Login account_id: 1 domain_id: 1service_offering_id: 9 reservation_id: a2a55809-abfa-4b6e-92f8-105cf8bef2a8 hypervisor_type: KVM disk_offering_id: NULL cpu: NULL ram: NULL owner: NULL speed: NULL host_name: NULL display_name: NULL desired_state: NULL1 row in set (0.01 sec) Key IssuesIt is the red annotation of two fields in the database. One table shows Disconnected, the other table shows Running, and the two system VMS are also Running on the CloudStack UI. Solution:Anyone who knows about these two virtual machines knows that this is a very powerful Virtual Machine and can be rebuilt after deletion. Generally, the two virtual machines have faults and are also solved through deletion and reconstruction, since the UI cannot be deleted, modify the corresponding field in the database and set its status to Destroyed. UPDATE vm_instance SET state = 'deststroed' WHERE name ='s-1-VM '; UPDATE vm_instance SET state = 'deststroed' WHERE name = 'v-2-VM '; then go back to the CloudStack UI to view 650) this. width = 650; "src ="/e/u/themes/default/images/spacer.gif "style =" background: url ("/e/u/lang/zh-cn/images/localimage.png") no-repeat center; border: 1px solid # ddd; "alt =" spacer.gif "/> 650) this. width = 650; "src =" http://www.bkjia.com/uploads/allimg/131229 /11532231U-0.png "title =" successfully created ssvm.png "alt =" 175843246.png"/> after the system detects that the status of the original two system VMS is Destroyed, the SSVM and CPVM are rebuilt, wait until the status shows Running, and the system returns to normal.

This article from the "dark magic king Teng cloud end" blog, please be sure to keep this source http://clovemfong.blog.51cto.com/3297559/1342180