Nova-conductor a single process consumes 100% CPU

The Nova-conductor process runs at run time, where a single process consumes CPU 100% periodically, with a period of approximately 2 minutes. debugging and troubleshooting, found that the reason is that nova-conductor in the execution of a database operation, the request data volume is huge, only the database lookup time is 10s, the return data size at 2MB. Causes the data to run out of CPU during serialization and deserialization, and for a longer duration. Details are as follows:

Operation Request: Object_class_action

Request parameters: {u ' objver ': U ' 1.6 ', U ' objmethod ': U ' get_by_filters ', U ' args ': [{u ' deleted ': True, U ' soft_deleted ': False, U ' host ': U ' node-144}], U ' objname ': U ' instancelist ', U ' Kwargs ': {u ' use_slave ': True}

Function: The information of all deleted instances on the compute node is queried according to the host information, note that the deleted=true of filter For instance database information is obtained here. Here is the database information for all deleted instances on the node-144 compute node, execution time 140.429696083s, where CPU utilization is basically 100%. As the OpenStack system continues to run, there will be more and more deleted virtual machine information in the database, and the data lookup operation will continue to grow and the CPU will take longer. Node-144 on the creation of deleted virtual machine more, so the query for the longest time, take up CPU 100% time is also the longest, for other compute nodes on the instancelist query will continue around 10s.

requested by: Nova-compute

function to execute the request (incomplete statistics):

• _run_pending_deletes (scheduled task, default 3m execution once)
• _cleanup_running_deleted_instances (scheduled task, default 30m execution once)

Recommendation:Clean up the database regularly!

Nova-conductor a single process consumes 100% CPU