Openstack Liberty Glance upload Image source analysis

The article is also published in the [Ceph China community], named Thomas, the other is not very clear, the need for readers, can leave a message to ask, thank you!

Creating a cloud host in OpenStack first has to be mirrored, while the Glance module provides mirroring service capabilities, including: Image discovery, retrieval, and storage, including: Glance-api and glance-registery two services, respectively, responsible for mirrored storage and metadata management. The following is based on the source code, analysis of the image upload process.

Upload image

First, through the Glance CLI upload image, to intuitively understand the next image upload process:

# Glance --Debug Image-Create --file CentOS-7.0-x64-20g.Qcow2 --Disk-format Raw --Container-format Bare --Visibility  Public

By opening it at the command line --debug , we can see that the glance CLI sends the following three different requests during the upload image:

What did the three requests do separately? Let's break it down by one by one.

Get Mirror Property Definition

Through the glance/api/v2/router.py.API route map defined in, we know that the first request above is glance/api/v2/schemas.py.Controller.image handled by the method, as follows:

#路由映射, code excerpt#完整的函数实现, please refer to glance/api/v2/router.py.api.__init__ def __init__(self, mapper):    load user-defined properties #从/etc/glance/schema-image.json fileCustom_image_properties = Images.load_custom_properties ()#创建glance/api/v2/schemas.py.controller InstancesSchemas_resource = Schemas.create_resource (custom_image_properties)#定义路由映射:    #curl-X get/schemas/image-Shemas_resource.imageMapper.connect ('/schemas/image ', Controller=schemas_resource, action=' image ', conditions={' method ': [' GET ']})#glance/api/v2/schemas.py.controller.image"" Returns a mirrored property dictionary with the following structure: [' name ': Image, ' properties ': xxx ' additionalproperties ': xxx ' definitions ': xxx ' requ ired ': xxx ' links ': xxx] "" "For dictionary values, see the following analysis process def image(self, req):    returnSelf.image_schema.raw ()

self.image_schemaglance/api/v2/schemas.py.Controller.__init__Initialize in method:

def __init__(self, custom_image_properties=None):    self.image_schema = images.get_schema(custom_image_properties)

It further calls the glance/api/v2/images.py.ImagesController.get_schema method:

 def get_schema(custom_properties=none):    #镜像的基本属性 (is a dictionary): definition and description of Id,name,status,    #通过glance The CLI upload image is successful, these field information is displayed on the shell interfaceProperties = Get_base_properties ()"" " a list of dictionaries containing 3 elements, like the name Mapping constraint [{' rel ': ' Self ', ' href ': ' {self} '}, {' rel ': ' Enclosure ', ' href ': ' {file} ' }, {' rel ': ' Describedby ', ' href ': ' {schema} '}, ' ""Links = _get_base_links ()#根据配置/etc/glance/glance-api.conf decision is to generate    #PermissiveSchema (default) or schema, the difference is Permissiveschema    #多设置了links参数    ifCONF.allow_additional_image_properties:schema = Glance.schema.PermissiveSchema (' image ', properties, links)Else: schema = Glance.schema.Schema (' image ', properties)#合并用户自定义属性    #属性合并很简单: Get the intersection of two attribute sets first, and then determine if the value of the intersection conflicts    #如果值冲突, throw an exception, or merge the DataSet    ifCustom_properties: forProperty_valueinchCustom_properties.values (): property_value[' Is_base '] =FalseSchema.merge_properties (custom_properties)returnSchema

Finally raw , the implementation is simple: Call the parent class's Schema return Mirror property dictionary, update the additionalProperties property, and then return the property dictionary to the caller

#glance/schema.py.permissiveschema  def Raw(self):Raw = Super (Permissiveschema, self). Raw () raw[' Additionalproperties '] = {' type ':' String '}returnRaw#glance/schema.py.schema def Raw(self):Raw = {' name ': Self.name,' Properties ': Self.properties,' Additionalproperties ':False,    }ifself.definitions:raw[' Definitions '] = Self.definitionsifself.required:raw[' Required '] = self.requiredifself.links:raw[' Links '] = Self.linksreturnRaw

Summary: The first request obtains the attribute dictionary definitions supported by the image, which is the basis for validating the user input parameters.

Update Database

The following analysis of the processing of the second request, based on the above --debug log and route map, we know that the request is handled by the glance/api/v2/images.py.ImagesController.create method:

 @utils. Mutating  def create(self, req, image, extra_properties, tags):     The "" " function is relatively concise, call the ' gateway ' method to create a ' image_factory ' and ' Image_repo ', followed by the try{}except code block, respectively, call two objects ' New_image ' method for a series of The checksum ' Add ' method adds the database entry and finally returns the ' image ' object to the caller; ' Gateway ' initialized in ' imagescontroller.__init__ ' method, is a ' glance/gateway.py.gateway ' instances, the instantiation of other objects is straightforward, and is no longer stated here. According to our Glance CLI command above, the values of the input parameters are as follows: Req: is a Request object that contains the requested information for the request Image: is a dictionary containing the properties of the request, as follows: {' Containe R_format ': U ' bare ', ' Disk_format ': U ' raw ', ' visibility ': U ' Public '} extra_properties: extended attribute dictionary, here Empty Tags: tags, here is empty " " "Image_factory = Self.gateway.get_image_factory (req.context) Image_repo = Self.gateway.get_repo (Req.context)#这里省略了, try{}except exception handling     #实现用户认证, policy checking, parameter checking, etc., please see the analysis belowImage = Image_factory.new_image (extra_properties=extra_properties, Tags=ta GS, **image)#进一步实现相关的检查, send message notifications and record dataImage_repo.add (image)returnImage

Then look at the get_image_factory method:

 def get_image_factory(self, context):    "" " according to the following code implementation, you can find a call chain between the objects: the return of the outermost object to the caller, called the internal method of the call to complete the relevant operation, after reaching the innermost object, and then follow the call chain to return. In fact, the following ' Get_repo ' method is also a similar implementation, the analysis below no longer repeat.    In addition ' Self.store_api ', ' self.store_utils ', ' Self.db_api ' and so on in the ' __init__ ' method The instantiation process is straightforward, here is no longer analyzed. """Image_factory = Glance.domain.ImageFactory () store_image_factory = Glance.location.ImageFactoryProxy (im Age_factory, Context, SELF.STORE_API, self.store_utils) quota_image_factory = g                                         Lance.quota.ImageFactoryProxy (store_image_factory, context, SELF.DB_API, self.store_utils) Policy_image_factory = policy. Imagefactoryproxy (quota_image_factory, Context, self.policy) Notifier_image_factory = glance.notifier.i Magefactoryproxy (policy_image_factory, context, Self.notifier)#用户可以通过 '/etc/glance/glance-api.conf in the    #property_protection_file ' option to configure the property policy, default to Disabled    ifProperty_utils.is_property_protection_enabled (): Property_rules = property_utils. Propertyrules (self.policy) pif = property_protections.                                                 Protectedimagefactoryproxy (notifier_image_factory, Context, Property_rules) Authorized_image_factory = authorization. Imagefactoryproxy (PIF, context)Else: authorized_image_factory = authorization. Imagefactoryproxy (notifier_image_factory, context)returnAuthorized_image_factory

Based on the above analysis, I have drawn the following class diagram:

You can see *ImageFactoryProxy that the class inherits from glance/domain/proxy.py.ImageFactory it, and it can also be guessed by the class name: The Mirror factory, which is used to create the encapsulated image object, and each subclass is also implemented: Permission check, message notification, policy check, quota check, etc.

In addition, classes *ImageFactoryProxy are dependent on each class *ImageProxy . Each *ImageProxy class is inherited from glance/domain/proxy.py.Image , and the class describes the property information of the mirror, including: name,image_id, status, and so on. Each *ImageProxy class is an extension of the pair Image .

The relationships of each class are clear, and new_image the following is the implementation:

#最外层的image_factory = #glance/api/authorization.py.ImageFactoryProxy, 作为调用入口image =      image_factory.new_image(extra_properties=extra_properties,                                            tags=tags, **image)

For a more intuitive display new_image of the call chain, see the following sequence diagram:

Other processes in each method are omitted from the sequence diagram ImageFactoryProxy.new_image .

You can see new_image the call from the leftmost
glance/api/authorization.py/ImageFactoryProxyTo the deepest (right number third) glance/domain/__init__.py/ImageFactory , it returns an domain/__init__.py.Image object, and then begins to layer back
( location.py/ImageFactoryProxy to authorization.py/ImageFactoryProxy ) call
glance/domain/proxy.py/Helper.proxymethod, the final result is: Returns an object that has been encapsulated at various *ImageProxy layers Image , as shown (from inside to outside):

`__init__.py/Image` <-  `location.py/ImagePorxy` <-  `authoriaztion.py/ImageProxy`

See Image The packaging process above, there is no more like the TCP/IP protocol stack of the packet process it! There is a packet, there will be a solution, the following to see:

""""add方法就是用来解包和封包的由上面的类图2，我们知道：image_repo = authorization.py/ImageRepoProxy, 输入参数image = authorization.py/ImageProxy"""image_repo.add(image)

Look at the picture and talk (see Figure 2 above):

At first glance, the sequence diagram is very similar to the above! That's right. You can see add the call from the leftmost
glance/api/authorization.py/ImageRepoProxyTo the deepest (right number two), the glance/db/__init__.py/ImageRepo first call to unproxy complete the unpacking, and then record the database entry (this time you can see the status of ' queued ' on the dashboard), and then began to layer back ( location.py/ImageRepoProxy to authorization.py/ImageRepoProxy ) the call
glance/domain/proxy.py/Helper.proxyMethods, ImageFactoryProxy like the preceding classes, each ImageRepoProxy class is completed in turn: Permission checks, attribute quota checks, policy checks, and informational notifications, and finally returns an object that has been encapsulated at various *ImageProxy layers for Image subsequent use.

Summary: Through the above analysis, we know that the second request in the upload image process, the main completion of the permission check, quota check, policy check, release notification and record database operations. Here's a look at the upload process for the image file.

Uploading image files

Based on glance CLI the above debug log and route map, we can easily find the entry to upload the image file:

#glance/api/v2/image_data.py/imagedatacontroller.upload@utils. Mutating def upload(self, req, image_id, data, size):    req: is a Request object that contains the details of the request image_id: The image ID generated during the second request, U ' 079ED99F-E5F6-49B1-86B4-695B56E26BD 9 ' data: An input object to control the subsequent image file data block upload size: The image is not specified in the command line, here is none as before Self.gateway is instantiated in the __init__ method, pointing to Glance/ga Teway.py/gateway. The Get_repo method returns the same object "" as the aforementioned class, Figure 2Image_repo = Self.gateway.get_repo (req.context) image =None    #省略try {}except exception handling    #get方法与前述的add方法类似, first remove the entry image_id points from the database,    #封装成 ' Domain/__init__.py/image ' object, which is then returned through a layer wrapper    # ' Authorization/imageproxy ' objectImage = Image_repo.get (image_id)#更新镜像状态为saving-' in-store 'Image.status =' Saving '    #省略try {}except exception handling    #save方法与上面的get方法一样, call ' Imagerepoproxy ' on a per-layer basis to complete the relevant    #检查, notification, and last update of the database entry status (this time can be seen on dashboard    #到状态为 ' in-store ')Image_repo.save (image)#和上面的save方法相似的处理方式, call ' Imageproxy ' set_data by layer    #, during which the user quotas are checked, notifications are sent, and finally based on the glance-api.conf file    #中配置存储后端上传镜像文件 (via the Add method) to the specified place storeImage.set_data (data, size)#镜像上传成功后 (after uploading a file in the ' Location.py/set_data method,    #修改状态为active), update the database status to active (this time you can    #Dashboard上看到状态为 ' in-run '), the final message is this:    "" " {' status ': ' Active ', ' name ': None, ' checksum ': ' d41d8cd98f00b204e9800998ecf8427e ', ' Created_at ': DA Tetime.datetime (6, 1, 2, 4, +), ' Disk_format ': U ' raw ', ' locations ': [{' URL ': ' Rbd://1ee20ded-caae-419d-9fe3 -5919f129cf55/images/079ed99f-e5f6-49b1-86b4-695b56e26bd9/snap ', ' status ': ' Active ', ' metadata ': {}}], ' proper  Ties ': {}, ' owner ': U ' 25520b29dce346d38bc4b055c5ffbfcb ', ' protected ': False, ' Min_ram ': 0, ' Container_format ': U ' bare ', ' Min_disk ': 0, ' is_public ': True, ' virtual_size ': None, ' id ': U ' 079ed99f-e5f6-49b1-86b4-695b56e26bd9 ', ' Size ': 0} ' ""Image_repo.save (Image, from_state=' Saving ')

Function Description Please refer to the above note, save method and add method of the process is very similar to the reader can be based on add the previous sequence diagram for further analysis; set_data the sequence diagram given below:

At this point, the process of uploading the image is finished. Hope to be useful to everyone. Today is Children's Day, here to wish you a happy holiday!

Openstack Liberty Glance upload Image source analysis