OpenStack Swift source code Introduction: overall business architecture and Proxy process

Source: Internet
Author: User
Tags openstack swift

OpenStack Swift source code Introduction: overall business architecture and Proxy process

The source code analysis of OpenStack has been widely used on the Internet, and the interpretation of each part is also very detailed. Here, I will record some key points of the Swift source code I have read Based on my own understanding, hoping to help the students who need it.

I. Overall framework of Swift

For example, the source code directory structure of Swift. Proxy is the frontend service access process. The account, container, and object directories are the business processing logic processes of the account, container, and object respectively. The common directory is some common tool code. Common is the processing logic of the hash ring. Next, we will introduce the source code logic of each process and some key mechanisms.

For the logical relationship between various business processes or modules, refer to the architecture diagram in Openstack Swift introduction.

Ii. Proxy Process Business Processing

First, you must master the PasteDeploy-based stack-based WSGI architecture. According to the layers defined by PasteDeploy, the Code flow defined by the configuration file can be quickly clarified, from middleware to server. Find the middleware on the outermost layer, which is the service entry. For the proxy process, you can give a simple business sequence diagram:

The division of labor at each layer is very clear. For example, in the default configuration file of the proxy process, exception handling is performed at the top, and unprocessed exceptions are thrown by all business processes, all of them will be processed here.

The Proxy process analyzes the resource paths composed of the requested URIaccount, iner, and object) and the request method put, del, and so on to analyze the specific types of resources requested currently, then decibel finds the controller that controls the resource, and the controller distributes requests to the specific resource server. The principle of distribution is consistent hash ring. Consistent hashing loops are generated by tools during system initialization. Specific steps are included in the Swift and Keystone standalone installation summary document.

In the introduction to Openstack Swift, we will introduce the specific node search process from the theoretical perspective. Use the md5 value plus Shift Method to Determine the part, and then find all the virtual nodes. The specific code is:

 
 
  1. container_partition, containers = self.app.container_ring.get_nodes(  
  2.  
  3.  self.account_name, self.container_name) 
  4. def get_nodes(self, account, container=None, obj=None):  
  5.  
  6.  """  
  7.  Get the partition and nodes  
  8. for an account/container/object.  
  9.  If a node is responsible  
  10. for more than one replica, it will  
  11.  only appear in the  
  12. output once. 
  13.  :param account: account name  
  14.  :param  
  15. container: container name  
  16.  :param obj: object name  
  17.  
  18.  :returns: a tuple of (partition, list of node dicts) 
  19.  Each node dict will have at least the following keys: 
  20.  ======  
  21. ===============================================================  
  22.  
  23.  id unique integer  
  24. identifier amongst devices  
  25.  weight a float of the  
  26. relative weight of this device as compared to  
  27.  
  28.  others;  
  29. this indicates how many partitions the builder will try  
  30.  
  31.  to assign  
  32. to this device  
  33.  zone integer indicating  
  34. which zone the device is in; a given  
  35.  
  36.  partition  
  37. will not be assigned to multiple devices within the  
  38.  
  39.  same zone  
  40.  
  41.  ip the ip address of the  
  42. device  
  43.  port the tcp port of the device  
  44.  
  45.  device the device's name on disk (sdb1, for  
  46. example)  
  47.  meta general use 'extra'  
  48. field; for example: the online date, the  
  49.  
  50.  hardware  
  51. description  
  52.  ======  
  53. ===============================================================  
  54.  
  55.  """  
  56.  part = self.get_part(account,  
  57. container, obj)  
  58.  return part,  
  59. self._get_part_nodes(part) 
  60. def get_part(self, account, container=None, obj=None):  
  61.   
  62. """  
  63.  Get the partition for an  
  64. account/container/object. 
  65.  :param account: account name  
  66.  :param  
  67. container: container name  
  68.  :param obj: object name  
  69.  
  70.  :returns: the partition number  
  71.  """  
  72.  
  73.  key = hash_path(account, container, obj, raw_digest=True)  
  74.  
  75.  if time() >; self._rtime:  
  76.  
  77.  self._reload()  
  78.  
  79.  part = struct.unpack_from('>;I', key)[0] >>  
  80. self._part_shift  
  81.  return part 
  82. def _get_part_nodes(self, part):  
  83.  part_nodes = []  
  84.  
  85.  seen_ids = set()  
  86.  for r2p2d in  
  87. self._replica2part2dev_id:  
  88.  if  
  89. part <; len(r2p2d):  
  90.  
  91.  dev_id =  
  92. r2p2d[part]  
  93.  
  94.  if dev_id  
  95. not in seen_ids:  
  96.  
  97.   
  98. part_nodes.append(self.devs[dev_id])  
  99.  
  100.   
  101. seen_ids.add(dev_id)  
  102.  return part_nodes 

Then, based on the quorum principle, it is determined that the current request requires at least a few nodes to return successfully. For example, if NWR is 322, at least two nodes must be successfully written to ensure that the write is successful. Reflected in the public make_request method:

 
 
  1. def make_requests(self, req, ring, part, method, path, headers,  
  2.   
  3. query_string=''):  
  4.  """  
  5.  Sends an  
  6. HTTP request to multiple nodes and aggregates the results.  
  7.  
  8.  It attempts the primary nodes concurrently, then iterates  
  9. over the  
  10.  handoff nodes as needed. 
  11.  :param req: a request sent by the client  
  12.  
  13.  :param ring: the ring used for finding backend servers  
  14.  
  15.  :param part: the partition number  
  16.   
  17. :param method: the method to send to the backend  
  18.  :param  
  19. path: the path to send to the backend  
  20.  
  21.   
  22. (full path ends up being /<$device>/<$part>/<$path>)  
  23.  
  24.  :param headers: a list of dicts, where each dict  
  25. represents one  
  26.  
  27.   
  28. backend request that should be made.  
  29.  :param query_string:  
  30. optional query string to send to the backend  
  31.  :returns: a  
  32. swob.Response object  
  33.  """  
  34.   
  35. start_nodes = ring.get_part_nodes(part)  
  36.  nodes =  
  37. GreenthreadSafeIterator(self.app.iter_nodes(ring, part))  
  38.   
  39. pile = GreenAsyncPile(len(start_nodes))  
  40.  for head in  
  41. headers:  
  42.   
  43. pile.spawn(self._make_request, nodes, part, method, path,  
  44.  
  45.   
  46. head, query_string, self.app.logger.thread_locals)  
  47.   
  48. response = []  
  49.  statuses = []  
  50.  for  
  51. resp in pile:  
  52.  if not resp:  
  53.  
  54.  continue  
  55.  
  56.  response.append(resp)  
  57.  
  58.  statuses.append(resp[0])  
  59.  
  60.  if self.have_quorum(statuses,  
  61. len(start_nodes)):  
  62.  
  63.  break  
  64.  
  65.  # give any pending requests *some* chance to finish  
  66.  
  67.  pile.waitall(self.app.post_quorum_timeout)  
  68.  
  69.  while len(response) <; len(start_nodes):  
  70.  
  71.   
  72. response.append((HTTP_SERVICE_UNAVAILABLE, '', '', ''))  
  73.   
  74. statuses, reasons, resp_headers, bodies = zip(*response)  
  75.   
  76. return self.best_response(req, statuses, reasons, bodies,  
  77.  
  78.   
  79. '%s %s' % (self.server_type, req.method),  
  80.  
  81.   
  82. headers=resp_headers)

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.