Openstack_swift source code analysis-create and join a ring? Detailed analysis of device source code algorithms

1. Create a ring code for detailed analysis

In the openstack_swift -- Ring organization architecture, we have analyzed the specific working process of the ring. Which of the following statements is added to the ring? Device, delete the device, and the implementation process of balance has been described in detail.

First, let's look at the ringbuilder class.

Def _ init _ (self, part_power, replicas, min_part_hours): # Why Max. 2 ** 32 If part_power> 32: raise valueerror ("part_power must be at most 32 (was % d)" % (part_power,) If replicas <1: raise valueerror ("replicas must be at least 1 (was %. 6f) "% (replicas,) If min_part_hours <0: Raise valueerror (" min_part_hours must be non-negative (was % d) "% (min_part_hours,) self. part_power = part_power self. re Plicas = replicas self. min_part_hours = min_part_hours self. parts = 2 ** self. part_power # Number of partitions is the number of virtual nodes self. Devs = [] # used to store devices? Will be added to the Devs self. devs_changed = false self. Version = 0

        self._last_part_moves_epoch = None        self._last_part_moves = None        self._last_part_gather_start = 0        self._remove_devs = []        self._ring = None

None of the dev attributes in Devs are: Dev = {'weight ': 100.0, 'zone': 0, 'IP': '123. 0.0.1 ', 'region': 0, 'parts': 0, 'id': 0, 'meta': 'Some meta data', 'device': 'sda1 ', 'parts _ wanted': 96, 'Port': 6000}

Which of the following is part_wanted? Number of virtual nodes required by the device after the device, and parts is the virtual node allocated to the device.

In swift-ring-builder object. Builder create 18 3 1

The create method of the command class is called. For details, refer to this method:

Def create (Self): "" Swift-ring-builder object. builder create 18 3 1swift-ring-builder <builder_file> Create <part_power> <replicas> <min_part_hours> creates <builder_file> with 2 ^ <part_power> partitions and <replicas>. <min_part_hours> is number of hours to restrict (Restriction) moving a partition more than once (more than once ). "If Len (argv) <6: Print commands. create. _ Doc __. strip () Exit (exit_error) builder = Ringbuilder (INT (argv [3]), float (argv [4]), INT (argv [5]) backup_dir = pathjoin (dirname (argv [1]), 'backups') Try: mkdir (backup_dir) Before t oserror as ERR: If err. errno! = Eexist: Raise builder. save (pathjoin (backup_dir, '% d. '% time () + basename (argv [1]) builder. save (argv [1]) # serialize exit (exit_success)

When you create an object for the first time, the system first determines whether an object exists in/etc/SWIFT. builder file, there is no need to create this file, and according to the 18 3 1 in the command to instantiate ringbuilder, and finally serialize builder.

2 join for ring? Device

After the. Builder file is created, do you need to add it? Device. Which of the following is join? A device command: Swift-ring-builder object. Builder add z2-127.0.0.1: 6020/sdb2 100

"Add" indicates "join? Z2 of the device is zone2, 127.0.0.1: 6020/sdb2 is the IP address of the device, the bound port, and the disk of the file stored in the device. 100 indicates the weight of the device.

See join below? Detailed device implementation:

Whether the object exists in/etc/SWIFT in the main method. builder file. In Step 1 create, we have created this file. Now we only need to deserialize it and instantiate the ringbuilder class with the stored data, and then call the add method of the commands class:

Def add (Self): "Swift-ring-builder <builder_file> Add [r <region>] z <zone>-<ip>: <port> [r <r_ip>: <r_port>]/<device_name >_< meta> <weight> [[r <region>] z <zone>-<ip>: <port> [r <r_ip>: <r_port>]/<device_name >_< meta> <weight>]... where <r_ip> and <r_port> are replication IP and port. orswift-ring-builder <builder_file> Add [-- region <region>] -- Zone <zone> -- IP <ip> -- Port <port> -- replication-IP <r_ip> -- rep Lication-port <r_port> -- device <device_name> -- meta <meta> -- weight <weight> adds devices to the ring with the given information. no partitions will be assigned to the new device until after running 'rebalance '. this is so you can make multiple device changes and rebalance them all just once. "If Len (argv) <5 or Len (argv) % 2! = 1: Print commands. add. _ Doc __. strip () Exit (exit_error) for new_dev in _ parse_add_values (argv [3:]): For Dev in builder. devs: if Dev is none: continue if Dev ['IP'] = new_dev ['IP'] And Dev ['Port'] = new_dev ['Port'] And Dev ['device'] = new_dev ['device']: print 'device % d already uses % s: % d/% S. '% (Dev ['id'], Dev ['IP'], Dev ['Port'], Dev ['device']) print "The on-disk ring builder is unchanged. \ n "exit (exit_error) dev_id = builder. add_dev (new_dev) # Call the add_dev method print ('device % s with % s weight got ID % s' % (format_device (new_dev), new_dev ['weight'], dev_id) builder. save (argv [1]) Exit (exit_success)

Add new? The device will be added to Devs [], and add_dev will be called. The detailed implementation of this method is as follows:

Def add_dev (self, Dev): "" add a device to the ring. this device dict shoshould have a minimum of the following keys: ========================================================== ============================ ID unique integer identifier amongst devices. ults to the next ID if the 'id' key is not provided in the dict weight a float of the relative weight of this device as compared to others; this indicates how each partitions the builder will try to assign to this device region integer indicating which region the device is in Zone integer indicating which zone the device is in; A given partition will not be assigned to multiple devices within the same (Region, zone) pair if there is any alternative IP the IP address of the device port the TCP port of the device's name on disk (sdb1, for example) meta general use 'extra 'field; for example: the online date, the hardware description ========================================== ========================================== .. note: this will not rebalance the ring immediately as you may want to make multiple changes for a single rebalance.: Param Dev: Device dict: Returns: ID of device Dev = {'weight': 100.0, 'zone ': 0, 'IP': '123. 0.0.1 ', 'region': 0, 'parts': 0, 'id': 0, 'meta': 'Some meta data', 'device': 'sda1 ', 'parts _ wanted': 96, 'Port': 6000} "If 'id' not in Dev: Dev ['id'] = 0 if self. devs: Dev ['id'] = max (d ['id'] for D in self. devs if D) + 1 if Dev ['id'] <Len (self. devs) and self. devs [Dev ['id'] is not none: Raise exceptions. duplicatedeviceerror ('duplicate device ID: % d' % Dev ['id']) # Add holes to self. devs to ensure self. devs [Dev ['id'] will be the dev while Dev ['id']> = Len (self. devs): Self. devs. append (none) Dev ['weight'] = float (Dev ['weight']) Dev ['parts'] = 0 self. devs [Dev ['id'] = Dev self. _ set_parts_wanted () # Set part_wanted self. devs_changed = true self. version + = 1 return Dev ['id']

This method is mainly used to obtain weight, assign values to the ID in the device Dev, and assign values to part_wanted. The part_wanted value is calculated as follows:

(Self. parts * self. replicas/sum (d ['weight'] for D in self. _ iter_devs () * Dev ['weight'])-dev ['parts']
Parts = 2 ** power (18 in the create method here) Replicas is the number of backups, where 3

That is, part_wanted = (number of virtual nodes * Number of backups/all added? And) * weight of the current device)-Number of virtual nodes allocated to the device

Now create a ring and add it to the ring? After the device explanation is complete and the two steps are completed, the balancing ring is required and assigned a value for replica2part2dev (the ing from the backup to the partition to the device). The balancing algorithm is the core algorithm of the ring, it will be explained in detail in the next article.

Due to my limited level, understanding errors may inevitably occur in this article. please correct me and communicate with me. Thank you!