Processing of large objects (greater than 5 Gb) in swift

This article is translated from the "managing large objects (greater than 5 Gb)" section in the official swift document "openstack Object Storage Administration Manual" and tested to verify the content in the document. PointHereYou can see the online document.

 

By default, the maximum size of a single object Uploaded By swfit is 5 GB. However, there is no limit on the size of the downloaded object. The concept of "unlimited download" is achieved by organizing the segmented object. Each segment of a large object is uploaded separately, and then a manifest file is created to indicate the segment contained in the object, the list indicates that all segments are organized into a separate object for download. This method increases the upload speed at the same time, because we can upload multiple parts of an object in parallel.

 

Use swift commands to manage segmented objects

The quickest way to try this feature is to use the openstack Object Storage client tool. When uploading an object, you can use the-s option to specify the size of each segment, for example:

 
# Swift upload test_container-S1073741824Large_file
 

This command divides large_file into 1 GB segments, and then uploads these segments in parallel. Once all the segments are uploaded, the swift client tool creates a manifest file for these segments to ensure that these segments can be downloaded as an object large_file.

The following command downloads the entire large object:

 
Swift download test_container large_file
 

The swift command line tool uses a strict conversion method to support segmented objects. In the preceding example, the swift command line tool uploads all segments to another container (test_container_segments). These segments are named like large_file/1290206778.25/21474836480/00000000, large_file/1290206778.25/21474836480/00000001.

The main benefit of using a separate container to store segments is that when we perform the listings operation on the main container, all segment names will not be displayed. Use<Name>/<timestamp>/<size>/<segment>This format clearly indicates the segment of an object because if you upload a new object with the same name, the previous object will not be overwritten and the configuration file will be updated.

The swift command line tool will help you manage these segments and delete old segments when they are deleted or overwritten. If necessary, you can use the-leave-segments option to overwrite these behaviors. S is useful when you want to manage large objects in multiple versions.

 

Directly use a large object

You can directly use HTTP requests instead of the SWIFT client tool to perform segment and list operations. You can directly upload parts and list files just like uploading other files.X-object-manifestThe header size is 0 bytes.

All segments of an object must be stored in the same container. They have a common object name prefix and are named in the order they deserve.. They can be different from the inventory file in the same container, which can keep the list of objects in the container clean.

The configuration file is only a file with a size of 0 bytes and hasX-object-manifest: <container>/<prefix>Common file in the header. <Container> is the name of the container where the object segments are located, and <prefix> is the public prefix of these segments.

The best way is to upload all parts before creating/updating the list file. In this case, the object will not be accessed by the download request before it is completely uploaded. You can also upload a bunch of new segments to another location, and then update the inventory file to point to the new location. In the process of uploading these new parts, the original list file can still be accessed, that is, the download request for this list file during the process of uploading the new segment, or return the original segment object.

Here is an example of using curl to perform a small multipart upload with 1 byte:

# First, upload all parts

$ Curl-x put-H '  X-auth-Token: <token>  ' HTTP: //  <Storage_url>/container/ Myobject/ 1 -- Data-binary '  1  '  $ Curl -X put-H '  X-auth-Token: <token>  ' HTTP: //  <Storage_url>/container/ Myobject/ 2 -- Data-binary '  2  ' $ Curl -X put-H '  X-auth-Token: <token>  ' HTTP: //  <Storage_url>/container/ Myobject/ 3 -- Data-binary '  3  ' 

# Then, create the inventory file

Curl-x put-H'X-auth-Token: <token>'-H'X-object-manifest: container/myobject/'HTTP://<Storage_url>/container/myobject -- Data-binary''
 

# Now we canUse these segments as an object for download

 
Curl-H'X-auth-Token: <token>'HTTP://<Storage_url>/container/myobject
 

 

Additional Notes for large objects

Execute a get or head request to the configuration file,X-object-manifest: <container>/<prefix>The object after connection is returned together, so you canX-object-manifestDetermine where the segment is obtained.

Execute the get or head request on the configuration file and the Response HeaderContent-LengthThe value is the total size of all segments it points to, so this value is dynamic. Therefore, when you upload additional parts after creating the inventory file, the connected objects will become larger, but you do not need to re-construct the inventory file.

Execute the get or head request on the configuration file and the Response HeaderContent-TypeAndContent-TypeSame. Therefore, you can change the value by resending the put request.Content-Type.

Execute the get or head request on the configuration file and the Response HeaderEtagThe value is the MD5 value of the character string that the etag of each segment points to. Therefore, this value is dynamic. In general, the etag value of an object in SWIFT is the MD5 value of the object content, which is also true for each individual segment. However, the inventory file itself does not produce such an etag value, so this method can at least be used to determine file changes.

 

History and background of Large Object Storage Service

Before the implementation method of the current version, the support for large objects has undergone several iterations.

The main reason for limiting the size of a single object in SWIFT isBalance different partitions on the ring. To ensure that the disks in the cluster can be used evenly, it is clear that we need to break down large files into smaller segments, and then merge small segments during the reading operation.

Before providing support for large objects, some applications have split the objects to be uploaded into some independent segments on the client, and then reorganized these segments. This design allows the client to back up and archive large datasets. However, due to network interruptions and other problems, it cannot improve performance and reduce errors. The main disadvantage of using this method is that because the original Division information is required to reorganize the object, this method is not practical in some scenarios (such as CDN sources.

To eliminate the client's requirement for storing objects larger than 5 GB, we initially designed a completely transparent support method for uploading large objects. To achieve completely transparent upload of large objects, the proxy server will be responsible for segmenting this large object during object upload without modifying any APIs. Therefore, all segments are completely hidden after the client API.

However, this implementation will cause some problems: it does not support concurrent uploading of an object on the client, and there is no error recovery mechanism. Compared with the benefits of a fully transparent segmentation method, its complexity is still too high.

In our current "User manifest" design, in order to provide a transparent large object Download Method for the client, we can also support multipart upload and keep the client APIs clean and tidy.

Another "display" User manifest implementation method has also been discussed. It needs to pre-define the format of the uploaded parts. Although this method can provide some potential advantages, to implement a simpler API, we should avoid pushing the burden to the client (in fact, determines the format of "X-object-manifest ).

During the development process, we noticed that this "implicit" user manifest method based on the public path prefix willWill be affected by the size of the final consistency windowIn theory, this situation occurs: when an object is uploaded for a short period of time, the get operation on the list file will return an invalid object. In fact, unless your swift runs in a highly concurrent upload test environment, you are unlikely to encounter this situation (in a highly concurrent upload test environment, do not run object-updaters and container-replicators ).

Like all features in openstack Object Storage, supporting large objects is a developing feature that will continue to improve and may change over time.

 

Test result 1: swift CLI uploads and downloads large files.

The size of the repost_backup2. SQL test file is 12.3 GB.

1) if you upload the file directly, an error is reported. The file is too large.

2) specify multipart upload, with each segment being 5 GB. The swift CLI tool divides the file into three segments and creates a configuration file.

3) after the upload is successful, check the container and find that the docs_segments container is added.

4) The target container only has the repost_backup2. SQL list file, and there is no segment object. The docs_segments container stores the segment object.

5) The multipart object is used as a large object for download.

 

Test result 2: curl multipart upload and merge download

1) Upload three 1-byte files with the public prefix "myobject /"

2) create a configuration file, specify the value of X-object-manifest as the public prefix "docs/myobject/", and name it the myobject you know.

3) view the upload result. Three segment objects and one list file are displayed.

4) download the myobject object. You can see that the returned results concatenate the three multipart files and return only one myobject object.

 

In addition, by carefully observing the HTTP header information, we can also look at the content described in the "additional notes for large objects" section.

 

Some Thoughts

Now, I think of some issues that need to be paid attention to when developing APIs:

1. Object type definition. Due to the existence of the pseudo directory, you need to add the subdir attribute to the object to distinguish the pseudo directory from the real object. Due to the existence of the inventory file, filter the X-object-manifest metadata for object attributes to determine the inventory file;
2. The listing operation of the container. In order to hide the segmentation process for users, how can the listing operation be "clean "? Simulate swift CLI to create a "container _ segments" container for each container, and restrict users from creating/accessing the container ending with "_ segments?
3. Based on 2, think of object version management, similar to creating "container _ versions" historical version containers for containers with versions, and restrict users from creating/accessing containers at the end of "_ versions?
4. Objects can be uploaded in parallel. Can they be downloaded in parallel?
5. The update operation of a large object can detect each segment of the object and only update the changed segment?

These are all work points that can be done. The first three constraints on the API layer are well implemented. The second two should weigh the implementation mechanism of swift and perform some tests for verification.

 

Update

In order to give you more space for thinking, I will not directly give the current idea below each of the above questions. The update here is = D:

1. About object type definition.

You do need to add oneX-object-manifestAttribute. Although we want to be transparent to SDK users during the segmentation process, we do not rule out that people also have the right to know the real identity of a list file. A test is conducted to obtain details in JSON format when listing is performed on objects in the iner. The following content is returned:

[
{"Hash": "hash", "last_modified": "2013-03-21t13: 51: 18.917260", "bytes": 1, "name": "1", "content_type ": "Application/X-WWW-form-urlencoded" },{ "Hash": "c81e728d9d4c2f636f067f89cc14862c", "last_modified": "2013-03-21t13: 51: 41.530770", "bytes": 1, "name": "2", "content_type ": "Application/X-WWW-form-urlencoded" },{ "Hash": "eccbc87e4b5ce2fe28308fd9f2a7baf3", "last_modified": "2013-03-21t13: 51: 59.653640", "bytes": 1, "name": "3", "content_type ": "Application/X-WWW-form-urlencoded"},{  "Hash": "comment", "last_modified": "2013-03-21t13: 55: 33.353110", "bytes": 0, "name": "myobject", "content_type ": "Application/X-WWW-form-urlencoded"  },{ "Hash": "comment", "last_modified": "2013-03-21t13: 54: 19.956880", "bytes": 1, "name": "myobject/1", "content_type ": "Application/X-WWW-form-urlencoded" },{ "Hash": "comment", "last_modified": "2013-03-21t13: 54: 32.005000", "bytes": 1, "name": "myobject/2", "content_type ": "Application/X-WWW-form-urlencoded" },{ "Hash": "comment", "last_modified": "2013-03-21t13: 54: 42.420390", "bytes": 1, "name": "myobject/3", "content_type ": "Application/X-WWW-form-urlencoded" },{  "Hash": "comment", "last_modified": "2013-03-21t13: 27: 49.747290", "bytes": 0, "name": "repost_backup2. SQL", "content_type ": "Application/X-SQL"  },{ "Hash": "comment", "last_modified": "2013-03-21t13: 16: 14.842310", "bytes": 10, "name": "testfile", "content_type ": "Application/octet-stream "}
]

The two objects marked with red are actually list files, but there is no difference in the list with other common objects except the length is 0. But you need to know that a common object can also be 0 in length. SoX-object-manifestIt is rarely used here.

However, if you execute the head or get operation on the inventory file, we can see thatX-object-manifestAnd the object length is no longer 0, but the sum of segmented objects, that is, the real length of a large object. Therefore, the application of adding the manifest attribute to the object is to "execute the head or get operation on the inventory file ".

2. The listing operation of the container.

Swift CLI has a good way to separate a segmentation object into another container, which can avoid many strange segmentation objects during the listing of the target container (the attributes of the segmentation object are no different from those of common objects, so it is difficult to filter ).

However, the containers created by swift CLI are no different from those created by Common Containers. Therefore, when listing is performed on the account, some xxx_segments containers are added, which is not good.

One idea is to restrict the names of user containers. names ending with "_ segments" cannot be used for containers;

Another idea is to keep oneX-container-meta-segmentProperties, similarProgramming LanguageThe reserved word in. The attribute value is the name of the container where the list file of the multipart object is located. (when a container shares one container, you can set other values. In short, you need this attribute to make a difference ). In this way, the SDK layer can filter based on X-container-meta-segment to hide the segment object during account listing.

However, there are still problems with the above two filtering improvements based on the API layer, that is, the number of containers displayed during the head operation on the account still includes the number of segmented containers, therefore, if we want to achieve "Transparency" perfectly, we need to inevitably move the source code, and the distribution of containers on the ring is also affected.

There is still no perfect solution in the sense, so weigh it as needed, or simply give up the idea of "Transparency", so that SDK users can also make some business constraints on their own.