Blog on the road-game resource compression, packaging, and patch updates

Nine years ago, I designed a resource package for Netease games and a data format for the patch package.

The purpose of the original design was to facilitate resolution and quickly locate the files in the resource package, so as to facilitate updates and save as much bandwidth as possible for each update. In the past few years, although the resource package format has been improved by the repairs and supplements of various projects, there have been no major changes in nature.

At the beginning, we directly connected the files to be packaged and attached the file index table at the end of the file. At the beginning, in order to quickly locate the file name, the file name is hashed. You can use the hash value to directly locate the file. The resource package does not store the file name information, but stores it in an additional index file. This index file is not released externally. Therefore, resolving a resource package directly cannot accurately restore the file name.

Btw and blizzard's mpq files are also processed in a similar way. Unless you guess the file name, it is difficult to restore the file name. Many mpq unpacking tools on the Internet include an additional file name list for a specific game.

Unlike many other game clients (such as Blizzard's MPQ file. In our package format, there are holes allowed between files. This is because the resource package files are relatively large. If you use traditional packaging software to operate: delete a file from the package, repackage or move internal data. When a player updates resources, there will be a large number of file IO operations. For example, when WOW or SC2 is updated, the download time of the update package usually takes up only a small part of the update time. Most of the time is spent on patching the existing resource package.

If the client is updated frequently, this will be annoying for users.

Therefore, when we delete files in the package, we did not move the data in the resource package, but left the space. If the newly added file is relatively small, the space will be reused. If it cannot be used, it will be wasted there. This is a bit like a memory management algorithm. After a long time, there will be some holes in the resource package, but it is acceptable.

At the same time, there is another way to update new resources. That is to package the files to be updated separately and store them on the user's hard disk with the same file name (different suffixes. When the game engine reads resources, it first searches in the updated resource package. This method is also used in the Quake/Doom series of Id soft.

To speed up patch updates. Our patch is not a small file in the saved resource package. Instead, repackage in incremental mode on the development machine. The patch file is actually the diff file of the entire resource package. Because of the preceding packaging scheme, the binary diff file can be very small. In particular, partial modification to some files has little impact on the entire resource package.

At the company, some later colleagues questioned this method and felt that it had little effect on reducing the patch volume. On the contrary, incremental packaging increases the time required to create patch packages. It is suggested to directly add updated small files in the patch, and then make patching on the final user Machine Based on small files.

Indeed, the binary diff function is limited. Many projects now use the text data format. A small modification will affect the diff result of the entire file. However, there are also historical reasons for the original design. Because hard disk I/O was slow 10 years ago, and westward journey also needed to achieve seamless loading of big maps during design. Therefore, the format of Map Files is specially designed. This method is suitable for modifying and updating map files. In addition, it also makes sense to update uncompressed image files.

 

Game Blog http://blog.codingnow.com/oiie/oiie/