Github inventory object Algorithm

Github inventory object Algorithm

$ Git clone https://github.com/torvalds/linux
Cloning into 'linux '...
Remote: Counting objects: 4350078, done.
Remote: Compressing objects: 100% (4677/4677), done.
Grouping objects: 4% (191786/4350078), 78.19 MiB | 8.70 MiB/s

The remote code library contains a total of 4350078 objects that need to be cloned.

This is called counting objects. Github needs to calculate the total number of objects to be cloned in real time.

This process is very slow. According to Github's disclosure, it takes 8 minutes to check a large library such as Linux kernel! That is, issuegit cloneAfter the command, it will wait for eight minutes before the real data transmission starts. Of course, this is intolerable. The Github team has been trying to solve this problem.

Later, they finally discovered a new algorithm, which now takes only 3 milliseconds to count once!

To understand this algorithm, you must first know what is a Git object. Simply put, objects are files, and the most important objects are three types.

Snapshot object (Commit) Directory object (Directory) File object (File)

Each time code is submitted, a commit object is generated with the corresponding name of the current "directory object. The "directory object" stores the subdirectories and file information contained in the Code root directory. Each subdirectory is another "directory object", and each file is a "file object", which contains the specific file content.

Therefore, "check object" is to check various commit, directories, files, and so on.git cloneAndgit fetchYou need to check the objects for all operations, because you need to know which object files are downloaded.

The original algorithm for checking objects is as follows.

1. Lists the latest commit of all local branches.
2. Lists the latest commit of all remote branches.
3. If the two are different, it means that the branch changes.
4. For each changed commit, the subdirectories and files of the change are checked.
5. Trace back to the parent node of the current commit, repeat Step 4 until the local and remote history is consistent.
6. Add all objects to be changed

The above process shows that "check object" is a file traversal algorithm, and the changed objects are checked one by one, which means a large number of file read operations. This process is very slow for Large Code libraries.

The new algorithm developed by the Github team is to create a Bitmap index, which generates a binary value for each commit.

Open the local Github Repository.git/objects/pack/Directory, you will see an index file and a data file, which is Bitmap. Simply put, these two files index all objects in the current code base and use a binary value to represent these objects. How many objects are there, and how many bits are there in the binary value. The Nth bit indicates the nth object in the data file.

Each commit has a binary value, indicating all objects contained in the current snapshot. The binary bits corresponding to these objects are 1, and the other binary bits are 0.

The advantage of this is that you do not need to read the commit object. As long as you read the binary value, you will know which nodes are included in the current commit. Even better, as long as the two binary values perform an XOR operation, they will know which bits (I .e. which objects) have changed. Moreover, because new objects are always added to the end of existing binary bits, you only need to read the extra bits to know what more objects the current commit has than the previous commit.

In this way, the "check object" becomes a comparison operation of binary values, so the speed is extremely fast. For more information, see the official document "Bitmap interpretation" and "Bitmap format".

Currently, this algorithm has been deployed in the production environment of Github, and users no longer have to wait to check objects. In addition, the Github team also merged it into Git, which means that Bitmap can be used for all Git implementations from now on, so there will certainly be more interesting usage in the future.

GitHub Tutorials:

Create a personal technical blog via GitHub

GitHub tutorials

Git tag management details

Git branch management

Git remote repository details

Git local Repository (Repository) Details

Git server setup and Client installation

Git Overview

Share practical GitHub tutorials

GitHub details: click here
GitHub: click here

This article permanently updates the link address: