11. Thoroughly parse the Hash table algorithm from start to end

11. Thoroughly parse the Hash table algorithm from start to end

Author: July, wuliming, pkuoliver
Source:Http://blog.csdn.net/v_JULY_v.
Note: This article is divided into three parts,
The first part is a detailed explanation of the Top K algorithm of Baidu interview questions; the second part is a detailed description of the Hash table algorithm; the third part is to build the fastest Hash table algorithm.
------------------------------------

 

Part 1: Explanation of the Top K Algorithm
Problem description
Baidu interview questions:
The search engine records all the search strings used for each search using log files. The length of each query string is 1-bytes.
Suppose there are currently 10 million records (these query strings have a relatively high degree of repetition, although the total number is 10 million, but if the repetition is not removed, there will be no more than 3 million records. The higher the repetition of a query string, the more users query it, that is, the more popular it is .), Please count the top 10 query strings. The memory required cannot exceed 1 GB.

Required knowledge:
What is a hash table?
A Hash table (also called a Hash table) is a data structure that is directly accessed based on the Key value. That is to say, It maps the key value to a location in the table to access records to speed up the search. This ing function is called a hash function, and the array storing records is called a hash function.

The hash table method is actually very simple, that is, to convert the Key into an integer using a fixed algorithm function called a hash function, and then perform the remainder operation on the array length, the remainder result is used as the subscript of the array, and the value is stored in the array space with the number as the base object.
When a hash table is used for query, the hash function is used again to convert the key to the corresponding array subscript and locate the space to obtain the value, the array positioning performance can be fully utilized for data locating.(The second and third parts of the article will elaborate on the Hash table).

Problem Analysis:
To count the most popular queries, you must first count the number of times each Query appears, and then find the Top 10 based on the statistical results. Therefore, we can design the algorithm in two steps based on this idea.
That is, the solution to this problem is as follows:Two steps:

Step 1: Query statistics
Query statistics are available in the following two methods:
1. Direct sorting
The first algorithm we come up with is sorting. First, we sort all the queries in this log, and then traverse the sorted Query to count the number of times each Query appears.

But there is a clear requirement in the question, that is, the memory cannot exceed 1 GB, there are 10 million records, each record is 225 bytes, it is obvious that it occupies 2 to 55 GB memory, this condition does not meet the requirements.

Let's recall the content in the Data Structure course. When the data volume is large and the memory cannot be loaded, we can sort it by external sorting. Here we can sort it by merging, because Merge Sorting has a better time complexity O (NlgN ).

After sorting, we traverse the sorted Query file, count the number of times each Query appears, and write it into the file again.

According to a comprehensive analysis, the time complexity of sorting is O (NlgN), and the time complexity of traversal is O (N). Therefore, the overall time complexity of this algorithm is O (N + NlgN) = O (NlgN ).

2. Hash Table Method
In the 1st methods, we used the sorting method to count the number of times each Query appears. The time complexity is NlgN. Can we have a better way to store the data, while the time complexity is lower?

The question shows that although there are 10 million queries, but because of the high repetition, there are actually only 3 million queries, each of which is bytes, we can consider putting them into the memory, now, we only need a suitable data structure. Here, Hash Table is definitely our priority, because the query speed of Hash Table is very fast, almost O (1) time complexity.

Then, our algorithm has: maintain a HashTable with the Key as the Query string and the Value as the number of occurrences of the Query. Read a Query each time. If the string is not in the Table, add the string and set the Value to 1. If the string is in Table, add one To the count of the string. In the endO (N)The processing of the massive data is completed within the time complexity.

Compared with algorithm 1, this method increases the time complexity by an order of magnitude, which is O (N), but not only the optimization of time complexity. This method only requires one IO data file, algorithm 1 has a large number of I/O operations. Therefore, algorithm 2 has better operability than algorithm 1 in Engineering.

Step 2: Find the Top 10
Algorithm 1: normal sorting
I don't want to go into details about sorting algorithms. We should note that the time complexity of sorting algorithms is NlgN. In this question, there are 3 million records, 1 GB memory can be used for storage.

Algorithm 2: Partial sorting
The requirement for the question is to find the Top 10, so we do not need to sort all the queries. We only need to maintain an array of 10 sizes, and put 10 queries in initialization, sort by the statistics of each Query from large to small, and then traverse the 3 million records. Each read record is compared with the last Query of the array. If it is smaller than this Query, continue to traverse, otherwise, the last row of data in the array is eliminated and added to the current Query. Finally, after all the data is traversed, the 10 queries in this array are the top 10 we are looking.

It is not difficult to analyze. In this way, the worst time complexity of the algorithm isN * KWhere K refers to the top number.

Algorithm 3: heap
In algorithm 2, we have optimized the time complexity from NlogN to NK. I have to say this is a big improvement. But is there any better way?

Analysis: In algorithm 2, after each comparison is completed, the operation complexity is K, because the elements need to be inserted into a linear table and sequential comparison is used. Here, we note that the array is ordered. We can use the binary search method every time we look for it. This reduces the complexity of the operation to the logK. However, the problem that arises is data movement, because the number of mobile data increases. However, this algorithm is better than algorithm 2.

Based on the above analysis, do you have a data structure that can quickly search and move elements? The answer is yes, that is, heap.
With the help of the heap structure, we can search, adjust, and move logs in a time range of log magnitude. So here, our algorithm can be improved to maintain a small root heap of K (10 in this question) and traverse the Query of 3 million,Comparison with the root element.

The idea is consistent with the above two algorithms, but the algorithm is in algorithm 3. We use the minimum heap data structure to replace the array, and the time complexity of searching the target element is O (K) reduced to O (logK ).
In this way, with the heap data structure and algorithm 3, the final time complexity will be reducedN'logkCompared with algorithm 2, it has made great improvements.

 

Summary:
So far, the algorithm has completely ended. After the first step, use the Hash table to calculate the number of times each Query appears, O (N). Then, step 2, use the heap data structure to find the Top 10, N * O (logK ). Therefore, our final time complexity is:O (N) + N' * O (logK ).(N is 10 million, N is 3 million ). If you have any better algorithms, please leave a comment. The first part is complete.

 

Part 2: detailed analysis of Hash Table Algorithms

What is Hash?
Hash is usually translated as "Hash", which is also directly translated as "Hash", that is, input of any length (also called pre- ing, pre-image ), the hash algorithm is used to convert an output with a fixed length. The output is the hash value. This type of conversion is a compression ing, that is, the space of hash values is usually much smaller than the input space, and different inputs may be hashed into the same output, instead, it is impossible to uniquely determine the input value from the hash value. Simply put, a function compresses messages of any length to a fixed-length message digest.

HASH is mainly used for encryption algorithms in the information security field. It converts information of different lengths into messy 128-bit codes. These encoding values are called HASH values. it can also be said that hash is to find a ing between the data content and the data storage address.

Arrays are characterized by ease of addressing and difficulty in insertion and deletion. linked lists are characterized by difficulties in addressing and insertion and deletion. So can we combine the two features to make a data structure that is easy to address and easily inserted and deleted? The answer is yes. This is the hash table to be mentioned. There are many different implementation methods for hash tables. What I will explain next is the most commonly used method-the zipper method, we can understand it as an array of linked lists ",

The left is obviously an array. Each member of the array contains a pointer pointing to the head of a linked list. Of course, this linked list may be empty or contain many elements. We distribute elements to different linked lists based on some features of the elements. We also find the correct linked list based on these features and then find this element from the linked list.

The method for converting element features into arrays is the hash method. Of course, there are more than one hash method, which are listed below:

1. Division hash
The most intuitive method is the hash method. The formula is as follows:
Index = value % 16
All those who have learned assembly know that the modulus is actually obtained through a division operation, so it is called the Division hash method ".

2. Square hash Method
Index is a very frequent operation, while multiplication is much more time-saving than Division (for the current CPU, we cannot feel it ), so we want to replace division with multiplication and a displacement operation. Formula:
Index = (value * value)> 28(Right Shift, divided by 2 ^ 28. Note: shift left to enlarge, Which is multiplication. Shift right to a smaller value, which is division.)
If the value distribution is relatively uniform, this method can produce good results, but the index calculated by the values of each element in the graph I drew above is 0-very failed. Maybe you still have a problem. If the value is large, will the value * value not overflow? The answer is yes, but we do not care about overflow in this multiplication, because we are not trying to get the multiplication result, but to get the index.

3. Fibonacci hash

The disadvantages of the square hash method are obvious, so can we find an ideal multiplier instead of using the value itself as the multiplier? The answer is yes.

1. For a 16-digit integer, the multiplier is 40503.
2. For a 32-bit integer, the multiplier is 2654435769.
3. For a 64-bit integer, the multiplier is 11400714819323198485.

How are these "ideal multiplier" obtained? This is related to a rule called the golden division rule, and the most classic expression that describes the golden division rule is undoubtedly the famous Fibonacci series, that is, the sequence in this form: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89,144,233,377,610,987,159 7, 2584,418 1, 6765,109 46 ,.... In addition, the Fibonacci sequence value is surprisingly consistent with the ratio of the orbital radius of the eight planets in the solar system.

For our common 32-bit integer, the formula is as follows:
Index = (value * 2654435769)> 28

If the Fibonacci hash is used, the figure above becomes like this:
Obviously, it is much better to use the Fibonacci hash method after adjustment than the original scatter method.

Applicability
The basic data structure to be deleted, which usually requires a total amount of data to be stored in the memory.

Basic principles and key points
Hash function Selection, for strings, integers, sorting, specific hash method.
For collision processing, one is open hashing, also known as the zipper method, and the other is closed hashing, also known as the open address method and opened addressing.

Extension
D in d-left hashing refers to multiple meanings. Let's first simplify this problem and take a look at 2-left hashing. 2-left hashing refers to dividing a hash table into two halves of the same length, namely T1 and T2, and configuring a hash function, h1 and h2 for T1 and T2 respectively. When a new key is stored, two hash functions are used for calculation to obtain the addresses h1 [key] and h2 [key]. In this case, you need to check the h1 [key] location in T1 and the h2 [key] location in T2. Which location has been stored (with collision) and there are many keys, store the new key in a location with less load. If the two sides are as many as one, for example, if both locations are empty or both of them store a key, the new key is stored in the T1 subtable on the left, and 2-left is also stored. When searching for a key, you must perform two hashes and query both locations at the same time.

Problematic instances (massive data processing)
We know that hash tables are widely used in massive data processing. For details, refer to another Baidu interview question:
Question: extract the IP address with the most visits to Baidu on a certain day from massive log data.
Solution: the number of IP addresses is still limited. A maximum of 2 ^ 32 ip addresses are allowed. Therefore, you can use hash to directly store IP addresses in the memory for statistics.

 

Part 3: fastest Hash Table Algorithm

Next, let's analyze the next fastest Hasb table algorithm.
Let's start with a simple question step by step: there is a huge string array, and then you will be given a separate string, so that you can find out whether the string exists in this array and find it, what do you do? There is one method that is the easiest, honestly from the beginning to the end, one by one comparison, until it is found, I think anyone who has learned programming can make such a program, but if a programmer gives such a program to a user, I can only comment it without words. Maybe it can really work,... this is the only way to do this.

The most suitable algorithm is to use HashTable (Hash table). First, we will introduce the basic knowledge. The so-called Hash is generally an integer, you can compress a string into an integer. Of course, in any case, a 32-bit integer cannot correspond to a string, but in the program, the Hash values calculated by the two strings may be very small, next let's look at the Hash algorithm in MPQ:

Function 1,The following function generates a cryptTable [0x500] with a length of 0x1280 (in hexadecimal number: 500).

Void prepareCryptTable ()
{
Unsigned long seed = 0x00100001, index1 = 0, index2 = 0, I;
 
For (index1 = 0; index1 <0x100; index1 ++)
{
For (index2 = index1, I = 0; I <5; I ++, index2 + = 0x100)
{
Unsigned long temp1, temp2;
 
Seed = (seed * 125 + 3) % 0x2AAAAB;
Temp1 = (seed & 0 xFFFF) <0x10;
 
Seed = (seed * 125 + 3) % 0x2AAAAB;
Temp2 = (seed & 0 xFFFF );
 
CryptTable [index2] = (temp1 | temp2 );
}
}
}

Function 2,The following function calculates the hash value of the lpszFileName string, where dwHashType is of the hash type.Function 3The GetHashTablePos function calls this function 2. The values can be 0, 1, and 2. This function returns the hash value of the lpszFileName string:

Unsigned longHashString(Char * lpszFileName, unsigned long dwHashType)
{
Unsigned char * key = (unsigned char *) lpszFileName;
Unsigned long seed1 = 0x7FED7FED;
Unsigned long seed2 = 0 xEEEEEEEE;
Int ch;
 
While (* key! = 0)
{
Ch = toupper (* key ++ );
 
Seed1 = cryptTable [(dwHashType <8) + ch] ^ (seed1 + seed2 );
Seed2 = ch + seed1 + seed2 + (seed2 <5) + 3;
}
Return seed1;
}

This algorithm of Blizzard is very efficient, called "One-Way Hash" (A one-way hash is a an algorithm that is constructed in such a way that deriving the original string (set of strings, actually) is always Ally impossible ). For example, the result of the string "unitneutralacritter. grp" obtained through this algorithm is 0xA26067F3.

Is it possible to improve the first algorithm by comparing the Hash values of strings one by one? The answer is: it is far from enough. If you want to get the fastest algorithm, you cannot compare the values one by one, usually constructHash table(Hash Table) to solve the problem, the Hash Table is a large array, the size of this array is defined according to the requirements of the program, such as 1024, each Hash value through the modulo operation (mod) corresponds to a position in the array. In this way, you can obtain the final result by comparing the position corresponding to the hash value of this string? Yes, it is the fastest O (1). Now let's take a closer look at this algorithm:

Typedef struct
{
Int nHashA;
Int nHashB;
Char bExists;
......
} SOMESTRUCTRUE;
A possible struct definition?

Function 3,The following functions are used to find whether the target string exists in the Hash table. If yes, the Hash value of the string to be searched is returned. If no, return-1.

IntGetHashTablePos(Har * lpszString, SOMESTRUCTURE * lpTable)
// The string to be searched in the Hash table for lpszString. lpTable is the Hash table that stores the string Hash value.
{
Int nHash = HashString (lpszString); // call function 2 and return the Hash value of the string lpszString to be searched.
Int nHashPos = nHash % nTableSize;
 
If (lpTable [nHashPos]. bExists &&! Strcmp (lpTable [nHashPos]. pString, lpszString ))
{// If the Hash value found exists in the table and the string to be searched is the same as the string at the corresponding position in the table,
Return nHashPos; // return the Hash value found after function 2 is called.
}
Else
{
Return-1;
}
}

 
Seeing this, I think everyone is thinking about a very serious problem: "What if the two strings have the same location in the hash table ?", After all, the size of an array is limited, which is highly probable. There are many ways to solve this problem. The first thing I think of is to use"Linked List", Thanks to the data structure I learned in the university for teaching this magic weapon. Many algorithms I have encountered can be converted into linked lists to solve this problem, as long as a chain table is attached to each entry of the hash table, it is OK to save all corresponding strings. This seems to have a perfect ending. If you leave the problem to me alone, then I may have to define the data structure and write the code.

However, Blizzard programmers use more sophisticated methods. The basic principle is: they use not a hash value but a hash value in the hash table.Three hash valuesTo verify the string.

 

MPQ uses a file name hash table to track all internal files. However, the format of this table is somewhat different from that of a normal hash table. First, it does not use Hash as the subscript and stores the actual file name in the table for verification. In fact, it does not store the file name at all. Instead, three different hashes are used: a subscript for the hash table, and two for verification. The two verification hashing Replace the actual file name.
Of course, two different file names will be hashed to three identical hashes. However, the average probability of this situation is: 1: 18889465931478580854784. This probability should be small enough for anyone. Now back to the data structure, the hash table used by Blizzard does not use the linked list, but uses the "extend" method to solve the problem. Let's look at this algorithm:

Function 4,LpszString is the string to be searched in the hash table; lpTable is the hash table storing the string hash value; nTableSize is the length of the hash table:

IntGetHashTablePos(Char * lpszString, MPQHASHTABLE * lpTable, int nTableSize)
{
Const int HASH_OFFSET = 0, HASH_A = 1, HASH_ B = 2;
 
Int nHash = HashString (lpszString, HASH_OFFSET );
Int nHashA = HashString (lpszString, HASH_A );
Int nHashB = HashString (lpszString, HASH_ B );
Int nHashStart = nHash % nTableSize;
Int nHashPos = nHashStart;
 
While (lpTable [nHashPos]. bExists)
{
/* If you only judge that this string exists in the table, you can compare the two hash values.
* The strings in the struct are compared. Will this speed up the operation? Reduce the space occupied by hash tables? This
* Where is the method generally used? */
If (lpTable [nHashPos]. nHashA = nHashA
& LpTable [nHashPos]. nHashB = nHashB)
{
Return nHashPos;
}
Else
{
NHashPos = (nHashPos + 1) % nTableSize;
}
 
If (nHashPos = nHashStart)
Break;
}
Return-1;
}

Explanation of the above procedure:

1. Calculate the three hash values of the string (one is used to determine the position, and the other two are used for verification)
2. view the position in the hash table
3. is the position in the hash table empty? If it is null, the string does not exist and-1 is returned.
4. If yes, check whether the other two Hash values match. If yes, the string is found and its Hash value is returned.
5. Move to the next position. If you have already moved to the end of the table, the query continues from the beginning of the table.
6. check whether it is back to the original position. If yes, the returned result is not found.
7. Return to 3

OK. This is the fastest Hash table algorithm mentioned in this article. What? Not fast enough? : D. Thank you for your criticism.

--------------------------------------------
1. A simple hash function:

/* The key is a string, and nTableLength is the length of the hash table.
* The hash value distribution obtained by this function is relatively uniform */
Unsigned long getHashIndex (const char * key, int nTableLength)
{
Unsigned long nHash = 0;

While (* key)
{
NHash = (nHash <5) + nHash + * key ++;
}

Return (nHash % nTableLength );
}

 

Supplement 2: a complete test procedure:
The array of the hash table is fixed length. If it is too large, it will be wasted. If it is too small, it will not reflect the efficiency. The proper array size is the key to the performance of hash tables. The size of a hash table is preferably a prime number. Of course, there will be different hash table sizes based on different data volumes. For applications with a small amount of data, the best design is to use a dynamically variable-size hash table. If you find that the size of the hash table is too small, for example, if the element is twice the size of the hash table, we need to expand the size of the hash table, which is generally doubled.

The following are possible values of the hash table size:

17, 37, 79,163,331,
673,136 1, 2729,471,109 49,
21911,438 53, 87719,175 447, 350899,
701819,140, 2807303,561, 11229331,
22458671,449 17381, 89834777,179 669557, 359339171,
718678369,143 7356741, 2147483647

 

The complete source code of the program is as follows, which has been tested in linux:# Include <stdio. h> <br/> // crytTable [] stores some data that will be used in the HashString function, initialize in the prepareCryptTable <br/> // function <br/> unsigned long cryptTable [0x500]; </p> <p> // The following function generates a value with a length of 0x500 (in combination with 10 hexadecimal numbers: 1280) cryptTable [0x500] <br/> void prepareCryptTable () <br/> {<br/> unsigned long seed = 0x00100001, index1 = 0, index2 = 0, i; </p> <p> for (index1 = 0; index1 <0x100; index1 ++) <br/>{< br/> for (index2 = index1, I = 0; I <5; I ++, Index2 + = 0x100) <br/>{< br/> unsigned long temp1, temp2; </p> <p> seed = (seed * 125 + 3) % 0x2AAAAB; <br/> temp1 = (seed & 0 xFFFF) <0x10; </p> <p> seed = (seed * 125 + 3) % 0x2AAAAB; <br/> temp2 = (seed & 0 xFFFF ); </p> <p> cryptTable [index2] = (temp1 | temp2 ); <br/>}</p> <p> // The following function computes the hash value of the lpszFileName string, where dwHashType is of the hash type, <br/> // call this function in the GetHashTablePos function below. The value can be 0, 1, or 2. Count <br/> // return the hash value of the lpszFileName string; <br/> unsigned long HashString (char * lpszFileName, unsigned long dwHashType) <br/>{< br/> unsigned char * key = (unsigned char *) lpszFileName; <br/> unsigned long seed1 = 0x7FED7FED; <br/> unsigned long seed2 = 0 xeeeeeeeeee; <br/> int ch; </p> <p> while (* key! = 0) <br/>{< br/> ch = toupper (* key ++); </p> <p> seed1 = cryptTable [(dwHashType <8) + ch] ^ (seed1 + seed2); <br/> seed2 = ch + seed1 + seed2 + (seed2 <5) + 3; <br/>}< br/> return seed1; <br/>}</p> <p> // test the three hash values of argv [1] In main: <br/> //. /hash "arr \ units. dat "<br/> //. /hash "unit \ neutral \ acritter. grp "<br/> int main (int argc, char ** argv) <br/>{< br/> unsigned long ulHashValue; <br/> int I = 0; </ P> <p> if (argc! = 2) <br/>{< br/> printf ("please input two arguments \ n"); <br/> return-1; <br/>}</p> <p>/* initialization array: crytTable [0x500] */<br/> prepareCryptTable (); </p> <p>/* print the values in the crytTable array [0x500] */<br/> for (; I <0x500; I ++) <br/>{< br/> if (I % 10 = 0) <br/>{< br/> printf ("\ n "); <br/>}</p> <p> printf ("%-12X", cryptTable [I]); <br/>}</p> <p> ulHashValue = HashString (argv [1], 0 ); <br/> printf ("\ n ---- % X ---- \ n", ulHashValue); </p> <p> ulHashValue = HashString (argv [1], 1 ); <br/> printf ("---- % X ---- \ n", ulHashValue); </p> <p> ulHashValue = HashString (argv [1], 2 ); <br/> printf ("---- % X ---- \ n", ulHashValue); </p> <p> return 0; <br/>}

Thanks:
1. http://blog.redfox66.com /.
2. http://blog.csdn.net/wuliming_ SC /. .

Copyright, protected by law. Reprinted. Please enter the source in the form of a link.