A simple algorithm for compressing strings in C language summary _c language

In an application, it is often necessary to compress a string into an integer, or a string hash. For example, the following questions:
(1) The search engine will record all the retrieval strings used by the user each time through the log file, the length of each query string is 1-255 bytes. Please find the most popular 10 search strings.
(2) There is a 1G size of a file, inside each row is a word, the word size does not exceed 16 bytes, memory limit size is 1M. Returns the 100 words with the highest frequency.
(3) There are 10 files, 1G per file, each row of each file is stored in the user's query, each file query may be repeated. You are asked to sort by the frequency of query.
(4) Given a, b two files, each store 5 billion URLs, each URL accounted for 64 bytes, the memory limit is 4G, let you find a, b file common URL.
(5) A text file, about 10,000 lines, one word per line, asking to count the first 10 words that appear most frequently.

All of these questions need to compress the string into an integer, or hash it to an integer M. Then, for example, m%16, you can place the string in a file numbered m%16, and the same string must be in the same file. Through this processing, you can divide a large file equivalent into several small files, and for small files, you can use the normal method of processing, sorting, hash_map and so on. Finally, the solution of the original problem can be obtained by synthesizing the processing results of these small files.
Some algorithms for string compression are described below.

Method 1: The simplest is to add all the characters together, the code is as follows:

unsigned long hashstring (const char *pstring, unsigned long tablesize)
{
 unsigned long hashvalue = 0;
 while (*pstring)
    hashvalue + = *pstring++;
 Return HashValue% tablesize
}

Analysis: If the length of the string is limited, and the hash table is larger, the waste is relatively large. For example, if the string is up to 16 bytes in length, then only the previous 16*127=2032 of the hash table is used. If the hash table contains 2729 items, then 2032 items will not be available.

Method 2: The last calculated hash value left 5 digits (Times 32), and the current keyword addition, can get a better uniform distribution effect.

unsigned long hashstring (const char *pstring,unsigned long tablesize)
{
 unsigned long hashvalue = 0;
 while (*pstring)
 HashValue = (hashvalue << 5) + *pstring++;
 Return HashValue% tablesize
}

Analysis: This method needs to traverse the entire string, if the string is relatively large, less efficient.

Method 3: using Huffman algorithm, assuming that only 0-9 of these 10 characters composed of strings, we use Huffman algorithm, directly to see examples:

#define SIZE int freq[size]; 
String Code[size]; 
string Word; 
 struct Node {int id; 
 int freq; 
 Node *left; 
 Node *right; 
 Node (int freq_in): ID ( -1), Freq (freq_in) {left = right = NULL; 
} 
}; 
 struct Nodeless {bool operator () (const node *a, const node *b) Const {return A->freq < b->freq; 
 
} 
}; 
 void Init () {for (int i = 0; i < Size; ++i) freq[i] = 0; 
for (int i = 0; i < word.size (); ++i) ++freq[word[i]; 
 } void Dfs (Node *root, string res) {if (root->id >= 0) Code[root->id] = res; 
  else {if (NULL!= root->left) Dfs (root->left, res+ "0"); 
 if (NULL!= root->right) Dfs (Root->right, res+ "1"); 
 } void Deletenodes (Node *root) {if (NULL = root) return; 
 if (null = = Root->left && NULL = = root->right) delete root; 
  else {deletenodes (root->left); 
  Deletenodes (Root->right); 
 Delete root; } void Buildtree () {priority_queue<node*, vectoR<node*&gt, nodeless> nodes; 
  The case for (int i = 0; i < Size; ++i) {//0 = = Freq[i] does not handle Node *newnode = new Node (freq[i)); 
  Newnode->id = i; 
 Nodes.push (NewNode); 
  while (Nodes.size () > 1) {Node *left = Nodes.top (); 
  Nodes.pop (); 
  Node *right = Nodes.top (); 
  Nodes.pop (); 
    Node *newnode = new node (left->freq + right->freq); 
    Newnode->left = left; 
    Newnode->right = right; 
 Nodes.push (NewNode); 
 Node *root = Nodes.top (); 
 DFS (root, String ("")); 
Deletenodes (root); 
 }