Summarization and analysis of massive data processing method of C + + algorithm _c language

The techniques commonly used in mass data processing
1. Bloom filtering
The basic bloom filtering supports fast insertion and lookup operations, and is a hash table technique. The basic data structure is very simple, the capacity of the bit array of M, K hash function, the input of n elements stored in the array.
Each time a new element is inserted, the K-hash of the element is calculated first, and the bit array corresponds to the hash value position of 1. When looking for an element, the same calculation of K hash value, and then query to see whether the corresponding bit array in the K-bit is 1, is to determine the existence of elements.
The basic bloom filtering algorithm can be used for fast weighing operations that allow errors. The intersection of the set, and the calculation of the set.
Bloom filtering has an improved version counting Bloom filtering can support the deletion of data, countering bloom filtering compared to the basic bloom filtering, The values of each bit in the bit array are expanded into multiple digits, and the basic bloom filtering is represented by 1bit. When inserting an element, all K bits are added to 1, minus 1 when deleted, and if k values are greater than 0 when they are found, they are determined to exist. A very important parameter in CBF is the number of digits per bit. It can be proved by theory that the number of digits is usually 4 enough to support the insertion of the same data 16 times.
bitmap can be seen as a special case of bloom filtering.
2. Hash Table Technology
D-left hash Hash Table load balancing technology. The hash table is divided into D segment, the D hash function is designed, and the load is selected to store the data. Find the D hash value and find it in section D, respectively.
Often used for statistical times.
3. Heap Technology
The heap has two typical applications:
Multi-route Merge sort
Ask TOPK
In the case of a multiple merge sort, the smallest heap is used in descending order with the largest heap, sorted in ascending order.
TOPK, when the maximum topk, with the smallest heap, topk minimum when the maximum heap. For maximum TOPK, use the minimum heap heap to maintain k values, when the newly scanned value is greater than the heap top element, the heap top element is deleted, and the new value is inserted. This scan the data, can be obtained TOPK maximum.
4. Double Barrel (multi-layer barrel) design
Hash table technology is a direct addr technology, but when the data range is widely distributed, and the data volume is very large, the use of the hash table directly direct addr technology is not, this can be used in multi-layer hash technology. The original data range is divided into small segments, each memory can be loaded, and the direct addr table technique can be used within the segment. You can quickly navigate to small segments with multiple tiers.