Summary and analysis of massive data processing methods using C ++ Algorithms

Technologies frequently used in Mass Data Processing
1. Bloom Filtering
The basic Bloom Filtering supports fast insert and search operations. It is a hash table technology. The basic data structure is very simple. The size is a m-Bit Array and k hash functions store the n input elements in the in-place array.
Each time a new element is inserted, the k hash indexes of the element are calculated first, and the position of the corresponding hash value of the bit array is 1. when you look for an element, calculate k hash values first, and then query to see if all k-bit values in the corresponding bit array are 1. If yes, it is determined that the element exists.
The basic Bloom Filtering algorithm can be used to allow fast deduplication of errors. Set intersection and Union calculation.
There is an improved version of Bloom Filtering. counting bloom filtering supports data deletion operations. Compared with the basic bloom filtering, the value of each bit in the bit array is extended to multiple digits, the basic bloom filtering is expressed as 1 bit. When an element is inserted, all k-bit values are added with 1, and all values are deleted with 1. If k-bit values are greater than 0, they are regarded as exist. A very important parameter is the number of bits in each bits. It can be proved theoretically that the number of digits is generally 4 enough, and the same data can be inserted 16 times.
Bitmap can be seen as a special case of bloom filtering
2. Hash Table Technology
D-left hash table load balancing technology. Divide the hash table into d segments and design d hash functions to select a suitable segment for data storage. When searching, you need to calculate d hash values and find them in segment d respectively.
It is often used for statistics.
3. Heap Technology
Heap has two typical applications:
Multi-path Merge Sorting
Calculate the TopK
The maximum heap is used for descending sorting and the minimum heap is used for ascending sorting.
When TopK is used, the minimum heap is used when the TopK is the largest, and the maximum heap is used for the TopK. When the topK is the largest, the minimum heap is used to maintain K values. When the value of the new scan is greater than the heap top element, the heap top element is deleted and a new value is inserted. In this way, the maximum value of the topK can be obtained after the data is scanned.
4. dual-layer bucket (multi-layer bucket) Design
The hash table technology is a direct addr technology, but when the data range is too wide and the data volume is very large, the direct addr technology using the hash table will not work, the multi-layer hash technology can be used. The original data range is divided into small segments, each segment of memory can be loaded, and the direct addr table technology can be used within the segment. You can quickly locate small segments with multiple levels.