Mass data processing-bitmap (BITMAP)

It's not hard to understand the word mass data. Mainly for a given amount of data is particularly large, occupy a particularly large memory situation. So what's the relationship to the bitmap? Look at the following example of Tencent's massive data.

For example: 4 billion unsigned integers that are not repeated are not ordered. Give an unsigned integer how to quickly determine whether a number is in these 4 billion numbers.

For this problem, we gave 4 billion unsigned integers, an integer of 4 bytes, that is 40*4=160 billion bytes, probably 16G of memory. Obviously not available on memory. So how are we going to find it. Since it is not repeated, it means that the integer either does not appear or appears once. The maximum value of an integer is more than 4.2 billion, that is, 2^32. At this point we can use each digit to indicate that the number exists or does not exist. If the 32-bit is a number, the original 16G data using bitmaps can save 500M of space. Perhaps we have just learned a hash table, using the method of accessing the address to quickly find out the corresponding value of the address. This is also the case, using the new hash table method to solve the mass data--- bitmap .

So here's the problem. What is a bitmap?

We use each digit to mark the state of existence of this number, set to 0 (nonexistent) and 1 (present);

The basic structure of a bitmap:

is a vector array of size_t types;

Vector<size_t> _array;

Basic functions for bitmaps:

For determining whether an unsigned integer exists in these 4 billion numbers.

(1) need to deposit these 4 billion numbers, use set will correspond 4 billion position 1;

(2) Use test to determine whether a bit is 0 or 1;

Note: The bitmap only considers the integer type

Implementation code for bitmaps: (vs2013)

#pragma once #include <iostream> using namespace std; #include <vector>//bitmap every bit of 0,1 flag this number exists or does not exist in the state class BitMap {public:bitmap (size_t size = 1024) {_array.resize (S)
	IZE/32+1);
		~bitmap () {} public://Set the status of this number to 1 void Set (const size_t& value) {size_t index = value>>5;
		
		size_t bit = value% 32;
	_array[index] |= (1<<bit);
		///Set this number to a nonexistent state of 0 void Reset (const size_t& value) {size_t index = value>>5;

		size_t bit = value% 32;
	_array[index] &= (~ (1<<bit));
		///test whether a number has occurred bool Test (const size_t& value) {size_t index = value>>5;

		size_t bit = value% 32;
	Return (_array[index] & (1<<bit));
} private:vector<size_t> _array;


};   void Bitmaptest () {BitMap BM (size_t (-1)); Maximum value for integers represented under 64-bit systems BM.
	Set (10); Bm.
	Set (100); Bm.
	Set (20); Bm.
	Set (500); COUT&LT;&LT;BM.
	Test <<endl; COUT&LT;&LT;BM.
	Test <<endl; COUT&LT;&LT;BM.
	Test <<endl; COUT&LT;&LT;BM. Test(<<endl;) }

Run Result: