Implementation of Bitmap for Massive Data Processing

Bitmap bitmap is often used to deal with massive data, such as 0.3 billion and 0.7 billion QQ re-query problems, telephone number de-duplication problems, can be processed using bitmap.

The bitmap method is simple, that is, to apply for a table composed of bits, you can set it to 0 or 1 at the corresponding position, so as to quickly achieve quick search without wasting space. The bitmap method is described in more detail on the Internet. This article provides a simple implementation of the bitmap method.

typedef char byte8;typedef byte8 * bitMap;int initBitMap(byte8 * & bitMap) {    bitMap = (byte8 *)malloc(1 << 29); //bitMap size: 2^29 * sizeof(byte8) = 2^32;    memset(bitMap, 0, 1 << 29);    return 0;}int setBitTrue(byte8 * bitMap, unsigned int loc) {    unsigned int segLength = sizeof(byte8) * 8;    bitMap[loc / segLength] |= (1 << (segLength - 1 - loc % segLength));    return 0;}int isTrue(byte8 * bitMap, unsigned int loc) {    unsigned int segLength = sizeof(byte8) * 8;    int ret = bitMap[loc / segLength] & (1 << (segLength - 1 - loc % segLength));    return ret > 0 ? 1 : 0;}int destroyBitMap(byte8 * bitMap) {    if (bitMap != NULL) {        free(bitMap);    }    bitMap == NULL;    return 1;}

Notes:

1. Calculate the size and range of the bitmap. The above 2 ^ 32 bits, subscript range: 0-2 ^ 32-1.

2. determine the basic storage method. The above uses a char with 8 bits

3. Don't forget to release space

Example: calculate the maximum number of consecutive numbers for an int integer sequence (hundreds of millions. For example: 100, 4,200, 1, 3, 2, the maximum number of consecutive numbers is 4 ).

Int longestconsecutive (vector <int> & num) {bitmap bm; initbitmap (BM); unsigned int I = 0; for (I = 0; I <num. size (); I ++) {unsigned int loc = num [I] + (unsigned INT) 1 <31); setbittrue (BM, Loc );} unsigned int max = 0; unsigned int COUNT = 0; for (I = 0; I <0 xffffffff; I ++) {If (istrue (BM, I )) {count ++; max = count> Max? Count: Max;} else {COUNT = 0 ;}} if (istrue (BM, I) {// The Last 0 xffffffff count ++; max = count> Max? Count: Max;} destroybitmap (BM); Return Max;} int main () {int arr [] = {100, 4,200, 1, 3, 2 }; vector <int> num; num. assign (ARR, arr + sizeof (ARR)/sizeof (INT); printf ("% u \ n", longestconsecutive (Num); Return 0 ;}

Note:

1. process negative numbers and move them right. Int range:-2 ^ 31 ~ 2 ^ 31-1, add 2 ^ 31 to all numbers, then the range is: 0 ~ 2 ^ 32-1. It is exactly the range of the unsigned Int. Suitable for Array subscript representation.

2.2 ^ 32-1 can be expressed by 0xffffffff.

3. Since I of the unsigned int type cannot be increased to 2 ^ 32, the for loop condition is '<' rather than '<='. Therefore, it must be executed once after the loop ends.