Interview-Ali-. Big Data topic-given a, b two files, each store 5 billion URLs, each URL accounted for 64 bytes, memory limit is 4G, let you find a, b file common URL?

If each URL size is 10bytes, then you can estimate the size of each file is 50gx64=320g, far greater than the memory limit of 4G, so it is not possible to fully load into the memory processing, you can use the idea of partition to solve.

STEP1: Traverse file A, hash (URL) for each URL%1000, and then according to the obtained value will be stored in the URL to 1000 small files (recorded as A0,a1,..., a999, each small file about 300M);

STEP2: Traverse file B, take and a the same way to store the URL to 1000 small files (recorded as B0,b1,..., b999);

Ingenious: After this processing, all possible identical URLs are saved in the corresponding small file (a0vsb0,a1vsb1,..., a999vsb999), the corresponding small file cannot have the same URL. Then we only ask for the same URL in the 1000-to-small file.

STEP3: When you ask for each pair of small file AI and bi in the same URL, you can store the AI URL into hash_set/hash_map. Then traverse each bi URL, see if it is in the hash_set just built, if so, then is the common URL, stored in the file can be.

The sketch is as follows (the left decomposition A, right decomposition B, the middle solves the same URL):

2. There is a 1G size of a file, each line is a word, the size of the word does not exceed 16 bytes, the memory limit size is 1M, requires the return of the highest frequency of 100 words.

STEP1: In sequential read file, for each word X, take hash (x)%5000, and then the value is stored in 5,000 small files (recorded as F0,f1,..., f4999), so that each file is about 200k, if some of the files over 1M size, You can continue to follow a similar method, until the decomposition of the small file size is not more than 1M;

STEP2: For each small file, count the occurrences of each file and the corresponding frequency (can be used trie tree/hash_map, etc.), and take out the most frequent 100 words (can be used with 100 nodes of the smallest heap), and the 100 words and the corresponding frequency deposited into the file, In this way, 5,000 documents were obtained;

STEP3: Merge These 5,000 files (similar to merge sort);

The sketches are as follows (splitting large problems, solving small problems, merging):

The sketches are as follows (splitting large problems, solving small problems, merging):

3. The existing massive log data is saved in a super large file, the file can not be read directly into the memory, the request from the day to extract the most visited Baidu the number of the IP.

STEP1: From this day's log data to access Baidu's IP out, write to a large file;

STEP2: Note that the IP is 32-bit, with a maximum of 2^32 IP. You can also use a mapping method, such as modulo 1000, to map the entire large file to 1000 small files;

STEP3: Find out the most frequent IP in each small text (can be used hash_map frequency statistics, and then find the most frequent) and the corresponding frequency;

STEP4: In these 1000 largest IPs, find the one with the highest frequency, that is, for the request.

The sketches are as follows:

Interview-Ali-. Big Data topic-given a, b two files, each store 5 billion URLs, each URL accounted for 64 bytes, memory limit is 4G, let you find a, b file common URL?