Sorting out massive data interview questions

1. Given two files a and B, each of which stores 50Hundreds of millions of URLsEach url occupies 64 bytes and the memory limit is 4 GB. Can you find the common URLs of files a and B?

Solution 1: it can be estimated that the size of each file is 50 GB × 64 = 320 GB, far greater than the memory limit of 4 GB. Therefore, it is impossible to fully load it into the memory.Processing. Consider adopting a divide-and-conquer approachMethod.

S traverses file a, obtains each url, and stores the url separately according to the obtained value1000 small files (marked. In this way, the size of each small file is about 300 MB.

S traverses file B and stores the url in the same way as a1000 small files (marked ). This wayProcessingAll the URLs that may be the same are in the corresponding small file (), and non-corresponding small files cannot have the same url. Then we just need to find1000 pairs of small files with the same url.

S. You can store the url of a small file in hash_set for the same url in each pair of small files. Then traverse each url of another small file to see if it is in the hash_set just constructed. If it is, it is a common url and saved in the file.

Solution 2: if a certain error rate is allowed, you can use the Bloom filter. The 4G memory can be approximately 34 billion bits. Map the URLs in one file to the 34 billion bits using the Bloom filter, read the URLs of another file one by one, and check whether the URLs are consistent with the Bloom filter. If yes, the url should be a common url (note that there will be a certain error rate ).

2. There are 10 files, each of which is 1 GB. each row of each file stores the user's query, and the query of each file may be repeated. Sort the query frequency.

Solution 1:

S reads 10 files in sequence and writes the query to the other 10 files according to the hash (query) % 10 results. In this way, the size of each newly generated file is about 1 GB (assuming that the hash function is random ).

S finds a machine with around 2 GB of memory, and uses hash_map (query, query_count) to calculate the number of times each query appears. Sort by the number of occurrences by means of fast/Heap/Merge Sorting. Output the sorted query and the corresponding query_cout to the file. In this way, 10 sorted files (marked as) are obtained ).

S. Merge and sort the 10 files (internal and external sorting are combined ).

Solution 2:

Generally, the total number of queries is limited, but the number of repetitions is large. For all queries, you can add them to the memory at one time. In this way, we can use the trie tree, hash_map, and so on to directly count the number of occurrences of each query, and then perform fast/Heap/Merge Sorting based on the number of occurrences.

Solution 3:

Similar to solution 1, but after hash is completed and divided into multiple files, it can be handed over to multiple files.Processing, Using a distributed architectureProcessing(Such as MapReduce), and then merge.

3. There is a 1 GB file with each row containing a word. The word size cannot exceed 16 bytes, and the memory size is limited to 1 MB. Highest return frequency100.

Solution 1: read each word x in an ordered file and save it to 5000 small files (marked as) based on this value. In this way, each file is about KB. If the size of some files exceeds 1 MBMethodContinue to split down and know that the size of the small file to be decomposed cannot exceed 1 MB. For each small file, calculate the words in each file and the corresponding frequency (trie tree/hash_map can be used), and obtain the most frequently occurring100Words (can contain100Node), and100Words and the corresponding frequency are stored in the file, and then 5000 files are obtained. The next step is to merge the 5000 files (similar to the merge and sort files.

4. Extract the IP address with the most visits to Baidu on a certain day with massive log data.

Solution 1: the first is this day, and the IP addresses in the logs accessing Baidu are obtained and written to a large file one by one. Note that the IP address is 32-bit and has a maximum of one IP address. You can also use ingMethodFor example1000. map the entire large file1000 small files, and then find the IP address with the highest frequency in each small text (hash_map can be used for frequency statistics, and then find the maximum frequency) and the corresponding frequency. And then100Find the IP address with the highest frequency among the 0 largest IP addresses.

5. Locate non-repeated integers among the 0.25 billion integers. The memory is insufficient to accommodate these 0.25 billion integers.

Solution 1: Use 2-Bitmap (2bit for each number, 00 indicates no, 01 indicates one time, 10 indicates multiple times, and 11 indicates no significance). Memory is required, it is also acceptable. Then scan the 0.25 billion Integers to check the corresponding bits in Bitmap. If 00 is changed to, 10 remains unchanged. After the descriptions are completed, view the bitmap and output the corresponding digit as an integer of 01.

Solution 2: similar questions can also be usedMethodTo partition small files.Method. Then, find the non-repeated integers in the small file and sort them. Then merge the elements to remove them.

6. Massive Data is distributed in100In the mind of the TV station, I want to find a way for colleges and universities to calculate the top 10 of this batch of data.

Solution 1:

S finds the top 10 items on each computer, which can be completed by using a heap containing 10 elements (top 10 smaller items, with the maximum heap, top 10 larger ones, with the minimum heap ). For example, if we want to increase the top 10, we should first adjust the first 10 elements to the smallest heap. If we find the top 10 elements, we will then scan the subsequent data and compare it with the heap top elements. If it is larger than the top 10 elements, use this element to replace the heap top and then adjust it to the minimum heap. The final heap element is the top 10.

S, find the top 10 on each computer, and then100The top 10 computers are combined.100Zero data records, similar to the aboveMethodFind the top 10.

7. How can I find the most repeated data?

Solution 1: First hash, then map the modulo to a small file, find the most repeated one in each small file, and record the number of repetitions. Find out the most repeated data in the previous step (For details, refer to the previous question ).

8. Tens of millions or hundreds of millions of data records (with duplicates) are collected to calculate the N data records with the most frequent occurrences.

Solution 1: Data of tens of millions or hundreds of millions can be stored in the memory of the current machine. Therefore, we recommend that you use hash_map/binary tree search/red/black tree to calculate the number of times. Then we can retrieve the first N data records that appear most frequently. We can use the heap mechanism mentioned in question 6th.

9.1000 thousand strings, some of which are repeated. You need to remove all the duplicates and keep the strings that are not repeated. How can I design and implement it?

Solution 1: it is more appropriate to use the trie tree, And hash_map should also work.

10. A text file contains about 10 thousand rows and one word per line. The first 10 words that most frequently appear must be counted. Please give your thoughts and analyze the time complexity.

Solution 1: consider time efficiency. Use the trie tree to count the number of times each word appears. The time complexity is O (n * le) (le indicates the word's level length ). Then we can find out the first 10 words that appear most frequently. We can use the heap method. As mentioned in the previous question, the time complexity is O (n * lg10 ). Therefore, the total time complexity is the greater of O (n * le) and O (n * lg10.

11. Find the first 10 frequently-seen words in a text file. However, this file is long and may contain hundreds of millions of lines or billions of lines. In short, it is impossible to read the memory at a time and ask the optimal solution.

Solution 1: first, the file is divided into multiple small files based on hash and modulo.MethodFind the 10 most common words in each file. Then mergeProcessingTo find the final 10 most frequently-seen words.

12.100Find the maximum number100Number.

Solution 1: As mentioned in the previous question100Element. Complexity is O (100W * lg100).

Solution 2: the idea of fast sorting is adopted. After each split, only a portion larger than the axis is considered.100When there are many, the traditional sorting algorithm is used for sorting.100. Complexity is O (100W *100).

Solution 3: Adopt the local elimination method. Before selection100Elements, sorted as sequence L. Then, the remaining element x is scanned at a time.100If the smallest element ratio is greater than the smallest element, delete the smallest element, and insert x into sequence L using the insertion sorting idea. It cyclically scans all elements. Complexity is O (100W *100).

13. Search for popular queries:

The search engine records all the search strings used for each search using log files. The length of each query string is 1-bytes. Suppose there are currently 10 million records, and these query strings have a relatively high number of repeated reads. Although the total number is 10 million, the number of duplicate reads cannot exceed 3 million if the number of duplicate reads is removed. The higher the repetition of a query string, the more users query it, and the more popular it is. Please count the top 10 query strings. The memory required cannot exceed 1 GB.

(1) Describe your solution to this problem;

(2) Please provide the mainProcessingProcess, algorithm, and algorithm complexity.

Solution 1: The trie tree is used, and the keyword field stores the number of times that the query string appears, not 0. At last, we sorted the occurrence frequency with the minimum push of 10 elements.

14. A total of N machines, each with N numbers. Each machine can store a maximum of O (N) numbers and operate on them. How do I find the middle number in the number?

Solution 1: First estimate the range of these numbers. For example, assume that these numbers are all 32-bit unsigned integers (total ). We divide the integers 0 to into N range segments. Each segment contains an integer. For example, the first field is 0 to, and the second segment is ,..., The Nth segment is. Then, scan the number of N on each machine and place the number in the first segment on the first machine. Put the number in the second segment on the second machine ,..., Place the number of the nth segment on the nth machine. Note that the number stored on each machine in this process should be O (N. Next we will count the number of machines in sequence and accumulate them at a time until we find the k machine. The number accumulated on this machine is greater than or equal, the sum on the second K-1 machine is less than, and this number is counted as x. Then the median we are looking for is in the first position of the k machine. Then we sort the number of k machines and find the number, that is, the median. The complexity is.

Solution 2: sort the numbers on each machine first. After sorting, we use the thought of merging and sorting to merge the numbers on the N machines to get the final sorting. Finding the first is what you want. The complexity is.

15. Maximum gap problem

Given n real numbers, the maximum difference between n real numbers on the real axis between the number of vectors 2 requires a linear time algorithm.

Solution 1: first thoughtMethodThat is, sort the n data first, and then scan it again to determine the maximum gap between the adjacent data. HoweverMethodIt cannot meet the requirements of linear time. So take the followingMethod:

S finds the largest and smallest data max and min in n data.

S using N-2 points of the same interval [min, max], that is [min, max] divided into n-1 intervals (before and after the Open interval), These intervals as the bucket, number, the upper bound of the bucket is the same as that of the bucket I + 1, that is, the size of each bucket is the same. The size of each bucket is :. In fact, the boundary of these buckets forms an arithmetic difference sequence (the first item is min, and the tolerance is). It is considered that min is placed in the first bucket, and max is placed in the n-1 bucket.

S puts n numbers into n-1 buckets: assigns each element to a bucket (number: index), and obtains the maximum and minimum data allocated to each bucket.

S maximum gap: in addition to the maximum and minimum data max and min N-2 data into n-1 barrels, the principle of the drawer shows that at least one bucket is empty, because each bucket has the same size, the maximum gap does not appear in the same bucket. It must be the gap between the upper bound of a bucket and the lower bound of a bucket in the climate, and the buckets between the bins (even if the connection is good) must be empty buckets. That is to say, the maximum gap is generated between the upper bound of bucket I and the lower bound of Bucket j. Scan once.

16. Merge multiple sets into a set without intersection: a set of strings is given. The format is as follows :. Merging a set whose intersection is not empty requires that there be no intersection between the merged sets. For example, the preceding example should be output.

(1) Describe your solution to this problem;

(2) give the mainProcessingProcess, algorithm, and algorithm complexity;

(3) describe possible improvements.

Solution 1: Use and query sets. First, all the strings are in the separate query set. Then, the two adjacent elements are merged sequentially according to the scanning of each set. For example, first check whether aaa and bbb are in the same and check the set. If not, check the set where aaa and bbb are located and, then, check whether the bbb and ccc are in the same and check the set. If they are not, check the set where they are located. Next, scan other sets. When all the sets are scanned and the set is queried. The complexity should be O (NlgN. For improvement, You can first record the root node of each node to improve the query. When merging, you can combine large and small, which also reduces complexity.

17. Maximum subsequence and maximum submatrix Problems

Maximum sub-sequence of an array: if an array is given, the elements have both positive and negative values, and find a continuous sub-sequence to make and maximum.

Solution 1: this problem can be solved through dynamic planning. It indicates the maximum subsequence ending with element I. This can be quickly implemented using code.

Maximum sub-matrix problem: Given a matrix (two-dimensional array), where the data is large and small, please find a sub-matrix to make the sum of the sub-matrix largest, and output this sum.

Solution 1: it can be solved using the same idea as the largest subsequence. If we determine the element between column I and column j, it is actually a maximum subsequence problem in this range. How to determine whether column I and column j can be searched by brute forceMethod.