Big Data class algorithm problem

1. There is a 1G size of a file, each line is a word, the size of the word does not exceed 16 bytes, memory limit size is 1M. Returns the highest 100 words in a frequency

1G has 2^26 words, 1M can save 2^16 words.

Step1: Use hash hash method, hash (x)/5000 to save a file word to 5,000 different spaces. (on average, each space is approximately 200k in size, some may be more than 1M, then the set continues to divide, knowing that it is less than 1M)

Step2: Each set is put into memory for hashmap processing, key is set to Word, and value is frequency.

Step3: For each set, the minimum heap sorting algorithm is used to find the highest level of 100. The 5,000 are then sorted in a merge order. (5000*100=500000<2^16)

Big Data class algorithm problem