Hadoop at the beginning

Source: Internet
Author: User

This time to do an IP segment matching the MapReduce program, recorded, some problems encountered, this time a total of several attempts:

  1. The login log is all added to the memory, then the total data in the Myinputformat processing and partitioning, first statistics the total amount of data, and then the data roughly evenly divided into several split. Then in the map of the setup () method to read the data of the small table IP segment, put into the list, at this time in the map function to read the message, the matching algorithm. But this, when the amount of data is small, when the amount of data is large, it will be reported memory overflow.

  2. First in the InputFormat all the re-weight after the UserID query, distributed to the different split as the map key, and then in each mapper, the Setup method to read the small table, in the map method to remove the current key of all login records, This record is compared with the IP segment data to produce the result. This method with the above common problem, loading data into memory, it is easy to cause memory overflow.

  3. In order to prevent memory overflow, you can put the data in HDFs, and then put the data in the small table in the memory, the log log will be read out to match the small table, there is no impact, but the data results are delayed, calculated because the matching loop too much, data processing speed is too slow.

  4. The data is stored in HDFs, then the first two bits of the IP address in login as key, and the first two bits in the IP segment as key, such as the actual login IP is, and the IP field is, At this point two tables at the same time 10.8 as the key value, so you can do the left and right join, and at this time the number of cycles is greatly reduced, efficiency is relatively fast.

Hadoop at the beginning

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.