Use of hadoop hprof

Source: Internet
Author: User

The mapreduce program we write is not necessarily efficient. We need to determine where the mapreduce bottleneck is. The hadoop Framework provides support for hprof. hprof can track CPU usage, heap usage, and thread lifecycles, which can be of great help for determining program bottlenecks.

To use hprof, we need to make some settings in jobconf. The specific operations are as follows:

Jobconf = new jobconf (CONF );

Jobconf. setprofileenabled (true); // enable hprof

Jobconf. setprofileparams ("-agentlib: hprof = depth = 8, CPU = sampl es, heap = sites, force = N," + "thread = Y, verbose = n, file = % s ");

// The value 8 indicates the stack call depth, which can be specified by the user.

Jobconf. setprofiletaskrange (true, "0-5 ");

// ID of the map task whose profile is required

Jobconf. setprofiletaskrange (false, "0-5 ");

// ID of the reduce task whose profile is required

With the above settings, you can enable hprof. After the mapreduce job is complete, you can view multiple profile files in the directory where the job is submitted. Open each profile file. At the bottom of the file, we can see the two most important statistics: the memory occupied by the object and the time consumed by stack calls.


 

Memory Object statistics and stack calls that generate this object (sorted in ascending order)

 

Sort the time consumed by stack calls (sorted from large to small)

The following figure shows the top 303703 stack calls. The order of function calls is displayed, as shown in:


 

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.