Sparkcontext main constructor Code extraction

Source: Internet
Author: User

This is a note that reads the code of the Sparkcontext class. When reading this class, the main task is to figure out how Sparkcontext is constructed, and the initialization of Java and C # class is put in one method, and Scala's main constructor code is almost scattered in the Sparkcontext class, This requires us to organize it, in order to have a structured reading work.

A few of the main things that are done in Sparkcontext's main constructor are:

1, all kinds of analytic conf (this I do not have a unified collection, really too much, if you want to see, directly to the Sparkcontext class search "conf")

2, the creation of sparkenv, and his internal components (Listenerbus, metrics system, Blockmanager, etc.) initialization, start

(Listenerbus, Metrics system profile affixed http://blog.csdn.net/allwefantasy/article/details/50449464)

3. Creation and start of TaskScheduler, Dagscheduler, Sparkui

On the Code Note map: (Sparkcontext main constructor code extraction)

Sparkcontext main constructor Code extraction

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.