Spark3000 disciple Seventh Lesson spark operation principle and Rdd decryption summary

Source: Internet
Author: User

I heard Liaoliang's seventh lesson tonight. Spark operating principle and rdd decryption, after-school assignment is: Spark Fundamentals, my summary is as follows:

1 Spark is a distributed memory-based computing framework that is particularly suitable for iterative computing

2 MapReduce is two-stage map and reduce, and spark is constantly iterative, more flexible, more powerful, and easier to construct complex algorithms.

3 Spark does not replace hive,hive for Data Warehouse storage, spark SQL is just the COMPUTE engine that replaces hive

4 Spark intermediate data can be in memory or on disk

5 partition is a collection of data

6 Note: The beginner performs several steps to pay attention to step test, otherwise I do not know where the wrong

7 var data = Sc.textfile ("/user") does not have to write hdfs://, judging by context

8 Read the file get Hadooprdd, remove the file index, get Mappartitionsrdd, so a series of shards of data distributed in different machines.

9 Mobile computing instead of moving data

In addition, Liaoliang teacher to say a message:

Write Spark in Java: More talent, easier integration with Java EE, easier maintenance, so all the examples in the later classes are both Scala and Java

Follow-up courses can be referred to Sina Weibo Liaoliang _dt Big Data Dream Factory: Http://weibo.com/ilovepains

Liaoliang China Spark First person, public number Dt_spark

Spark3000 disciple Seventh Lesson spark operation principle and Rdd decryption summary

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.