Spark: A more powerful distributed data computing project than Hadoop

Source: Internet
Author: User
Keywords Provide data calculation dfs
Tags aliyun analysis code computing core technology data data calculation data computing

"http://www.aliyun.com/zixun/aggregation/37954.html" Spark is a distributed data rapid analysis project developed by the University of California, Berkeley AMP Its core technology is flexible Resilient distributed datasets, which provide a richer MapReduce model than Hadoop, enable rapid iteration of datasets in memory to support complex data mining algorithms and graph computation algorithms.

Spark was developed using Scala, using Mesos as the underlying scheduling framework, tightly integrated with hadoop and Ec2, directly reading hdfs or S3 files for calculation and writing the results back to hdfs or S3 as part of Hadoop and Amazon's cloud computing ecosystem . Spark is a small and exquisite project, the project's core part of the code only 63 Scala files, fully reflects the beauty of streamlining.

Spark dependency

Map Reduce Model: As a distributed computing framework, Spark uses the MapReduce model. In its place, Google's Map Reduce and Hadoop traces the heavy, it is clear that it is not a big innovation, but a slight innovation. Under the premise of keeping the basic idea unchanged, it draws on, imitates and relies on its ancestors, adding a little improvement, greatly improving the efficiency of MapReduce. Functional programming: Spark is written by Scala and the supported language is also Scala. One of the reasons is that Scala supports functional programming. This led to the Spark code concise, and secondly to make Spark-based development process, but also particularly simple. A complete MapReduce, Hadoop need to create a Mapper class and Reduce class, and Spark only need to create a corresponding map function and reduce function, the amount of code is greatly reduced. Mesos: Spark will be distributed to run things to consider, have given Mesos, not Care, which is why it can streamline the code. HDFS and S3: Spark support two kinds of distributed storage systems: HDFS and S3. Should be considered the most mainstream of the two. File system read and write function is provided by Spark, with the help of Mesos distributed implementation.

Spark and Hadoop contrast

Spark intermediate data into memory, for iterative computing more efficient. Spark is more suitable for iterative computing more ML and DM operations. Because in Spark there is an abstract concept of RDD. Spark is more generic than Hadoop. There are many types of data sets that Spark provides, unlike Hadoop that provides only Map and Reduce operations. Such as map, filter, flatMap, sample, groupByKey, reduceByKey, union, join, cogroup, mapValues, sort, partionBy and other types of operations, Spark these operations called Transformations. Also provide Count, collect, reduce, lookup, save and other actions. These diverse dataset operation types provide convenience for users developing upper-level applications. The communication model between the various processing nodes is no longer the only Data Shuffle model as Hadoop does. Users can name, materialize, control the storage of intermediate results, partitions and so on. It can be said that the programming model is more flexible than Hadoop. However, due to the nature of RDDs, Spark does not apply to applications that have async fine-grained update status, such as web service storage or incremental web crawlers and indexes. It is not suitable for the incremental modification of the application model. Fault tolerance. Fault Tolerances are implemented through checkpoints when calculating distributed datasets. There are two ways to checkpoint, one is checkpoint data and the other is logging the updates. Users can control which approach is used to achieve fault tolerance. Usability. Spark improves usability by providing rich Scala, Java, Python APIs, and interactive shells.
Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.