Spark essay (II): Deep Learning

I. How to Choose coarse granularity and Fine Granularity

From the bottom layer to understand coarse granularity and fine granularity.

Layer 1: A class with three attribute values. To query all instances of this class, the level of fine-grained query is the attribute value, that is, the attribute value of each instantiated object is queried in sequence three times; the coarse-grained query by object directly queries all the instantiated objects of this class, query once. The query results are the same, but the query methods are different. Such instances include query operations in the database, full table query, and gradual query.

Layer 2: a dataset that contains multiple objects. When you perform operations on a dataset, the fine-grained processing method obtains each object in the dataset, and then executes the corresponding operations. The number of executions is the number of objects. The coarse-grained operations are directly performed on the entire dataset, perform operations on the objects in the dataset in sequence, regardless of the characteristics of the objects. The example of a dataset application with coarse granularity is Spark's RDDs.

Layer 3: A distributed application allows some nodes in the cluster to execute the computation it provides cyclically. When resources are allocated to nodes, the fine-grained allocation method checks each node required for application execution and allocates resources to these nodes. the coarse-grained allocation method is based on the application, directly allocate the resources required by the application to the application for processing. The fine-grained instances of distributed systems are mesos.

Therefore, coarse granularity ignores the internal details of the entire object, or assimilation the internal details in the calculation process to achieve the effect of block-based execution; fine-grained computing focuses on any attribute of an object and the execution steps, or the reuse of the underlying computing to achieve the effect of point-based execution.

In the RDDs operation, narrow dependencies converts an RDD to a new RDD. The operation object is the RDD dataset, and the corresponding map operation is directly performed on the <K, V> In the RDD. Therefore, the write operation is generally performed in units of the entire RDD, And the coarse-granularity mode is better. In the read operation, the coarse-granularity addressing mode is used to read the RDD, when you need to read the content in RDD for action operations, you can use fine-grained addressing.

Mesos adopts fine-grained sharing. One advantage of this is that although some tasks do not execute fine-grained tasks at the same time, long tasks and short tasks can still share space. The framework determines which resources are required based on the task length. Long tasks generally require more resources. Then mesos allocates resources to the Framework (this policy can be specified by the user), but the framework determines which resources to receive, the accepted resources can be used to execute tasks (long tasks or short tasks). The unacceptable resources are recycled by mesos and allocated to other frameworks. This avoids the embarrassment that long tasks cannot obtain resources, it also avoids short tasks from being executed because long tasks occupy too many resources. Can this method be applied to other aspects of long and short tasks? After a process in spark is determined as straggler, it is similar to a normal process in the relationship between long tasks and short tasks. At least their resource requirements should be similar, speculative execution of straggler can certainly solve some problems. However, if they make two resource judgments when applying for resources, the execution process of starggler will not affect the execution of normal processes, will it improve the system performance?

 

Ii. Distribution of functions in the system

Mesos's central idea: defines an interface that can ensure resource sharing and utilization as little as possible, and other work is pushed to frameworks.

(Unfinished, to be continued)

Spark essay (II): Deep Learning