Spark Data Partitioning

The Spark program can reduce network traffic overhead by partitioning. partitioning is not good for all scenarios: for example, if a given rdd is scanned only once, then there is absolutely no need for partitioning, and partitioning is helpful only if the data is multiple times in a key-based operation such as connecting. Suppose we have a constant large file UserData, and the small data generated every 5 minutes events, now requires UserData to do a join operation once every 5 minutes after the events data is produced. The code example for this procedure is as follows:

Val sc = new Sparkcontext ()
val userData = Sc.sequencefile[userid,linkinfo] ("hdfs://..."). Persist
def Processnewlogs (logfilename:string) {
    val events = Sc.sequencefile[userid, LinkInfo] (logfilename)
    //rdd of ( UserID, (userinfo,linkinfo)) pairs
    val joined = Usersdata.join (events)
    val offtopicvisits = joined.filter {
        //Expand the tuple into its
        (UserId, (UserInfo, linkinfo)) = 
            !userinfo.topics.contains (Linkinfo.topic)
    }. Count ()
    println ("Number of visits to non-subscribed opics:" + offtopicvisits)
}

By default, the join () operation hashes the primary key of the two rdd to a partition, sends the same elements of the primary key to the same machine over the network, and then connects with the same primary key.

1. Diagram of data flow when Partitionby () is not used on UserData:

The disadvantage is that UserData is hashed and mixed across nodes (shuffle) each time it is called, and resources and time are consumed more severely-although the data never changes.

The workaround is to partition and persist the UserData:

Val userData = Sc.sequencefile[userid,linkinfo] ("hdfs://...")
. Partionby (new Hashpartiotioner)
. Persist

2. Diagram of data flow when using Partitionby ():

Because partitioning and persistence are specified when constructing UserData, when Spark calls Userdata.join (events), it does not re-shuffle the UserData data, but only shuffle the events. Sending a record of the specific key in events to the machine on which the corresponding partition of the UserData is located (as shown in the image above) greatly reduces the need to do network traffic and speeds up the program run time.

Attention:

Partitionby () must be present at the same time as persist (), otherwise the hash and shuffle are recalculated when the partition's data is next called.