Spark executor memory allocation on yarn _spark

A executor corresponds to a JVM process.

From the point of view of Spark, the memory occupied by executor is divided into two parts: Executormemory and Memoryoverhead

First, Executormemory

Executormemory is the Java heap area of the JVM process. The size is set by the property spark.executor.memory. You can also use parameters--executor-memory settings when Spark-submit commands.

The memorystore used to cache RDD data is located in this area.

Memorystore the proportion of space occupied by attributes Spark.storage.memoryFraction and spark.storage.safetyFraction control

Related Source:

Core/src/main/scala/org/apache/spark/storage/blockmanager.scala/** return to the total
amount of storage memory Available. *
Private def getmaxmemory (conf:sparkconf): Long = {
  val memoryfraction = conf.getdouble (" Spark.storage.memoryFraction ", 0.6)
  val safetyfraction = conf.getdouble (" spark.storage.safetyFraction ", 0.9)
  (Runtime.getRuntime.maxMemory * memoryfraction * safetyfraction). Tolong
}

So, a executor is used to store the rdd space = (executormemory–memory_used_by_runtime) * spark.storage.memoryFraction * Spark.storage.safetyFraction

Second, Memoryoverhead

Memoryoverhead is the amount of space that is occupied by the JVM process in addition to the Java heap, including the method area (permanent generation), the Java Virtual machine stack, the local method stack, the memory used by the JVM process itself, direct memory (directly Memory), and so on. Set by Spark.yarn.executor.memoryOverhead, in MB.

Related Source:

Yarn/common/src/main/scala/org/apache/spark/deploy/yarn/yarnsparkhadooputil.scala
  Val MEMORY_OVERHEAD_ FACTOR = 0.07
  val memory_overhead_min = 384

//yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ Yarnallocator.scala
  protected Val memoryoverhead:int = Sparkconf.getint ("Spark.yarn.executor.memoryOverhead",
    Math.max ((Memory_overhead_factor * executormemory). ToInt, memory_overhead_min))
......
      Val totalexecutormemory = executormemory + memoryoverhead
      numpendingallocate.addandget (missing)
      LogInfo (s) Would allocate $missing executor containers, each with $totalExecutorMemory MB "+
        S" Memory including $memoryOverhead M B overhead ")

Iii. related issues

If there is not enough space to store the RDD, the RDD partition that is stored is overwritten by the post storage. When you need to use data that is missing partitions, the missing data is recalculated

If there is not enough memory for the Java heap or the permanent generation, a variety of oom exceptions will be generated and executor will be terminated. Spark will reapply for a container run executor. The tasks on the failed executor and the stored data are recalculated on the other executor.

If the sum of the Executormemory+memoryoverhead (JVM process total memory) exceeds the container capacity during the actual operation. Yarn will kill container directly. There are no exception records in the executor log. Spark will also reapply for container run executor.

In situations where JVM process memory is larger than the Java heap, you should set the Memoryoverhead to a value that is large enough to set the Memoryoverhead to a value that is large enough to prevent the JVM process from being killed because the actual memory footprint is exceeded. If the default value (Math.max (Memory_overhead_factor *executormemory). Toint,memory_overhead_min) is not large enough, You can manually set a larger value by Spark.yarn.executor.memoryOverhead.

Resources:

http://www.wdong.org/wordpress/blog/2015/01/08/spark-on-yarn-where-have-all-my-memory-gone/

Http://stackoverflow.com/questions/28404714/yarn-why-doesnt-task-go-out-of-heap-space-but-container-gets-killed