Spark Parameter Optimization

Spark has certain requirements on memory, and memory is insufficient because of GC and OOM.

1. By default, a worker memory of 0.6 is used for cache and 0.4 for task. You can set this value to increase the cache size of each worker.

Spark. Storage. memoryfraction 0.8

2. Set the number of parallel tasks to improve the reducer Efficiency

Spark. Default. parallelism 10

This will appear in version 1.0.2Java. Lang. illegalargumentexception: Can't zip RDDs with unequal numbers of partitions, which has been resolved in pr1763

3. Spark temporary directory
Spark. Local. DIR/data/spark/tmp

On 1.0Spark_local_dirs (standalone, mesos ),Local_dirs (yarn) Environment VariableOverwrite

Spark Parameter Optimization