Hadoop/yarn/mapreduce memory allocation (configuration) scheme

based on the recommended configuration of Horntonworks, a common memory allocation scheme for various components on Hadoop cluster is given. The right-most column of the scenario is a 8G VM allocation scheme that reserves 1-2g memory to the operating system, assigns 4G to Yarn/mapreduce, and of course includes hive, and the remaining 2-3g is reserved for hbase when it is necessary to use HBase.

Configuration File	Configuration Setting	Value calculation	8G VM (4G for MR)
Yarn-site.xml	Yarn.nodemanager.resource.memory-mb	= Containers * Ram-per-container	4096
Yarn-site.xml	Yarn.scheduler.minimum-allocation-mb	= Ram-per-container	1024
Yarn-site.xml	Yarn.scheduler.maximum-allocation-mb	= Containers * Ram-per-container	4096
Mapred-site.xml	Mapreduce.map.memory.mb	= Ram-per-container	1024
Mapred-site.xml	Mapreduce.reduce.memory.mb	= 2 * Ram-per-container	2048
Mapred-site.xml	Mapreduce.map.java.opts	= 0.8 * Ram-per-container	819
Mapred-site.xml	Mapreduce.reduce.java.opts	= 0.8 * 2 * Ram-per-container	1638
Yarn-site.xml (check)	Yarn.app.mapreduce.am.resource.mb	= 2 * Ram-per-container	2048
Yarn-site.xml (check)	Yarn.app.mapreduce.am.command-opts	= 0.8 * 2 * Ram-per-container	1638
Tez-site.xml	Tez.am.resource.memory.mb	= Ram-per-container	1024
Tez-site.xml	Tez.am.java.opts	= 0.8 * Ram-per-container	819
Tez-site.xml	Hive.tez.container.size	= Ram-per-container	1024
Tez-site.xml	Hive.tez.java.opts	= 0.8 * Ram-per-container	819

 

Hadoop/yarn/mapreduce memory allocation (configuration) scheme