Making Hadoop Java process heap larger?

Posted on Mar 24, 2018 by Eric Ma In QA

In Hadoop 2.5.0, I use ‘ps -aux’ and find the Java process has options:

-Xmx1000m

However, my nodes have 32GB memory.

In yarn-env.sh, you can find:

# For setting YARN specific HEAP sizes please use this
# Parameter and set appropriately
# YARN_HEAPSIZE=1000

In hadoop-env.sh, you can find:

# The maximum amount of heap to use, in MB. Default is 1000.
#export HADOOP_HEAPSIZE=
#export HADOOP_NAMENODE_INIT_HEAPSIZE=""

So, uncomment them and change them to the required values you want. Restart Hadoop and it will go.

Categories