how change my policy of scheduling in hadoop?

Posted on In QA

I want to change policy of scheduling in Hadoop, how to I can change job order in map reduce automatically.

Assume you are using Hadoop 2 / YARN.

The configuration parameter named yarn.resourcemanager.scheduler.class controls the class to be used as the resource scheduler for YARN/Hadoop.

The default value for the scheduler class (check more at https://www.systutorials.com/241248/hadoop-2-yarn-default-configuration-values/) is:

org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler

which is the Capacity Scheduler.

To change the scheduler used, you need to assign the scheduler class in yarn-site.xml in the etc/hadoop/ directory under Hadoop installation. Taking the fair scheduler shipped with Hadoop as an example, we can add to yarn-site.xml:

<property>
  <name>yarn.resourcemanager.scheduler.class</name>
  <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>

Each scheduler may have its own options for configurations, such as capacity scheduler and fair scheduler. You may check their manuals to configure and tune them.

Eric Ma

Eric is a systems guy. Eric is interested in building high-performance and scalable distributed systems and related technologies. The views or opinions expressed here are solely Eric's own and do not necessarily represent those of any third parties.

Leave a Reply

Your email address will not be published. Required fields are marked *