how change my policy of scheduling in hadoop?
Posted on In QAI want to change policy of scheduling in Hadoop, how to I can change job order in map reduce automatically.
Assume you are using Hadoop 2 / YARN.
The configuration parameter named yarn.resourcemanager.scheduler.class
controls the class to be used as the resource scheduler for YARN/Hadoop.
The default value for the scheduler class (check more at https://www.systutorials.com/241248/hadoop-2-yarn-default-configuration-values/) is:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
which is the Capacity Scheduler.
To change the scheduler used, you need to assign the scheduler class in yarn-site.xml in the etc/hadoop/ directory under Hadoop installation. Taking the fair scheduler shipped with Hadoop as an example, we can add to yarn-site.xml:
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>
Each scheduler may have its own options for configurations, such as capacity scheduler and fair scheduler. You may check their manuals to configure and tune them.