How to set the number of mappers and reducers of Hadoop in command line?
Posted on In QAHow to set the number of mappers and reducers of Hadoop in command line?
Number of mappers and reducers can be set like (5 mappers, 2 reducers):
-D mapred.map.tasks=5 -D mapred.reduce.tasks=2
in the command line.
In the code, one can configure JobConf variables.
job.setNumMapTasks(5); // 5 mappers
job.setNumReduceTasks(2); // 2 reducers
Note that on Hadoop 2 (YARN), the mapred.map.tasks and mapred.reduce.tasks are deprecated and are replaced by other variables:
mapred.map.tasks --> mapreduce.job.maps
mapred.reduce.tasks --> mapreduce.job.reduces
Using map reduce.job.maps on command line does not work. Is there a particular syntax to use?
You can add the options to the command like
bin/hadoop jar -Dmapreduce.job.maps=5 yourapp.jar ...
