How migration thread works inside of Linux Kernel

Abstract
In computer systems, resources have to be balanced so that the performance will be better based on the same hardware. In Linux Kernel system, we will see some migration kernel threads running as daemons to do this kind of jobs as follows. In this article, we will discuss how Linux Kernel balances its hardware/software resources (NOTE: this article is based on Linux Kernel 4.7 and other versions may have some differences).

$ ps aux | grep migration
root         9  0.0  0.0      0     0 ?        S    Nov17   0:00 [migration/0]
root        15  0.0  0.0      0     0 ?        S    Nov17   0:00 [migration/1]
root        21  0.0  0.0      0     0 ?        S    Nov17   0:00 [migration/2]
root        27  0.0  0.0      0     0 ?        S    Nov17   0:00 [migration/3]
root        33  0.0  0.0      0     0 ?        S    Nov17   0:00 [migration/4]

Details
In Linux Kernel, each CPU has one migration daemon to do migration jobs. We will not discuss the algorithms/mechanisms about how Linux Kernel balance resources and we will only discuss how Linux Kernel implement migration thread and how to migrate threads among CPUs. In How set_affinity works inside of Linux Kernel, we have discussed when the thread is running/TASK_WAKING, it will trigger stop_one_cpu to do migration jobs but we didn’t talk about who/when will do these kind of jobs inside of Linux Kernel. Actually, when you look inside of stop_one_cpu, you will find some clues as follows.

stop_one_cpu
--cpu_stop_queue_work
---__cpu_stop_queue_work
----list_add_tail and wake_up_process

In stop_one_cpu, the migration job will be packed as a work and call cpu_stop_queue_work to get the stopper thread of the CPU. And then, call __cpu_stop_queue_work to put this work into the work queue of stopper thread upon this CPU. At last, wake up the stopper thread to do migration. The stopper thread is named migration thread in the Linux Kernel (see following source codes).

504 static struct smp_hotplug_thread cpu_stop_threads = {
505         .store                  = &cpu_stopper.thread,
506         .thread_should_run      = cpu_stop_should_run,
507         .thread_fn              = cpu_stopper_thread,
508         .thread_comm            = "migration/%u",
509         .create                 = cpu_stop_create,
510         .park                   = cpu_stop_park,
511         .selfparking            = true,
512 };
513 
514 static int __init cpu_stop_init(void)
515 {
516         unsigned int cpu;
517 
518         for_each_possible_cpu(cpu) {
519                 struct cpu_stopper *stopper = &per_cpu(cpu_stopper, cpu);
520 
521                 spin_lock_init(&stopper->lock);
522                 INIT_LIST_HEAD(&stopper->works);
523         }
524 
525         BUG_ON(smpboot_register_percpu_thread(&cpu_stop_threads));
526         stop_machine_unpark(raw_smp_processor_id());
527         stop_machine_initialized = true;
528         return 0;
529 }
530 early_initcall(cpu_stop_init);

We find that Linux Kernel migration thread upon each CPU is initialized by early_initcall which is done when the kernel is started as follows.

start_kernel
--rest_init
---kernel_init
----kernel_init_freeable
-----do_pre_smp_initcalls
------do_one_initcall

Linux Kernel migration thread daemon is one of the early_initcall to be initialized. There are lots of init calls for Linux Kernel to do, which are like following.

814 static char *initcall_level_names[] __initdata = {
815         "early",
816         "core",
817         "postcore",
818         "arch",
819         "subsys",
820         "fs",
821         "device",
822         "late",
823 };

Up to here, I think everything is clear, migration daemon thread in Linux Kernel will be initialized when Kernel is started. And it will be woken up to do migration jobs once there is a migration work to do (NOTE: Here, I only consider the case for sched_setaffinity and, maybe, there are some other ways in Linux Kernel to wake up migration thread daemon). In addition, for other kernel thread daemon (like softirqd, kworker, etc), they have the similar way (like migration thread daemon) created by Linux Kernel.

Conclusion
In Linux Kernel, each CPU will have one migration thread daemon to do resource balance jobs. If we call sched_setaffinity system call to migrate one thread from source CPU to destination CPU and if the thread is running or at the TASK_WAKING status, the migration jobs will be packed into the work queue of source CPU’s migration thread, then, the migration thread will be waken up to finish migration job.

References
1, Linux Kernel Source Codes – https://www.kernel.org/
2, Linux Cross Reference – http://lxr.free-electrons.com/