How To Debug Linux Kernel With Less Efforts

Introduction

In general, if we want to debug Linux Kernel, there are lots of tools such as Linux Perf, Kprobe, BCC, Ktap, etc, and we can also write kernel modules, proc subsystems or system calls for some specific debugging aims. However, if we have to instrument kernel to achieve our goals, usually we would not like to pay more efforts like above solutions since we’d like to achieve our aims quickly and easily. In this article, I will introduce a way to debug Linux Kernel with less efforts.

Example – Debug Linux Kernel Scheduling Subsystems

In this example, I want to know how long a thread is executing on one CPU so I would like to add one flag, say ‘enable_flag’, to be one switch for this feature. At the same time, of course, I need to know which thread I would like to debug so I set another parameter, say ‘enable_pid’. See following patch for details.


--- ./core.c 2016-11-19 20:17:41.000000000 -0500
+++ /usr/src/linux-3.16.39/kernel/sched/core.c 2017-01-04 22:23:37.419571125 -0500
@@ -90,6 +90,29 @@
#define CREATE_TRACE_POINTS
#include

+//added by Weiwei Jia
+#include +int enable_flag = 0;
+module_param(enable_flag, int, 0664);
+EXPORT_SYMBOL_GPL(enable_flag);
+
+int enable_tick1 = 0;
+module_param(enable_tick1, int, 0664);
+EXPORT_SYMBOL_GPL(enable_tick1);
+
+int enable_tick2 = 0;
+module_param(enable_tick2, int, 0664);
+EXPORT_SYMBOL_GPL(enable_tick2);
+
+int enable_debug = 0;
+module_param(enable_debug, int, 0664);
+EXPORT_SYMBOL_GPL(enable_debug);
+
+int enable_pid = 0;
+module_param(enable_pid, int, 0664);
+EXPORT_SYMBOL_GPL(enable_pid);
+//ended
+
#ifdef smp_mb__before_atomic
void __smp_mb__before_atomic(void)
{
@@ -2797,6 +2820,7 @@
unsigned long *switch_count;
struct rq *rq;
int cpu;
+ s64 diff = 0; //added by Weiwei Jia

need_resched:
preempt_disable();
@@ -2855,6 +2879,29 @@
rq->curr = next;
++*switch_count;

+ //added by Weiwei Jia
+ do_gettimeofday(&(next->__ts));
+ next->__start_ts = (s64) ((next->__ts).tv_sec * 1000000 + (next->__ts).tv_usec);
+ prev->__end_ts = next->__start_ts;
+ if (enable_flag == 1 && prev->pid == enable_pid) {
+ //printk(KERN_INFO "%lld\n", prev->__end_ts - prev->__start_ts);
+ //diff = (prev->se).sum_exec_runtime - (prev->se).prev_sum_exec_runtime;
+ diff = prev->__end_ts - prev->__start_ts;
+ printk(KERN_INFO "%lld\n", diff);
+#if 1
+ printk(KERN_INFO "next process id is %d\n", next->pid);
+ if (diff < 2000) { + printk(KERN_INFO "------------------------------------------\n"); + printk(KERN_INFO "Start timestamp is %lld microseconds\n", prev->__start_ts);
+ printk(KERN_INFO "End timestamp is %lld microseconds\n", prev->__end_ts);
+ printk(KERN_INFO "Timeslice is %lld micorseconds\n\n", diff);
+ dump_stack();
+ printk(KERN_INFO "------------------------------------------\n");
+ }
+#endif
+ }
+ //ended
+
context_switch(rq, prev, next); /* unlocks the rq */
/*
* The context switch have flipped the stack from under us


“EXPORT_SYMBOL_GPL(enable_flag)” will export this parameter as one file under “/sys/module/core/parameters” directory, and you can write zero or not zero in “/sys/module/core/parameters/enable_flag” so that it will start this feature or not. You will also need to write the thread id into “/sys/module/core/parameters/enable_pid” to debug this specific thread. Now, you will find that this solution is very easy and you will pay less efforts. Please also remember that these parameters can also be used in other places with external definition.

Conclusion

This article presents a easy and quick way for system developers to debug Linux Kernel.

References

[1] http://elixir.free-electrons.com/linux/v3.16.39/source