Tuesday, August 6, 2013

system cpu usage and Linux kernel profiling

One of my systems was showing high system CPU utilization. While trying to diagnose this high system cpu usage, I discovered the profiler built into the kernel.

To enable Linux kernel profiling, you must set the profile parameter on the command line [1].

Boot the kernel with command line option profile=2 (or some other number instead of 2). This will cause a file /proc/profile to be created. The number given after profile= is the number of positions EIP is shifted right when profiling. So a large number gives a coarse profile. The counters are reset by writing to /proc/profile. The utility readprofile will output statistics for you. It does not sort - you have to invoke sort explicitly.

Here is my sorting command and some of the output, in case you were interested. 
# /shared/readprofile | sort -nr | head -20
2420865 total                                      1.2731
1511144 default_idle                             16979.1461
449875 shrink_active_list                       503.7794
 59360 schedule                                  22.6825
 44295 shrink_inactive_list                      24.4184
 38420 list_del                                 417.6087
 31976 generic__raw_read_trylock                1880.9412
 27377 release_pages                             92.8034
 24292 __list_add                               296.2439
 18459 list_add                                 1845.9000

This allows me to look up each of these functions in the kernel source and make guesses as to why the system is spending so much time performing those tasks. 

References:
Posted by at
Labels: , , , , , , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)