Saturday, January 24, 2009

Kernel Profiling

I got an opportunity to try my hands on a proprietary OS ( lets call it OS-XYZ, If I call it OSX, Steve would sue me)and when I compared the performance with a normal linux machine, OS-XYZ seemed to slow down the system by a factor of 3. The test program runs in userspace and library calls ( eg. glibc ) are translated to system calls in kernel space. The best approach to investigate is bottoms-up approach. See if everything is going fine in the kernel.
The next step is to do "kernel profiling" to find the exact difference and see if the kernel patches in OS-XYZ are affecting the system performance. Please see introduction to kernel profiling.
You need to make sure that the following points are taken care before starting off -
1) Kernel should be compiled in debug mode by executing
make menuconfig
Check the kernel .config file for
CONFIG_PROFILING=y
CONFIG_OPROFILE=m
CONFIG_DEBUG_INFO=y
CONFIG_FRAME_POINTER=y

2) Copy the
System.map
file to say
/boot/System.map
.

3) Boot with the new kernel with profile=2 option. This will generate a profile file
/proc/profile

Execute the following commands to do the profiling when running the test_program.

readprofile -r
readprofile -M1
./test_program
readprofile -v -m /boot/System.map | sort -n +2 | tail -40

This will print the functions called in kernel space, number of ticks, and number of times each function is called. Also see man readprofile


Once you are done with the profiling, examine the data and compare the sources in
/usr/src/linux/arch/mips/kernel/
and get going.

Checklist -
-> Make sure all kernel modules ( *.ko) are present in the patched kernel.
-> Think of ways to restore the box, in case kernel fails to load up or on kernel panic.

No comments: