Adding Performance Counters to the Kernel
From Debian Clusters
Contents |
Adding Performance Counters to the Kernel
One of the patches to the Linux kernel that provides an advantage to those wanting to tune and profile application performance is the perfctr patch. perfctr provides a kernel-level interface to the hardware performance counters present in many of today's common CPU architectures.
This document describes two components: the augmentation of the kernel to accommodate access to the the performance counters and the performance counters API library to access and manipulate these counters from user level. At the time of this writing, these components are available at
Note that a snapshot of the perfctr code base is commonly included in the PAPI download from UTK. However, the perfctr code base included in the UTK package is not as updated, and somewhat slanted toward the more common RPM-based kernels. The perfctr code base from posted Mikael Pettersson (the perfctr architect and project author) is, at the time of this writing, the more Debian-friendly approach to integrating perfctr support into your Debian kernel.
Note: The perfctr code base versioning has little to do with the Linux kernel version that you're running. In this example, we'll be using the 2.6.24 Linux kernel code base along with the 2.6.32 version of the perfctr code base.
Get the code
Before fetching the code for support of hardware performance counters, make sure that you already have the source code for the kernel that you intend to patch. See Setting up the kernel build environment for instructions on how to obtain the kernel source code.
If your kernel source tree has been used to rebuild a kernel previously, your kernel source tree will need to be reset to a condition where the process of applying a patch can be done reliably. Before proceeding below, save a copy of your current kernel configuration and clean out the kernel source tree:
vulture:/usr/src/linux-source-2.6.24# cp .config /tmp/config-bak vulture:/usr/src/linux-source-2.6.24# make-kpkg clean vulture:/usr/src/linux-source-2.6.24# make-kpkg clean vulture:/usr/src/linux-source-2.6.24# make mrproper vulture:/usr/src/linux-source-2.6.24# cp /tmp/config-bak .config
If you find yourself without a .config in the Linux source directory, the recommended starting point for generating a new kernel would be to start with the configuration file for an existing, "known-working" kernel. In Debian, the kernel configurations for kernels that have been installed from Debian packages (i.e., the "Debian way") are found under /boot/.
perfctr
Once your kernel source code is installed you can add performance counter functionality to the kernel source tree, recompile the kernel, and install the kernel image. The first of the two components that will be used to support access to the hardware counters is the actual kernel code base that will need to be patched into the kernel, from Mikael Pettersson:
vulture:/usr/local/src# wget http://user.it.uu.se/~mikpe/linux/perfctr/2.6/perfctr-2.6.32.tar.gz
Untar the software
vulture:/usr/local/src# tar xvfz perfctr-2.6.32.tar.gz
As with all software packages, read the README and INSTALL files found under the directory where the software was unpacked. The devil is in the details, the gist is in this writeup. The general procedure is to leverage the update-kernel script that is included with the perfctr code base. This script is run from your Linux source directory:
vulture:/usr/local/src/perfctr-2.6.32# cd /usr/src/linux-source-2.6.24 vulture:/usr/src/linux-source-2.6.24# /usr/local/src/perfctr-2.6.32/update-kernel Found patch for kernel 2.6.24 Applying kernel source patch for 2.6.24 (auto-detected) patching file CREDITS patching file Documentation/ioctl-number.txt patching file MAINTAINERS patching file arch/arm/Kconfig patching file arch/arm/kernel/process.c patching file arch/i386/Kconfig patching file arch/i386/kernel/entry.S patching file arch/i386/kernel/i8259.c patching file arch/i386/kernel/process.c patching file arch/powerpc/Kconfig patching file arch/powerpc/kernel/process.c patching file arch/x86_64/Kconfig patching file arch/x86_64/kernel/entry.S patching file arch/x86_64/kernel/i8259.c patching file arch/x86_64/kernel/process.c patching file drivers/Makefile patching file include/asm-arm/processor.h patching file include/asm-arm/system.h patching file include/asm-i386/mach-default/irq_vectors.h patching file include/asm-i386/mach-visws/irq_vectors.h patching file include/asm-i386/processor.h patching file include/asm-i386/system.h patching file include/asm-powerpc/processor.h patching file include/asm-x86_64/hw_irq.h patching file include/asm-x86_64/irq.h patching file include/asm-x86_64/processor.h patching file include/asm-x86_64/system.h patching file kernel/exit.c patching file kernel/sched.c patching file kernel/timer.c Installing new kernel source files
Once the kernel source has been patched, configure the kernel to use the new perfctr code:
vulture:/usr/src/linux-source-2.6.24# make oldconfig HOSTCC scripts/basic/fixdep HOSTCC scripts/basic/docproc HOSTCC scripts/kconfig/conf.o HOSTCC scripts/kconfig/kxgettext.o SHIPPED scripts/kconfig/zconf.tab.c SHIPPED scripts/kconfig/lex.zconf.c SHIPPED scripts/kconfig/zconf.hash.c HOSTCC scripts/kconfig/zconf.tab.o HOSTLD scripts/kconfig/conf scripts/kconfig/conf -o arch/i386/Kconfig * * Linux Kernel Configuration ... Enable seccomp to safely compute untrusted bytecode (SECCOMP) [N/y/?] n * * Performance-monitoring counters support * Performance monitoring counters support (PERFCTR) [N/m/y/?] (NEW)
You should be prompted at this point for perfctr configuration options that weren't present in the kernel prior to patching your kernel. Enter "m" at this point to enable the perfctr capabilities as a module in your kernel.
Once you've enabled the perfctr capability as a module in the kernel, you will be prompted for additional perfctr configuration options. The most reasonable choices shown below.
Performance monitoring counters support (PERFCTR) [N/m/y/?] (NEW) m Additional internal consistency checks (PERFCTR_DEBUG) [N/y/?] (NEW) n Init-time hardware tests (PERFCTR_INIT_TESTS) [N/y/?] (NEW) y Virtual performance counters support (PERFCTR_VIRTUAL) [N/y/?] (NEW) y Global performance counters support (PERFCTR_GLOBAL) [N/y/?] (NEW) y
At this point, you can rebuild the kernel:
vulture:/usr/src/linux-source-2.6.24# CONCURRENCY_LEVEL=8 make-kpkg kernel-image --initrd --append-to-version=-perfctr0
CONCURRENCY_LEVELspecifies the number of parallel compilation tasks that your system is able to support (and this is the "Debian way" of filling outmake's-joption during the compilation process).kernel-imagemeans that we're building a kernel package that contains the bootable kernel code and all of the supporting infrastructure, such as the initial ramdisk, as specified by--initrd.- Finally,
--append-to-versionprovides you with a way to mark and track your kernel versions and patches, and helps you avoid clobbering an existing kernel that works with a newly-built one that doesn't.
Once make-kpkg finishes, a Debian kernel-image will be created in /usr/src. Install the kernel on your system with:
vulture:/usr/src/linux-source-2.6.24# cd /usr/src/ vulture:/usr/src# dpkg -i linux-image-2.6.24-perfctr0_2.6.24-perfctr0-10.00.Custom_i386.deb
Do a quick sanity check on /boot/grub/menu.lst, making sure that at least one "known-working" kernel is available in your grub boot menu. Boot into your new kernel.
Once you've booted into the new kernel, enable the performance counters with modprobe -a perfctr, and verify that they loaded successfully by tailing /var/log/messages
vulture:~# modprobe -a perfctr vulture:~# dmesg | tail -13 Please email the following PERFCTR INIT lines to mikpe@it.uu.se To remove this message, rebuild the driver with CONFIG_PERFCTR_INIT_TESTS=n PERFCTR INIT: vendor 5, family 6, model 9, stepping 10, clock 997166 kHz PERFCTR INIT: NITER == 64 PERFCTR INIT: loop overhead is 264 cycles PERFCTR INIT: rdtsc cost is 21.6 cycles (1649 total) PERFCTR INIT: rdpmc cost is 19.6 cycles (1520 total) PERFCTR INIT: rdmsr (counter) cost is 33.5 cycles (2409 total) PERFCTR INIT: rdmsr (evntsel) cost is 62.6 cycles (4276 total) PERFCTR INIT: wrmsr (counter) cost is 66.4 cycles (4519 total) PERFCTR INIT: wrmsr (evntsel) cost is 93.0 cycles (6220 total) PERFCTR INIT: read cr4 cost is 30.1 cycles (2191 total) PERFCTR INIT: write cr4 cost is 226.7 cycles (14777 total) PERFCTR INIT: sync_core cost is 159.2 cycles (10456 total)
One last tweak is in order so as to allow users access to the performance counters. When perfctr loads into the kernel, the device interface that is set up from udev sets the permissions to 660, with root:root user and group ownership. This leaves regular users out in the cold.
vulture:~# ls -l /dev/perfctr crw-rw---- 1 root root 10, 182 2008-05-18 21:35 /dev/perfctr
To automate the setting of permissions in udev, add the following to /etc/udev/rules.d/020_permissions.rules
KERNEL=="perfctr",MODE="0666"
This can go pretty much anywhere in the 020_permissions.rules file, as long as it is above the final LABEL="permissions_end" closing line. Once you've made that adjustment, you can do the following sanity check:
vulture:/etc/udev/rules.d# rmmod perfctr vulture:/etc/udev/rules.d# /etc/init.d/udev stop Stopping the hotplug events dispatcher: udevd. vulture:/etc/udev/rules.d# /etc/init.d/udev start Starting the hotplug events dispatcher: udevd. Synthesizing the initial hotplug events...done. Waiting for /dev to be fully populated...done. vulture:/etc/udev/rules.d# modprobe -a perfctr vulture:/etc/udev/rules.d# ls -l /dev/perfctr crw-rw-rw- 1 root root 10, 182 2008-05-18 21:58 /dev/perfctr
Now, and in the future, users can access the performance counters in their applications. That is, if only there was an API to do so. Luckily, that's where PAPI comes in.
Congratulations. You now have performance counter functionality enabled in your customized kernel. To load the perfctr module automatically upon system restart, add perfctr to /etc/modules
vulture:~# echo perfctr >> /etc/modules
PAPI
The second of the two components that will be used to support access to the hardware counters comes from the Performance Application Programming Interface project out of UTK. Find your way into /usr/local/src and issue
vulture:/usr/local/src# wget http://icl.cs.utk.edu/projects/papi/downloads/papi-3.6.0.tar.gz ... vulture:/usr/local/src# tar xvfz papi-3.6.0.tar.gz vulture:/usr/local/src# cd papi-3.6.0
Take a look at the INSTALL.txt and README files in this directory, before hopping into the src directory to configure and build the package:
vulture:/usr/local/src/papi-3.6.0/# cd src vulture:/usr/local/src/papi-3.6.0/src# ./configure vulture:/usr/local/src/papi-3.6.0/src# make
If you haven't already, boot into a kernel with perfctr support enabled so that the subsequent test modes can be performed:
vulture:/usr/local/src/papi-3.6.0/src/# make fulltest ... Running C Tests Running ctests/johnmay2: johnmay2.c PASSED Running ctests/profile_pthreads: profile_pthreads.c PASSED ... Running ftests/eventname: eventname.F PASSED Running ftests/highlevel: highlevel.F PASSED Running ftests/strtest: strtest.F PASSED Running ftests/fmatrixpapi2: fmatrixpapi2.F PASSED
If things fail abysmally at this step, check to make sure that the perfctr module is loaded and the device /dev/perfctr has the correct permissions for the user running the tests.
Once you're convinced everything is good to go, install the system-level binaries, libraries, and test programs:
vulture:/usr/local/src/papi-3.6.0/src/# make install-all
Pay particular attention to the codes in the ctest and ftests directories. The sample codes in these directories give very helpful examples of how to interact with the perfctr functionality in the kernel. In particular, ftests/avail, utils/papi_avail and utils/papi_native_avail for codes that probe, set, and report the performance counters available on your particular architecture.
