Avoiding CPU Speed Scaling – Running CPU At Full Speed

10
2411

If you have a server on collocation with the super-fast CPU, that doesn’t mean that you get it running at full power.

Here is a preamble. When doing inventory of Linux servers, that I’m administering, I’ve found that some of them are running on lower CPU speed, then they could. This can be easily checked with this command:

#grep -E '^model name|^cpu MHz' /proc/cpuinfo

What you can see:

model name      : Intel(R) Core(TM) i7 CPU         920  @ 2.67GHz
cpu MHz         : 1600.000
...

or

model name      : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz
cpu MHz          : 1596.000
...

or even

model name      : Intel(R) Core(TM)2 CPU          4400  @ 2.00GHz
cpu MHz         : 1000.000
model name      : Intel(R) Core(TM)2 CPU          4400  @ 2.00GHz
cpu MHz         : 600.000

Oops, we are paying for 2 Core 2Ghz CPU that runs on 600Mhz on one core and 1000Mhz on another!!!

There will be other lines for all the CPUs/cores/threads, probably with the same values.

This feature is nice, if we are running workstation, but what I’ve noticed, we do have the same CPU throttling on Ubuntu Server 10.04 builds and on CentOS 5.3, 5.4 and 5.5 builds (thus on RedHat too).

After hours of digging google, I’ve found that:
– this problem is very common
– there are several bug reports about this issue
– this is not BIOS settings problem, because on dual boot systems, CPU runs at full speed on Windows
– there are no 100% working solutions or they are too difficult to find
– this is not a bug, but a ‘feature’ of the new kernels, it is implemented differently on 2.6.18 (CentOS) and 2.6.32 (Ubuntu).

Here is a tip how to disable it on running system:

1) Check that ‘kondemand’ thread is running, run as root: ย “pgrep -lf ondemand”

the output should be like:

# uname -a
Linux boston07 2.6.18-164.6.1.el5 #1 SMP Tue Nov 3 16:18:27 EST 2009 i686 i686 i386 GNU/Linux
# pgrep -lf ondemand
1444 kondemand/0
1445 kondemand/1

2) Check that current cpu speed differs from the maximum:

# grep -E '^model name|^cpu MHz' /proc/cpuinfo
model name      : Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz
cpu MHz         : 1596.000
model name      : Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz
cpu MHz         : 1596.000

3) Change CPU governor from ‘ondemand’ to ‘performance’ for all CPUs/cores, run as root:

for CPUFREQ in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do [ -f $CPUFREQ ] || continue; echo -n performance > $CPUFREQ; done

4) Check that your changes have been applied:

# grep -E '^model name|^cpu MHz' /proc/cpuinfo
model name      : Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz
cpu MHz         : 2394.000
model name      : Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz
cpu MHz         : 2394.000

5) If you are running ‘cpuspeed’, ‘cpufreqd’, ‘powerd’ or other daemons, that can control CPU stepping, just stop them, if you really need to run your system on 100% of the CPU speed.
On CentOS:

# service cpuspeed stop

6) On Linux 2.6.32 (On RedHat 6, and Oracle Unbreakable Linux 6) remove CPU scaling kernel modules:

# lsmod | grep ondemand
cpufreq_ondemand        8764  0 
freq_table              3751  2 cpufreq_ondemand,acpi_cpufreq
# rmmod cpufreq_ondemand acpi_cpufreq freq_table

Ensure that no ‘kondemand’ kernel threads are running:

# pgrep -lf kondemand
#

7) To make changes permanent (on reboot):

– On Ubuntu, modify /etc/init.d/ondemand script:

change this

echo -n ondemand > $CPUFREQ

to this:

echo -n performance > $CPUFREQ

OR ALTERNATIVELY just remove all references to ondemand from /etc/rc?.d/

rm -f /etc/rc?.d/S99ondemand

– On CentOS, just create a new script /etc/init.d/ondemand:

#! /bin/bash
#
# ondemand sets cpu govermor
#
# chkconfig: 2345 10 90
#
# description: Set the CPU Frequency Scaling governor to "performance"
#
### BEGIN INIT INFO
# Provides: $ondemand
### END INIT INFO

PATH=/sbin:/usr/sbin:/bin:/usr/bin

case "$1" in
    start)
        for CPUFREQ in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
        do
                [ -f $CPUFREQ ] || continue
                echo -n performance > $CPUFREQ
        done
        ;;
    restart|reload|force-reload)
        echo "Error: argument '$1' not supported" >&2
        exit 3
        ;;
    stop)
        ;;
    *)
        echo "Usage: $0 start|stop" >&2
        exit 3
        ;;
esac

then enable it:

chmod +x /etc/init.d/ondemand
chkconfig ondemand on
service ondemand restart

I’m using ‘ondemand’ name of the script, this may be a little bit misleading (because really it is a ‘performance’), but you may change it.

Originally posted 2016-01-22 17:22:09.

10 COMMENTS

  1. Some of my servers are responding to the script, but many report “Error: argument ‘restart’ not supported”. Any idea why some work and others don’t? I have HP servers that seem to work fine, but the Dell’s of various models do not. All servers are pxe loaded with the same image, software and kernel.

  2. I tried this. It worked, but then my server showed extremely high load averages, and subsequently crashed. There were other things going on, so this might not have been the cause, but after rebooting the server, the cores went back to the lower speed, and the load averages went back to normal. Again, correlation is not necessarily causation, but I am going to be careful.

  3. I have tried it, when i use service ondemand start it works good. When i place it into the chkconfig with chkconfig ondemand on it will start but not change the cpu speed. What do i wrong.
    I use Centos7 64bit
    Regards
    Pascal Schonewille

  4. The Linux CPU Frequency scaling governors are designed to quickly scale up the CPU frequency when increased load is detected. So maybe you may not actually want to do what is suggested here, especially if you pay your own power bill. Before you follow these steps, first try loading your server with something like stress, then check the CPU frequencies again. If they are now running at full speed and you think your server loads can handle the 100 ยตs ramp up time, you might consider leaving things as they are.

  5. Found this old thread and wanted to comment. There is some kind of bug
    on certain hardware or BIOSes that can result in the range of available
    frequencies being locked to maximum. It doesn’t happen on first boot,
    but after a few hours something happens and scaling_max_freq will =
    scaling_min_freq and can’t be changed at all. This happens no matter
    what governor is used (even performance), with the original cpufreq
    driver or the pstate driver. My motherboard is too old to have any more
    BIOS updated (and it’s not being BIOS limited anyway). The answer?
    Disabling SpeedStep entirely. I don’t care if Turbo doesn’t kick in,
    it’s still better to be running at 3.3GHz all the time instead of
    1.6GHz! So if you run into this bug, disable SpeedStep. Keep your fans
    clean and running, and make sure the intel_powerclamp module is loaded
    as this should take care of force-idling your CPU (and slowing it down
    yes) if it really does get dangerously hot.

LEAVE A REPLY

Please enter your comment!
Please enter your name here