Problems with the ondemand CPU frequency governor on a N2+

[BernieW]

I've built the 4.9.337+ kernel from sources and I'm running it on a N2+. I've tried to enable the ondemand CPU frequency governor to reduce power consumption but I get the following errors repeating in the syslog. Does the ondemand governor work?

Code: Select all

Aug 25 18:08:11 odroid kernel: [  242.712374@0] Call trace:
Aug 25 18:08:11 odroid kernel: [  242.712385@0] [<ffffff80090885ec>] __switch_to+0x9c/0xc0
Aug 25 18:08:11 odroid kernel: [  242.712394@0] [<ffffff8009c34714>] __schedule+0x284/0x7e0
Aug 25 18:08:11 odroid kernel: [  242.712401@0] [<ffffff8009c34cb0>] schedule+0x40/0xa8
Aug 25 18:08:11 odroid kernel: [  242.712410@0] [<ffffff80097e8b28>] cpufreq_freq_transition_begin+0x230/0x278
Aug 25 18:08:11 odroid kernel: [  242.712419@0] [<ffffff8009853e60>] meson_cpufreq_set_target+0x130/0x330
Aug 25 18:08:11 odroid kernel: [  242.712426@0] [<ffffff80097e907c>] __cpufreq_driver_target+0x174/0x4e8
Aug 25 18:08:11 odroid kernel: [  242.712431@0] [<ffffff80097eedf0>] od_dbs_timer+0x148/0x1a8
Aug 25 18:08:11 odroid kernel: [  242.712437@0] [<ffffff80097f300c>] dbs_work_handler+0x44/0x78
Aug 25 18:08:11 odroid kernel: [  242.712445@0] [<ffffff80090c68f0>] process_one_work+0x218/0x530
Aug 25 18:08:11 odroid kernel: [  242.712451@0] [<ffffff80090c6c58>] worker_thread+0x50/0x4f0
Aug 25 18:08:11 odroid kernel: [  242.712458@0] [<ffffff80090cda08>] kthread+0xf8/0x100
Aug 25 18:08:11 odroid kernel: [  242.712463@0] [<ffffff8009083750>] ret_from_fork+0x10/0x40
Aug 25 18:08:11 odroid kernel: [  242.712518@0] INFO: task cpufreq-set:3105 blocked for more than 120 seconds.
Aug 25 18:08:11 odroid kernel: [  242.721894@0]       Not tainted 4.9.337+ #8
Aug 25 18:08:11 odroid kernel: [  242.723375@0] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 25 18:08:11 odroid kernel: [  242.731081@0] cpufreq-set     D    0  3105   3098 0x00000000
Aug 25 18:08:11 odroid kernel: [  242.731090@0] Call trace:
Aug 25 18:08:11 odroid kernel: [  242.731108@0] [<ffffff80090885ec>] __switch_to+0x9c/0xc0
Aug 25 18:08:11 odroid kernel: [  242.731119@0] [<ffffff8009c34714>] __schedule+0x284/0x7e0
Aug 25 18:08:11 odroid kernel: [  242.731125@0] [<ffffff8009c34cb0>] schedule+0x40/0xa8
Aug 25 18:08:11 odroid kernel: [  242.731133@0] [<ffffff8009c35140>] schedule_preempt_disabled+0x28/0x40
Aug 25 18:08:11 odroid kernel: [  242.731141@0] [<ffffff8009c36a2c>] __mutex_lock_slowpath+0xdc/0x1b0
Aug 25 18:08:11 odroid kernel: [  242.731147@0] [<ffffff8009c36b64>] mutex_lock+0x64/0x78
Aug 25 18:08:11 odroid kernel: [  242.731157@0] [<ffffff80097f36a4>] cpufreq_dbs_governor_limits+0x2c/0xb0
Aug 25 18:08:11 odroid kernel: [  242.731167@0] [<ffffff80097ea078>] cpufreq_governor_limits.part.2+0x30/0x40
Aug 25 18:08:11 odroid kernel: [  242.731174@0] [<ffffff80097ebb48>] cpufreq_set_policy+0x400/0x440
Aug 25 18:08:11 odroid kernel: [  242.731180@0] [<ffffff80097ebd48>] store_scaling_governor+0x88/0xc8
Aug 25 18:08:11 odroid kernel: [  242.731186@0] [<ffffff80097e9e18>] store+0xb0/0xc8
Aug 25 18:08:11 odroid kernel: [  242.731195@0] [<ffffff80092db8a8>] sysfs_kf_write+0x58/0x80
Aug 25 18:08:11 odroid kernel: [  242.731201@0] [<ffffff80092dab1c>] kernfs_fop_write+0xfc/0x1e0
Aug 25 18:08:11 odroid kernel: [  242.731209@0] [<ffffff8009246f30>] __vfs_write+0x48/0x118
Aug 25 18:08:11 odroid kernel: [  242.731214@0] [<ffffff8009247f24>] vfs_write+0xac/0x1b0
Aug 25 18:08:11 odroid kernel: [  242.731219@0] [<ffffff80092495ac>] SyS_write+0x5c/0xc8
Aug 25 18:08:11 odroid kernel: [  242.731225@0] [<ffffff80090837c0>] el0_svc_naked+0x34/0x38
Aug 25 18:09:22 odroid systemd[1]: cpufrequtils.service: start operation timed out. Terminating.
Aug 25 18:09:22 odroid systemd[1]: cpufrequtils.service: Failed with result 'timeout'.
Aug 25 18:09:22 odroid systemd[1]: cpufrequtils.service: Unit process 3105 (cpufreq-set) remains running after unit stopped.
Aug 25 18:09:22 odroid systemd[1]: Failed to start LSB: set CPUFreq kernel parameters.

[mctom]

I've had problems with ondemand on N2+, namely the whole desktop froze for exact same period (7 seconds I think) when I was about to impose a sudden high CPU load (like for example start a youtube movie).
The dmesg said "PLL did not lock", which indicates that changing frequency did not work as expected. It may be a driver issue, not necessarily a hardware issue.
Which is an irony, as now I'm professionally designing microelectronic PLLs.

But let me add, on the bright side, ondemand does not provide any significant power saving at all. Working at full clock speed, ARM CPUs executing NOP (No Operation) are also very efficient. You're not losing much by working in Performance mode, and I'd encourage you to just execute less instead, if you really want to save power.

[odroid]

We didn't test 'ondemand' governor on Kernel 4.9.
It must be worth to try Kernel 5.15 or 6.1 for CPU governor tests.
viewtopic.php?f=52&t=44177

[BernieW]

I have tried the 6.1 Kernel and that seems to resolve the issue. However, it appears that the 6.1 kernel has increased the USB speed and therefore also increased power used by some USB devices. I am testing a SanDisk extreme pro 4TB drive. On the 4.9 kernel I get about 30 MiB/s when writing to the drive and on the 6.1 kernel I get about 320 MiB/s with the same drive. The increased speed would be nice if this drive would lower power when idle but it does not. I really only need about 20 MiB/s so the faster USB speed is not really gaining me anything here. Is there any way to limit the USB to high-speed rather than super-speed and, hopefully, lower power a bit?

By the way, I have disabled many devices that we do not need for our application which is an autonomous data acquisition and recording system. I've gotten idle power down to about 1.4W but with the 4TB drive drive installed, on the 6.1 kernel the idle power is about 4W and averages 5.4 W when recording. With the 4.9 kernel idle power was higher at about 2.1 W but recording was about 5.6 W. So It's better but could be lower if the drive was not a power pig.

[rooted]

You could try pinning the USB IRQ to a single small core or use a USB 2.0 cable or plug it into OTG which uses the USB 2.0 hub.

[BernieW]

I think your USB storage device worked in USB 2.0 (480mbps) mode with Kernel 4.9 while it is working in USB 3.0 (5Gbps) mode with Kernel 6.1.
It must be worth to compare lsusb -t outputs to narrow down the issue of the high power consumption.

[BernieW]

Strangely the only difference is I see is 6.1 uses the uas driver and 4.9 uses the usb-storage driver:

Code: Select all

root@odroid:/media/usb0# lsusb -t -v
/sys/bus/usb/devices/2-1/rx_lanes: No such file or directory
/sys/bus/usb/devices/2-1/tx_lanes: No such file or directory
/sys/bus/usb/devices/usb1/rx_lanes: No such file or directory
/sys/bus/usb/devices/usb1/tx_lanes: No such file or directory
/sys/bus/usb/devices/1-1.1/rx_lanes: No such file or directory
/sys/bus/usb/devices/1-1.1/tx_lanes: No such file or directory
/sys/bus/usb/devices/1-1/rx_lanes: No such file or directory
/sys/bus/usb/devices/1-1/tx_lanes: No such file or directory
/sys/bus/usb/devices/usb2/rx_lanes: No such file or directory
/sys/bus/usb/devices/usb2/tx_lanes: No such file or directory
/sys/bus/usb/devices/2-1.4/rx_lanes: No such file or directory
/sys/bus/usb/devices/2-1.4/tx_lanes: No such file or directory
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
        ID 05e3:0620 Genesys Logic, Inc.
        |__ Port 4: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
            ID 0781:55ae SanDisk Corp.
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        ID 05e3:0610 Genesys Logic, Inc. Hub
        |__ Port 1: Dev 3, If 0, Class=Vendor Specific Class, Driver=pl2303, 12M
            ID 067b:23a3 Prolific Technology, Inc.
root@odroid:/media/usb0# uname -a
Linux odroid 4.9.337-33 #1 SMP PREEMPT Thu Jul 20 18:05:01 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Code: Select all

root@odroid:/media/usb0$ lsusb -t -v
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
        ID 05e3:0620 Genesys Logic, Inc.
        |__ Port 4: Dev 3, If 0, Class=Mass Storage, Driver=uas, 5000M
            ID 0781:55ae SanDisk Corp.
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        ID 05e3:0610 Genesys Logic, Inc. Hub
        |__ Port 1: Dev 3, If 0, Class=Vendor Specific Class, Driver=pl2303, 12M
            ID 067b:23a3 Prolific Technology, Inc.
root@odroid:/media/usb0$ uname -a
Linux odroid 6.1.52+ #5 SMP PREEMPT Fri Sep 15 09:15:18 ADT 2023 aarch64 aarch64 aarch64 GNU/Linux

[BernieW]

Add usb-storage.quirks=0781:55ae:u to the kernel parameter to disable UAS.
viewtopic.php?p=346701#p346701

[BernieW]

Adding the kernel parameter did change the driver to usb-storage but there was no decrease in power used by the device. I was confused so I went back and rebuilt the 4.9 kernel to test it and, lo and behold, the device is using USB2 speed. This one also reduces the power consumption of the device by about 1W. Not sure what change I may have made made that limits it to USB2 speed. I'll have to see if the original issue with the ondemand governor still occurs with this build of 4.9.

Code: Select all

root@odroid:~$ lsusb -v -t
/sys/bus/usb/devices/usb1/rx_lanes: No such file or directory
/sys/bus/usb/devices/usb1/tx_lanes: No such file or directory
/sys/bus/usb/devices/1-1/rx_lanes: No such file or directory
/sys/bus/usb/devices/1-1/tx_lanes: No such file or directory
/sys/bus/usb/devices/usb2/rx_lanes: No such file or directory
/sys/bus/usb/devices/usb2/tx_lanes: No such file or directory
/sys/bus/usb/devices/1-1.4/rx_lanes: No such file or directory
/sys/bus/usb/devices/1-1.4/tx_lanes: No such file or directory
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/0p, 5000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        ID 05e3:0610 Genesys Logic, Inc. Hub
        |__ Port 4: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 480M
            ID 0781:55ae SanDisk Corp.
root@odroid:~$ uname -a
Linux odroid 4.9.337+ #1 SMP PREEMPT Tue Sep 19 13:29:03 ADT 2023 aarch64 aarch64 aarch64 GNU/Linux

[odroid]

Very weird, but glad to know where the difference in power consumption of one watt comes from.

[mctom]

The question is are you actually conserving energy this way. It may use less power but for a longer time.
A Smart Power 3 power plot and a numerical integral would give definitive answers here..

[BernieW]

I don't have a Smart Power 3 but I am logging voltage and current once per second using an Arduino with a Texas Instruments INA219 device. The average power when recording sensor data to the SanDisk SSD for this build of the 4.9 kernel is about 4.6W vs 5.9W for the stock 4.9 kernel and 5.6W for the 6.1 kernel.

Agreed that writing data faster to the SSD could be a win if the Sandisk SSD reduced power when it's idle but that does not seem to be happening. From the plots, instantaneous power varies between 3 and 9W for this 4.9 build and between 4 and 12W for the 6.1 kernel and the stock 4.9 kernel.

[mctom]

INA219 does not measure average power, it doesn't have integrating capabilities. You are measuring instantaneous power in essentially random time points.
INA series can easily operate at higher frequencies, which I suggest to max out and do a moving average on the software side. You will get a decent 4-digit resolution and actual 2-digit accuracy this way.

And USB SSD is by far not the most energy efficient storage option available.. I assume industrial SD card or eMMC are out of question?

Nevertheless, thanks for sharing your findings, it's very interesting how changing a driver might help.

[BernieW]

Thanks for the pointers on the INA219; I'll definitely try to measure at a higher frequency.

The reason we're using the USB SSD is for capacity; it's a 4TB drive. I don't think there is a eMMC module with that capacity and we'd need multiple SD cards to get the same capacity.

[mctom]

If this was my project, I'd probably look into designing a USB drive with 16 SD cards in it. Boy that would be fun to design.

If you can switch SSD off at the times of inactivity (if your application software knows when it is needed), there are methods of turning USB power off in N2+.
https://wiki.odroid.com/odroid-n2/troub ... /hub_reset
I would suggest unmounting and mounting this drive back in a clean manner accordingly.