ODROID Xu4 Suddenly Heats up and Shuts Down

[hop]

I have been kicking this around for better part of a week and still have no solutions.

Suddenly, for no apparent reason, my ODROID shuts down due to thermal issues. It does not feel hot at all, but adding a 120mm fan nearby on high seems to allow it to run longer before failing.
It has the passive cooled HSF with the blue fan. Kernel is 4.14.55-133.

Code: Select all

Linux odroid 4.14.55-133 #1 SMP PREEMPT Sat Jul 14 18:18:06 UTC 2018 armv7l armv7l armv7l GNU/Linux

I wrote a script that shows the 5 temperatures every second. Although I see sudden fluctuations, I do not understand how the sensors can go so high so fast. Is this a software issue? Bad temp sensors? Bad sensor reading process?
Here is a section of that output before it shutdown again. The last entry is the last line shown before shutdown. Honestly, how does a temp sensor go from 52 Celsius to 119 Celsius in one second?? Or 40 to 95??? Below this is the kernel log. I will include the whole thing if needed.
Right now my usually VERY stable ODROID is completely unusable. I hope there is a fix. With the passive HSF set to 100% manually to always be on AND a 120mm 12vdc cooling fan blowing on it at max RPM's, I am at a loss as what else I can do. Also, using CPUFREQ, the governor is set to conservative.

Code: Select all

2019-02-03 16:04:54 30000 33000 37000 32000 31000
2019-02-03 16:04:55 31000 34000 48000 35000 31000
2019-02-03 16:04:56 36000 36000 42000 45000 33000
2019-02-03 16:04:57 37000 37000 43000 46000 34000
2019-02-03 16:04:58 38000 38000 45000 47000 34000
2019-02-03 16:04:59 39000 38000 45000 47000 35000
2019-02-03 16:05:00 38000 38000 45000 49000 35000
2019-02-03 16:05:01 47000 39000 45000 42000 36000
2019-02-03 16:05:02 50000 41000 48000 52000 38000
2019-02-03 16:05:03 47000 41000 48000 47000 38000
2019-02-03 16:05:04 52000 43000 50000 47000 39000
2019-02-03 16:05:05 119000 41000 47000 43000 38000
2019-02-03 16:05:06 53000 39000 49000 45000 36000
2019-02-03 16:05:07 48000 39000 45000 44000 36000
2019-02-03 16:05:08 43000 37000 43000 40000 35000
2019-02-03 16:05:09 37000 37000 44000 38000 35000
2019-02-03 16:05:10 37000 37000 43000 38000 35000
2019-02-03 16:05:11 37000 37000 44000 38000 34000
2019-02-03 16:05:12 37000 37000 42000 38000 34000
2019-02-03 16:05:13 37000 36000 42000 38000 34000
2019-02-03 16:05:14 37000 36000 41000 38000 34000
2019-02-03 16:05:15 37000 37000 45000 39000 34000
2019-02-03 16:05:16 35000 37000 46000 41000 34000
2019-02-03 16:05:17 38000 37000 44000 38000 35000
2019-02-03 16:05:18 40000 37000 43000 42000 35000
2019-02-03 16:05:19 95000 40000 47000 43000 35000

kernel log

Code: Select all

Feb  3 15:45:30 odroid kernel: [ 6123.131341] mmc_host mmc0: Bus speed (slot 0) = 88888889Hz (slot req 52000000Hz, actual 44444444HZ div = 1)
Feb  3 15:45:30 odroid kernel: [ 6123.131922] mmc_host mmc0: Bus speed (slot 0) = 200000000Hz (slot req 200000000Hz, actual 200000000HZ div = 0)
Feb  3 15:45:30 odroid kernel: [ 6123.133241] mmc_host mmc0: Bus speed (slot 0) = 50000000Hz (slot req 52000000Hz, actual 50000000HZ div = 0)
Feb  3 15:45:30 odroid kernel: [ 6123.133659] mmc_host mmc0: Bus speed (slot 0) = 266666666Hz (slot req 200000000Hz, actual 133333333HZ div = 1)
Feb  3 16:05:20 odroid kernel: [ 7312.972927] thermal thermal_zone0: critical temperature reached (116 C), shutting down
Feb  3 16:15:42 odroid kernel: [    0.000000] Booting Linux on physical CPU 0x100
Feb  3 16:15:42 odroid kernel: [    0.000000] Linux version 4.14.55-133 (root@1604_builder_armhf) (gcc version 5.4.1 20160904 (Ubuntu/Linaro 5.4.1-2ubuntu1~16.04)) #1 SMP PREEMPT Sat Jul 14 18:18:06 UTC 2018
Feb  3 16:15:42 odroid kernel: [    0.000000] CPU: ARMv7 Processor [410fc073] revision 3 (ARMv7), cr=10c5387d

Also thought I would include this... is this normal?

Code: Select all

Feb  3 16:15:42 odroid kernel: [    3.734149] thermal thermal_zone0: failed to read out thermal zone (-22)
Feb  3 16:15:42 odroid kernel: [    3.740363] thermal thermal_zone1: failed to read out thermal zone (-22)
Feb  3 16:15:42 odroid kernel: [    3.747032] thermal thermal_zone2: failed to read out thermal zone (-22)
Feb  3 16:15:42 odroid kernel: [    3.753706] thermal thermal_zone3: failed to read out thermal zone (-22)
Feb  3 16:15:42 odroid kernel: [    3.760430] thermal thermal_zone4: failed to read out thermal zone (-22)

[rooted]

It's one of the thermal sensors malfunctioning, I don't think it's software.

As a workaround you can echo a fake temperature which will keep it from shutting down. I can tell you how when I get home.

[rooted]

You can do this:

Code: Select all

echo 60000 | sudo tee /sys/devices/virtual/thermal/thermal_zone0/emul_temp

Thermal will not work correctly but as it isn't already at least it will let your device function.

[hop]

Thank you for the fix. I had read about this elsewhere but was worried about the device actually burning up. Is there any way to tell if temperature is also a problem? I guess if the temperature sensors are in close proximity, they would be affected and show high also, and they are not showing high at all except they jump 10°c within a second sometimes. Nothing like zone0 though.

Is there any chance of repairing that temperature sensor? I have a SMD rework station and have worked with SMD before. The sensors are probably built into the CPU or between the board and the CPU I'll bet though. Not worth it if that is the case. Since my fan is on all the time now, I could probably add a one-wire digital sensor and monitor overall heat that way.

Been running for an hour now and the other temps are fairly low and stable. Thank you!

Code: Select all

2019-02-04 03:53:00 60000 26000 29000 24000 25000
2019-02-04 03:53:01 60000 26000 29000 24000 25000
2019-02-04 03:53:02 60000 30000 34000 31000 28000
2019-02-04 03:53:03 60000 32000 36000 34000 30000
2019-02-04 03:53:04 60000 33000 38000 35000 31000
2019-02-04 03:53:05 60000 34000 39000 36000 31000
2019-02-04 03:53:06 60000 34000 38000 36000 31000
2019-02-04 03:53:07 60000 36000 42000 45000 32000
2019-02-04 03:53:08 60000 37000 44000 43000 33000
2019-02-04 03:53:09 60000 35000 44000 43000 32000
2019-02-04 03:53:10 60000 32000 36000 33000 30000
2019-02-04 03:53:11 60000 34000 41000 39000 31000
2019-02-04 03:53:12 60000 33000 38000 34000 31000
2019-02-04 03:53:13 60000 30000 34000 29000 29000
2019-02-04 03:53:14 60000 29000 32000 27000 28000
2019-02-04 03:53:15 60000 28000 31000 27000 27000
2019-02-04 03:53:16 60000 28000 31000 26000 27000
2019-02-04 03:53:17 60000 27000 31000 26000 27000
2019-02-04 03:53:19 60000 27000 30000 26000 26000
2019-02-04 03:53:20 60000 27000 30000 26000 26000
2019-02-04 03:53:21 60000 27000 30000 25000 26000
2019-02-04 03:53:22 60000 27000 30000 26000 26000
2019-02-04 03:53:23 60000 27000 30000 25000 26000
2019-02-04 03:53:24 60000 27000 30000 25000 26000
2019-02-04 03:53:25 60000 27000 30000 25000 26000
2019-02-04 03:53:26 60000 27000 30000 25000 26000
2019-02-04 03:53:27 60000 27000 30000 25000 26000
2019-02-04 03:53:28 60000 27000 29000 25000 26000
2019-02-04 03:53:29 60000 27000 30000 25000 26000

[memeka]

You can delete 2Ghz and 1.9Ghz nodes from arch/arm/boot/dts/exynos5800.dtsi, then make dtbs and overwrite your odroidxu4 dtb to keep the cores at max 1.8Ghz, you should not have issues there,
Or you can configure cpufrequtils in ubuntu (sudo apt-get install cpufrequtils) to set max freq to 1.8Ghz, edit /etc/default/cpufrequtils

Code: Select all

ENABLE="true"
GOVERNOR="ondemand"
MAX_SPEED=1800000

[rooted]

Is there any chance of repairing that temperature sensor? I have a SMD rework station and have worked with SMD before. The sensors are probably built into the CPU or between the board and the CPU I'll bet though. Not worth it if that is the case. Since my fan is on all the time now, I could probably add a one-wire digital sensor and monitor overall heat that way.

I really don't know but I have to assume they are part of the SoC and can't be repaired.

[hop]

Thank you Rooted and Memeka, I appreciate it!

The unit has been on now for 7 hours and no signs of stress, or fire, which is good!

There is still the unsolved mystery with the unregistered temp sensors. I read in another thread that it was a kernel bug and was supposedly fixed in 4.14. Obviously it is not with my version of 4.14 so still wondering if that has any effect on my boot and shut down issue.

Code: Select all

Feb  3 16:15:42 odroid kernel: [    3.734149] thermal thermal_zone0: failed to read out thermal zone (-22)
Feb  3 16:15:42 odroid kernel: [    3.740363] thermal thermal_zone1: failed to read out thermal zone (-22)
Feb  3 16:15:42 odroid kernel: [    3.747032] thermal thermal_zone2: failed to read out thermal zone (-22)
Feb  3 16:15:42 odroid kernel: [    3.753706] thermal thermal_zone3: failed to read out thermal zone (-22)
Feb  3 16:15:42 odroid kernel: [    3.760430] thermal thermal_zone4: failed to read out thermal zone (-22)

By the way, I also realized that the temp override did not persist through reboot. I had to go through a few reboots until I had enough time to shell in and pasted the fix. How can I make that permanent?

Thanks again for all your help and getting my machine back to running again!!

[rooted]

You can add it to /etc/rc.local I believe

I know about the bug but this seems different since your kernel revision is up to date, just bump this thread after around the 8th since Hardkernel is on Lunar New Year's holiday

[hop]

Will do, thank you for the heads up!

And I like your avatar. Léon, The Professional is one of my favorite Luc Bessen movies, one of my favorites overall. Gary Oldman was phenomenal, and those shades Jean Reno is sporting are the coolest hands down. "No women no kids"

[hop]

Rooted, I figured out how to make your fix stick, meaning persistent, when I reboot my Odroid Xu4. I needed to write a bash script, save it in /etc/init.d/, make it executable, and then run sudo update-rc.d myscriptname.sh defaults. When I rebooted AND power-cycled, the zone0 temp override persisted. To prevent the error that is thrown from update-rc.d about "missing LSB tags and overrides" I just included the header tags from the /etc/init.d/skeleton script at the beginning of my bash script. Changing skeleton to the name of my script to be compliant. Doing this fixed the error being thrown.

Code: Select all

### BEGIN INIT INFO
# Provides:          skeleton
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Example initscript
# Description:       This file should be used to construct scripts to be
#                    placed in /etc/init.d.  This example start a
#                    single forking daemon capable of writing a pid
#                    file.  To get other behavoirs, implemend
#                    do_start(), do_stop() or other functions to
#                    override the defaults in /lib/init/init-d-script.
### END INIT INFO

Thank you sir for your guidance and I appreciate it!

[rooted]

That's the proper way, but /etc/init.rc would also work

I love Leon: The Professional, but being a SciFi buff The Fifth Element is my favorite Luc Besson film. Jean Reno is top notch, he has an effortless way about him that makes him stand out in a sea of actors.

[hop]

I love Leon: The Professional, but being a SciFi buff The Fifth Element is my favorite Luck Besson film. Jean Reno is top notch, he has an effortless way about him that makes him stand out in a sea of actors.

Ya, true. I've seen 5th Element about a thousand times and love it every time. Only thing missing is Jean Reno. And I saw him again in The DaVinci Code and it was weird seeing him on the other side of the law. Also liked him in the first Mission Impossible but still hate the stupid helicopter through the tunnel chase. Jean deserved a better end than that.

"That's the proper way, but /etc/init.rc would also work " I try to keep it compliant when I can. Quick fixes seem to be my undoing, eventually. It's the reason I feel dirty with the zone0 temp override. I would have rather fixed it. Can't help but feel there is dead hardware in my media server creation. Sigh.

[mad_ady]

What, you mean that's not how @rooted looks in real life? Added the professional to my watch list

[rooted]

What, you mean that's not how @rooted looks in real life? Added the professional to my watch list

Nope

Me a while back, hat was crooked because I leaned back on the couch... not a hipster.



@mad_ady

Do you know what patch fixed a similar issue to this?

[hop]

Here is the thread that addresses the "failed to readout thermal zone" issue. At the end of the thread, odriod, the site admin states that the errors are normal. Sigh. I guess that closes the book on a software solution. I guess I am stuck with an Odroid Xu4 with a busted temp sensor that has to be bypassed. The unit is still running after the fixes I mentioned above in this thread, but I don't like it. Not enough to keep me from buying any more of these, but I hope they address it in the manufacturing process and fix it.
viewtopic.php?t=31192

[memeka]

@hop - the soc sensor is manufactured by Samsung, not hardkernel.
Also, I believe you are the 3rd or 4th case I have heard of with damaged thermal sensor, since 4(5?) years ago when HK started using this soc. I have 5 XU4 and 1 XU3, and none have had this issue

[hop]

Which makes me a pretty unlucky individual to say the least. I have to say though, writing in the ability to "spoof" a temperature was rather convenient to bypass the issue. Perhaps the designers had the foresight to envision a scenario where a temperature sensor would fail. Since I do plan to buy a few more of these, and maybe a few C1's (one unit per drive as opposed to my current setup of powered USB3 ports with 4 drives), I will be ready if this issue surfaces yet again.

Please know, I'm not mad at all. The fact that the designers saw fit to include temp sensors on cores in the first place shows the level of engineering that went into this awesome board. I would not have even expected the inclusion and it makes me even more of a fan!

Thanks guys for the help and incite!! I certainly know where to go for my issues in the future. You guys are a blessing for us end-users, and I appreciate it.

[odroid]

There was a Kernel patch to ignore a temperature value from a broken sensors long time ago.
But this is the first time I hear all five sensors were broken.

Which OS image do you use?
Please try the latest Ubuntu 18.04 Ubuntu image to confirm whether the sensors are really damaged or not.
https://wiki.odroid.com/odroid-xu4/os_i ... 4/20181203

[hop]

I do not understand why my kernel is 4.14.55-133 instead of 4.14.85 as your link suggests. I apologize if I missed something obvious. My Ubuntu is the version you suggested. Is it because my install is headless? I do not use the HDMI at all.

Also, the other 4 temp sensors seem to be fine, or did you see something wrong in the output I posted at the beginning of this thread? Only zone0 was acting erratic. Thanks for the response sir!

EDIT: Previous to this post, I did try an Ubuntu 18.04 with kernel 4.14 minimal image on an SD card, flipped the switch and booted with it, and it shutdown due to overheat on zone0.

Code: Select all

Ubuntu 18.04.1 LTS (GNU/Linux 4.14.55-133 armv7l)
~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic

~# hostnamectl
   Static hostname: odroid
         Icon name: computer
        Machine ID: dc87f36fc06c441a85ff7269ba4d50fb
           Boot ID: 06f0a6fff6f8422a931b96266f388168
  Operating System: Ubuntu 18.04.1 LTS
            Kernel: Linux 4.14.55-133
      Architecture: arm

[odroid]

4.14.55 should be fine.
But you can update it with this instruction.
https://wiki.odroid.com/odroid-xu4/os_i ... st-upgrade

BTW, what is the values from five temp sensors?

Code: Select all

odroid@odroid:~$ uname -a
Linux odroid 4.14.94-155 #1 SMP PREEMPT Wed Jan 23 05:19:08 -02 2019 armv7l armv7l armv7l GNU/Linux

odroid@odroid:~$ cat /sys/devices/virtual/thermal/thermal_zone*/temp
42000
40000
41000
41000
36000 

[hop]

Sorry, that was the output of my bash script. It's the same thing except the time and date was added, and the temps were placed side by side instead of in rows.
Also now that I look at it, I read it from a different path. They should be the same though shouldn't they? Even if they are not, there was still a kernel shutdown by a high read on zone0. I copied my script to temps2 and changed the path to what you showed and I got the same type of numbers.

Code: Select all

~/scripts# ./temps
2019-02-07 14:06:30 60000 29000 33000 28000 28000
2019-02-07 14:06:31 60000 29000 33000 28000 28000
2019-02-07 14:06:32 60000 29000 33000 28000 28000
2019-02-07 14:06:33 60000 29000 33000 28000 28000
^C
~/scripts# cp temps temps2
~/scripts# nano temps2
~/scripts# ./temps2
~/scripts# ./temps2
2019-02-07 14:12:49 60000 29000 32000 27000 27000
2019-02-07 14:12:50 60000 29000 32000 27000 28000
2019-02-07 14:12:51 60000 29000 33000 27000 27000
2019-02-07 14:12:52 60000 29000 32000 27000 28000

Copied script with your path used instead...

Code: Select all

#!/usr/bin/env bash

while true;
do
 temps=$( cat /sys/devices/virtual/thermal/thermal_zone*/temp );
 date=$(date +"%Y-%m-%d %T") ;
 echo $date $temps ;
 sleep 1 ;
done

What I used to get the output at the beginning of this post.

Code: Select all

~/scripts# cat temps
#!/usr/bin/env bash

while true;
do
 temps=$( cat /sys/class/thermal/thermal_zone*/temp );
 date=$(date +"%Y-%m-%d %T") ;
 echo $date $temps ;
 sleep 1 ;
done

[rooted]

It seems it's just thermal_zone0 that's faulty.

[odroid]

I think one of five temperature sensor seems to be broken too.

[hop]

Thanks guys for clarifying. Since rooted's fix, it stays up now. Would be a longer time but I manually restarted for other reasons.
~# uptime
14:43:41 up 3 days, 22:33, 1 user, load average: 0.44, 0.36, 0.20

[mopedfahrer]

Hello,
just for information, that there are still more XU4 with critical-temperture-shutdown "into the wild".
My XU4, bought in Aug. 2015 was always running with Open Media Vault NAS, without any high load, and started some weeks ago to shutdown after critical temperature messages in zone0 and zone 1...

[odroid]

@mopedfahrer,
Which Kernel and OMV and Debian version do you run?

[mopedfahrer]

Hello @odroid,
my uname says:

Linux 4.14.150-odroidxu4 #2 SMP PREEMPT Mon Oct 28 08:07:45 CET 2019 armv7l GNU/Linux
root@nas-200:/#

and my OMV version is :

Release: 5.5.23-1
Codename: Usul

I tried yesterdays the command line of @rooted to override temporarily the sensor data, but also ordered a new xu4...

brds, mopedfahrer

[mopedfahrer]

@odroid By the way, I had some terminal windows open, running the armbian monitoring and htop tool. Then the alert messages came, as possible to see in the picture. But also can be recogniced, that the cpu cores had nothing to do...

[joy]

But also can be recogniced, that the cpu cores had nothing to do...

Staying in just IDLE state after booting without any load, it's common to get very small percentage values of all cores like that (values after decimal point).
I also have the values of 0~3% in idle state, keep 2~3% for a single core like core 5.

BTW, even in case of small load, you have the warning message 'critical temperature reached (115 C)'.

Early versions of kernel 4.14 had some serious stability issue,
so, could you update the kernel package using this command?
viewtopic.php?p=219329#p219329

Code: Select all

$ sudo apt update -y
$ sudo apt-get install linux-image-xu3

[mopedfahrer]

Hello @joy

the picture does not show values from booting, but the last one's before shutdown

the "sudo apt-get install linux-image-xu3" shows "E: Unable to locate package linux-image-xu3"

Are you sure, that "4.14.29-125 from 2018" from your viewtopic post is newer than my Kernel from "Linux 4.14.150-odroidxu4 #2 SMP PREEMPT Mon Oct 28 08:07:45 CET 2019 armv7l GNU/Linux" ?

[joy]

Are you sure, that "4.14.29-125 from 2018" from your viewtopic post is newer than my Kernel from "Linux 4.14.150-odroidxu4 #2 SMP PREEMPT Mon Oct 28 08:07:45 CET 2019 armv7l GNU/Linux" ?

No. Your version is newer.
I needed to check the problematic version but I missed that part. That's my mistake.
The version 4.14.150 is not too old version.

[mopedfahrer]

So, finally my problem is hopefully solved with a new XU4-device...
Thanks for all the technical background here.