4/5 M1 odroids stopped working at the same time

Post Reply
disharmony
Posts: 12
Joined: Sun Sep 18, 2022 6:56 pm
languages_spoken: english
Has thanked: 0
Been thanked: 1 time
Contact:

4/5 M1 odroids stopped working at the same time

Post by disharmony »

I am a bit flabbergasted, I have 5 odroids running in a cluster and since it's a test cluster I frequently shut them down.

Today I decided to do a reinstall, so I turned them off by pulling the power while they were running. And all 5 odroids were running fine.

I then hooked up a keyboard, hdmi and a USB stick, put power back on and the device didn't start. Nothing, no HDMI, no LED"s except for the orange ethernet LED and the onboard led that is supposed to be red is burning just a little bit, but it appears to be almost completely off.

Then I took the 2nd device, working fine. No problems at all.

Then I took the 3rd, 4th and 5th and they all are having the same issues as the 1st.

I tried to hookup the uart, but only get garbage. Which is weird, I used the UART in the past with success, baudrate and settings are set to 1500000 and appear all correct, but nothing than garbage. Also the working device doesn't give me a proper UART output. (After spending a day I am a bit tired now, will try again tomorrow but can't see what could be wrong).

Anybody any suggestions ? What is weird is tha the red led is illuminting for like 1%, it's extremely dimm. I tried the PWR and reset pins, but all without luck. Devices have been running for +/- 2 years.

User avatar
rooted
Posts: 10612
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 821 times
Been thanked: 733 times
Contact:

Re: 4/5 M1 odroids stopped working at the same time

Post by rooted »

Your power supply may have had a voltage spike when unplugged, it would be about the only thing to explain this.

User avatar
tobetter
Posts: 13094
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: Many
Location: Paju, South Korea
Has thanked: 996 times
Been thanked: 2275 times
Contact:

Re: 4/5 M1 odroids stopped working at the same time

Post by tobetter »

@disharmony,

Firstly, connect one of ODROID-M1 to your PC using micro USB cable and see if your PC can see your ODROID-M1 as a new USB device when it is powered. It would be good if you can run lsusb in Linux PC and share the output of it.

Secondly, flash the image in the link below to SD card and try to boot from it with your ODROID-M1. If your ODROID-M1 shows Petitboot screen over HDMI display, then I think your ODROID-M1 is not fried yet, probably SPI flash memory is erased so you can recover it.

Lastly, do you know what OS image is installed to your ODROID-M1 and what's the boot media?

disharmony
Posts: 12
Joined: Sun Sep 18, 2022 6:56 pm
languages_spoken: english
Has thanked: 0
Been thanked: 1 time
Contact:

Re: 4/5 M1 odroids stopped working at the same time

Post by disharmony »

Hi,

the broken device using USB is listed as

Bus 001 Device 005: ID 2207:350a Fuzhou Rockchip Electronics Company

I compiled rkdeveloptool but all the info commands seem failing.

However I hooked up another ttl usb converter and the UART port does work now. The output I get is:

Code: Select all

....
U-Boot SPL board init
U-Boot SPL 2017.09 (Mar 06 2022 - 16:29:16)
Trying to boot from MMC1
MMC error: The cmd index is 0, ret is -110
mmc_init: -110, time 4
spl: mmc init failed with error: -110
Trying to boot from MMC2
MMC error: The cmd index is 1, ret is -110
Card did not respond to voltage select!
mmc_init: -95, time 9
spl: mmc init failed with error: -95
Trying to boot from MTD2
No misc partition
## Verified-boot: 0
No matching DT out of these options:

No matching DT out of these options:

No matching DT out of these options:

SPL: failed to boot from all boot devices
### ERROR ### Please RESET the board ###
# Reset the board to bootrom #
I have skipped the first part of the output, but can attach the complete dump if that helps.

You mentioned a link but I didn't see any, can you share it again ?

User avatar
tobetter
Posts: 13094
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: Many
Location: Paju, South Korea
Has thanked: 996 times
Been thanked: 2275 times
Contact:

Re: 4/5 M1 odroids stopped working at the same time

Post by tobetter »

disharmony wrote:
Mon Apr 01, 2024 5:44 pm
Hi,

the broken device using USB is listed as

Bus 001 Device 005: ID 2207:350a Fuzhou Rockchip Electronics Company

I compiled rkdeveloptool but all the info commands seem failing.

However I hooked up another ttl usb converter and the UART port does work now. The output I get is:

Code: Select all

....
U-Boot SPL board init
U-Boot SPL 2017.09 (Mar 06 2022 - 16:29:16)
Trying to boot from MMC1
MMC error: The cmd index is 0, ret is -110
mmc_init: -110, time 4
spl: mmc init failed with error: -110
Trying to boot from MMC2
MMC error: The cmd index is 1, ret is -110
Card did not respond to voltage select!
mmc_init: -95, time 9
spl: mmc init failed with error: -95
Trying to boot from MTD2
No misc partition
## Verified-boot: 0
No matching DT out of these options:

No matching DT out of these options:

No matching DT out of these options:

SPL: failed to boot from all boot devices
### ERROR ### Please RESET the board ###
# Reset the board to bootrom #
This log exaplin me that the firmware in SPI is broken somehow and your have any other bootloader in SD card or eMMC.
I have skipped the first part of the output, but can attach the complete dump if that helps.

You mentioned a link but I didn't see any, can you share it again ?
I used to forget to add the link, so...here it is. :)
https://ppa.linuxfactory.or.kr/images/p ... 304.img.xz

disharmony
Posts: 12
Joined: Sun Sep 18, 2022 6:56 pm
languages_spoken: english
Has thanked: 0
Been thanked: 1 time
Contact:

Re: 4/5 M1 odroids stopped working at the same time

Post by disharmony »

Many thanks for the link.

With that SD image the device boots.

So I also downloaded the spiboot-20240109.img, renamed that to spiboot.img and got it flashed back in.
I repeated that 5 times (have to re-copy the spiboot.img over after every device is flashed because it's deleted automatically to prevent a flash loop), and now all 4 devices are working again!

They boot fine, without SD after the flashing. I just wonder what could be the cause of this and if the internal storage might be bad.

Anyway, I am very happy now! Thank you for the quick replies!!!!

User avatar
tobetter
Posts: 13094
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: Many
Location: Paju, South Korea
Has thanked: 996 times
Been thanked: 2275 times
Contact:

Re: 4/5 M1 odroids stopped working at the same time

Post by tobetter »

disharmony wrote:
Mon Apr 01, 2024 6:42 pm
Many thanks for the link.

With that SD image the device boots.

So I also downloaded the spiboot-20240109.img, renamed that to spiboot.img and got it flashed back in.
I repeated that 5 times (have to re-copy the spiboot.img over after every device is flashed because it's deleted automatically to prevent a flash loop), and now all 4 devices are working again!

They boot fine, without SD after the flashing. I just wonder what could be the cause of this and if the internal storage might be bad.

Anyway, I am very happy now! Thank you for the quick replies!!!!
Glad to hear your ODROID-M1 are back.

Can you please let me know the output of uname -a after Linux is started, also would be nice if you have /dev/mtd* in Linux, not Petitboot, as well?
Practically the problem happens when the firmware in SPI flash memory is corrupted so, your ODROID-M1 is not able to load bootloader. This happens, IMO, when SPI flash memory is physically corrupted or user space app in Linux intended to remove it.

disharmony
Posts: 12
Joined: Sun Sep 18, 2022 6:56 pm
languages_spoken: english
Has thanked: 0
Been thanked: 1 time
Contact:

Re: 4/5 M1 odroids stopped working at the same time

Post by disharmony »

I just reflashed the odroid's, so I don't have the old uname -a output anymore, just the new one:

Linux odroid01 6.1.0-odroid-arm64 #1 SMP PREEMPT Wed, 21 Feb 2024 05:58:56 +0000 aarch64 aarch64 aarch64 GNU/Linux

I don't have the /dev/mtd* devices, I could look into the mtd driver if I can get those devices working.

The strange thing is that yesterday, prior to shutting the odroids down I removed power of the powerblock a couple of hours before, and all of them came back online without any problem. After that I removed power one-by-one, not all at once, and suddenly 4 out of 5 had this issue.

So I think I can rule out that something in userspace corrupted the data since between that time I didn't actually do anything on the devices.

Fortunately I know what to do if it happens again, and if it does happen again I will drop a message here as well to let you know.
These users thanked the author disharmony for the post:
tobetter (Mon Apr 01, 2024 9:23 pm)

Post Reply

Return to “Issues”

Who is online

Users browsing this forum: No registered users and 1 guest