[solved] h4 ultra; NICs link refuse to go up in OS; work in uefi

Post Reply
martin00
Posts: 96
Joined: Wed Jan 02, 2019 7:10 pm
languages_spoken: english
ODROIDs: h2, h3+, h4 ultra
Location: /dev/earth0
Has thanked: 18 times
Been thanked: 18 times
Contact:

[solved] h4 ultra; NICs link refuse to go up in OS; work in uefi

Post by martin00 »

Yesterday I ran into a problem I'm not able to get out of. No matter what OS I boot NIC link stays down:

Code: Select all

node04(~)# dmesg |grep igc
[    4.148882] igc 0000:01:00.0: PTM enabled, 4ns granularity
[    4.202936] igc 0000:01:00.0 (unnamed net_device) (uninitialized): PHC added
[    4.226901] igc 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x1 link)
[    4.234161] igc 0000:01:00.0 eth0: MAC: 00:1e:06:45:6b:0f
[    4.252733] igc 0000:02:00.0: PTM enabled, 4ns granularity
[    4.308196] igc 0000:02:00.0 (unnamed net_device) (uninitialized): PHC added
[    4.333903] igc 0000:02:00.0: 4.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x1 link)
[    4.343923] igc 0000:02:00.0 eth1: MAC: 00:1e:06:45:6b:10
[    4.516686] igc 0000:02:00.0 enp2s0: renamed from eth1
[    4.546666] igc 0000:01:00.0 enp1s0: renamed from eth0
[   12.180824] igc 0000:01:00.0 enp1s0: entered allmulticast mode
[   12.180892] igc 0000:01:00.0 enp1s0: entered promiscuous mode
[   49.005040] igc 0000:01:00.0 enp1s0: left allmulticast mode
[   49.010693] igc 0000:01:00.0 enp1s0: left promiscuous mode
node04(~)#

node04(~)# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1e:06:45:6b:0f brd ff:ff:ff:ff:ff:ff
3: enp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1e:06:45:6b:10 brd ff:ff:ff:ff:ff:ff
node04(~)#

node04(~)# ip link set enp1s0 up
node04(~)# ip link set enp2s0 up
node04(~)#
node04(~)# ip -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128
enp1s0           DOWN
enp2s0           DOWN
node04(~)#
This is true for FreeBSD, Linux and Windows I tested. Linux example here is from proxmox, I've destroyed default bridge, that's why it has additional msgs about entering/leaving modes).
Network connection itself is OK, these NICs do work in UEFI. I can PXE boot over them without problem.
UEFI version is ADLN-H4_B1.02.

I tried:
- boot different OSs, different versions of Linux
- power off, remove battery, let it stay unplugged for longer time (over hour or so)
- reset to defaults in UEFI

I thought maybe ASPM has something to do with it ? Well the config is the default one:

Code: Select all

                               Aptio Setup - AMI
                    Chipset
┌────────────────────────────────────────────────────┬─────────────────────────┐
│  PCI Express Configuration                        ▲│The control of Active    │
│                                                   █│State Power Management   │
│  DMI Link ASPM Control    [Auto]                  █│of the DMI Link.         │
│                                                   █│                         │
│  PCI Express Root Port    Lane configured as      █│                         │
│  1                        USB/SATA/UFS            █│                         │
│  PCI Express Root Port    Lane configured as      █│                         │
│  2                        USB/SATA/UFS            █│                         │
│► PCI Express Root Port 3                          █│                         │
│► PCI Express Root Port 4                          █│─────────────────────────│
│  PCI Express Root Port    Not present in this SKU █│→←: Select Screen        │
│  5                                                █│↑↓: Select Item          │
│  PCI Express Root Port    Not present in this SKU ░│Enter: Select            │
│  6                                                ░│+/-: Change Opt.         │
│► PCI Express Root Port 7                          ░│F1: General Help         │
│  PCI Express Root Port    Not present in this SKU ░│F2: Previous Values      │
│  8                                                ░│F3: Optimized Defaults   │
│► PCI Express Root Port 9                          ▼│F4: Save & Exit          │
│                                                    │ESC: Exit                │
└────────────────────────────────────────────────────┴─────────────────────────┘
                    Version 2.22.1289 Copyright (C) 2024 AMI
                    
ASPM is disabled for all ports I can configure.

This happened when I wanted to configure windows2go and used this box. Windows 10 booted, I installed missing drivers. NICs came up but had links down. Since then I was not able to bring them up in any OS. I cannot say it's because of it but issue occurred after it.

During my troubleshooting (i.e.after this issue occurred) I noticed two other issues:
- when serial console is plugged in system doesn't want to boot from powered off state
I double-checked the connections and voltages (3.3)
even if h4 is without power amber LED is on when serial console is plugged in
- nvme card is randomly found/not found across reboots. Card is firmly slotted and screwed
Last edited by martin00 on Fri Jun 14, 2024 4:58 pm, edited 1 time in total.

User avatar
odroid
Site Admin
Posts: 42543
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 3735 times
Been thanked: 2103 times
Contact:

Re: h4 ultra; NICs link refuse to go up in OS; work in uefi

Post by odroid »

I believe you already tried Ubuntu 24.04 + stock kernel 6.8.
There seems to be a weird electricity floating ground issue somewhere in the AC power path.

BTW, did you connect only 3 wires (Rx, Tx, GND) for serial console interface?

martin00
Posts: 96
Joined: Wed Jan 02, 2019 7:10 pm
languages_spoken: english
ODROIDs: h2, h3+, h4 ultra
Location: /dev/earth0
Has thanked: 18 times
Been thanked: 18 times
Contact:

Re: h4 ultra; NICs link refuse to go up in OS; work in uefi

Post by martin00 »

odroid wrote:
Tue Jun 11, 2024 7:25 pm
BTW, did you connect only 3 wires (Rx, Tx, GND) for serial console interface?
I went with the wiki and connected all four of them including VCC. I guess that's the source of the weird electricity, I was introducing 3.3V via pin4.

Indeed I tested it on 6.8.4-2-pve. But other kernels (and OSes) didn't have problem before. It's after that Window10 I can't wake up those NICs in OS at all.

User avatar
odroid
Site Admin
Posts: 42543
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 3735 times
Been thanked: 2103 times
Contact:

Re: h4 ultra; NICs link refuse to go up in OS; work in uefi

Post by odroid »

Have you used our official USB-UART cable? If yes, you need to connect four wires as the wiki page.
https://www.hardkernel.com/shop/usb-uar ... -kit-copy/

How about trying Windows 11 23H2 temporarily with the latest Intel NIC driver to see what happens?
Although it is unlikely, if there is a possibility that the Windows driver for the Intel NIC has changed the contents of the flash memory device dedicated to the I226-V chip, I think it is worth testing with the latest Windows.

martin00
Posts: 96
Joined: Wed Jan 02, 2019 7:10 pm
languages_spoken: english
ODROIDs: h2, h3+, h4 ultra
Location: /dev/earth0
Has thanked: 18 times
Been thanked: 18 times
Contact:

Re: h4 ultra; NICs link refuse to go up in OS; work in uefi

Post by martin00 »

No, I'm using generic USB2uart connector with CP2102 inside; cables need to be connected separetely (pin-to-pin, I don't have the cable). It's time proven little device. I'm not sure why I connected vcc too, probably as I saw it in the first pic at wiki. I was lucky, no harm was done (at least I don't see it).
The issue with the NIC link was there before I connected console.

It does feel like Windows10 did something to it. I could have reflashed firmware as more aggressive approach but I'm more curious to know why this is. Why can't any other OS wake the link up.
I will install win11 to see if I can mitigate the issue.

I'm wondering if somebody else hit the same issue.

martin00
Posts: 96
Joined: Wed Jan 02, 2019 7:10 pm
languages_spoken: english
ODROIDs: h2, h3+, h4 ultra
Location: /dev/earth0
Has thanked: 18 times
Been thanked: 18 times
Contact:

Re: h4 ultra; NICs link refuse to go up in OS; work in uefi

Post by martin00 »

I did install win11 on the box. During its installation and final setup NIC was up but was not connected to network. I assumed that from the fact that win11 didn't let me finish installation before I connect to internet. I had to use usb wifi dongle to connect to network and finish the installation process (didn't know win11 requires that).
Once I was in I saw that driver was not attached to it (no ethernet device). I waited for latest-greatest updates (jun '24), drivers for NICs got installed. The same thing happened - as soon as driver got attached link went down and that was it.
I learned two things: 1) once OS attaches driver link goes down and stays down 2) I really don't like win11

I decided to flash the firmware with the same version - issue persists.
I switched the jumper to boot to backup firmware (1.0) - issue persists.
Annoyingly uefi is still able to PXE boot (which doesn't help that much as once kernel boots up and attaches driver link goes down)

As I was going through the firmware settings I noticed that if I enable network stack (PXE ip4 is enabled) I see this in Advanced->Driver Health:

Code: Select all

┌────────────────────────────────────────────────────┬─────────────────────────┐
│► Intel(R) 2.5G Ethernet Controller 0.10.04         │Provides Health Status   │
│  Healthy                                           │for the                  │
│► Intel(R) 2.5G Ethernet Controller 0.10.06         │Drivers/Controllers      │
│  Healthy                                           │                         │
│► Intel(R) 2.5G Ethernet Controller 0.10.06         │                         │
│  Healthy                                           │                         │
│                                                    │
Doesn't make sense to see 3 adapters, but I don't know what was there when it was working. The 0.10.04 version shows "No Controller Found" error.

Now I'm getting a bit worried because I'm getting out of ideas.
Last edited by martin00 on Wed Jun 12, 2024 4:28 pm, edited 1 time in total.

User avatar
odroid
Site Admin
Posts: 42543
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 3735 times
Been thanked: 2103 times
Contact:

Re: h4 ultra; NICs link refuse to go up in OS; work in uefi

Post by odroid »

There are Intel NIC dedicated flash memory components near the I226-V chips. It is NOT the flash memory for the BIOS.
We will check if there is any way to reprogram the NIC flash memory on the Linux command line.
Give us a couple of days.
These users thanked the author odroid for the post:
martin00 (Wed Jun 12, 2024 4:13 pm)

User avatar
mad_ady
Posts: 12061
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4 (HC1, HC2), C1+, C2, C4 (HC4), N1, N2, N2L, H2, H3+, Go, Go Advance, M1, M1S
Location: Bucharest, Romania
Has thanked: 663 times
Been thanked: 1309 times
Contact:

Re: h4 ultra; NICs link refuse to go up in OS; work in uefi

Post by mad_ady »

Annoyingly uefi is still able to PXE boot (which doesn't help that much as once kernel boots up and attaches driver link goes down)
In the mean time you may want to try with PXE boot disabled, so that uefi doesn't try to bring up your nics.

martin00
Posts: 96
Joined: Wed Jan 02, 2019 7:10 pm
languages_spoken: english
ODROIDs: h2, h3+, h4 ultra
Location: /dev/earth0
Has thanked: 18 times
Been thanked: 18 times
Contact:

Re: h4 ultra; NICs link refuse to go up in OS; work in uefi

Post by martin00 »

mad_ady wrote:
Thu Jun 13, 2024 1:01 am
Annoyingly uefi is still able to PXE boot (which doesn't help that much as once kernel boots up and attaches driver link goes down)
In the mean time you may want to try with PXE boot disabled, so that uefi doesn't try to bring up your nics.
I tried that (I think default config is even network stack disabled).

Now I wonder if that win2go did mess something up or if that was coincidence. I wonder if I can do some magic with ethtool and flash/access these components @odroid mentioned.
Anyway it's a bit sad that I can get to this state.

User avatar
odroid
Site Admin
Posts: 42543
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 3735 times
Been thanked: 2103 times
Contact:

Re: h4 ultra; NICs link refuse to go up in OS; work in uefi

Post by odroid »

@martin00,

We sent you a private message. Please check it.
These users thanked the author odroid for the post:
martin00 (Thu Jun 13, 2024 10:19 pm)

Post Reply

Return to “Issues”

Who is online

Users browsing this forum: No registered users and 3 guests