odroid xu4 won't boot with kernel parameter 'quiet'

[b0b0tiken]

Hello everyone,

We noticed about half of our emmcs not booting if the kernel parameter 'quiet' is passed as a boot argument.

I also upgraded to the latest kernel (ubuntu image) and the same issue was present.

To replicate the bug:

in /medit/boot/boot.ini

setenv bootargs ... <--- add quiet, sync and reboot

In the serial output, I get:

Starting kernel ...

[ 0.148403] CPU4: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[ 0.168373] CPU5: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[ 0.176713] CPU6: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[ 0.184697] CPU7: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[ 1.167349] exynos-hdmi 14530000.hdmi: Failed to get supply 'vdd': -517
[ 1.182343] devfreq devfreq0: Couldn't update frequency transition information.


And that's it, board hangs in there; fan is always on, blue led is turned off.


My guess this is some kind of a timing issue masked by kernel delays during the boot logging ? Any clues on how to fix this ?

Cheers,

b0b0

[b0b0tiken]

More updates:

Turns out, with the upgraded kernel, on a cold boot it works. However, the system hangs during a reboot (u-boot / kernel loads and starts but hangs) in all the cases.

Very strange bug.

[odroid]

It is really weird.

[joy]

Hi b0b0tiken,
I've tried this issue and can easily reproduce it with "quiet" option.

I think it could be related to tiny DELAY caused by debug logs during booting and it can affect drivers' init process.
And I found this booting lock-up is caused by abnormal initialization of regulator driver during booting (especially reboot).

1. Kernel code : linux/drivers/regulator/core.c
https://github.com/hardkernel/linux/blo ... ore.c#L912

Code: Select all

		if (target_min != current_uV || target_max != current_uV) {
			rdev_info(rdev, "Bringing %duV into %d-%duV\n",
				  current_uV, target_min, target_max)
			ret = _regulator_do_set_voltage(
				rdev, target_min, target_max);
			if (ret < 0) {
				rdev_err(rdev,
					"failed to apply %d-%duV constraint(%d)\n",
					target_min, target_max, ret);
				return ret;
			}
		}

2. Booting log

Code: Select all

[    1.300233] i2c /dev entries driver
[    1.315758] vdd_ldo9: Bringing 3300000uV into 3000000-3000000uV
[    1.333584] vdd_sd: Bringing 3300000uV into 2800000-2800000uV
[    1.353212] vdd_ldo30: Bringing 1800000uV into 3300000-3300000uV
[    1.381239] vdd_1.8v_ldo: Bringing 1850000uV into 1500000-1500000uV

To verify my assumption, I tried the following temporary ways.
(1) I modified that log before _regulator_do_set_voltage as rdev_err that shows logs via serial console ( make some delay )
or
(2) added some delay at the line, mdelay(10)

Code: Select all

		if (target_min != current_uV || target_max != current_uV) {
			rdev_info(rdev, "Bringing %duV into %d-%duV\n",
				  current_uV, target_min, target_max)
			/* just for test */
			mdelay(10);

and then I can confirm booting works OK with "quiet" option for both of cold and warm booting.

It looks that some ramp_delay for regulator_enable_delay may be needed to run XU4 more stably.
As I checked the logic so far, there is no ramp_delay for regulator driver of exynos5422 XU4.

Code: Select all

Starting kernel ...

[    0.148390] CPU4: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[    0.168368] CPU5: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[    0.176687] CPU6: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[    0.184677] CPU7: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[    1.184757] exynos-hdmi 14530000.hdmi: Failed to get supply 'vdd': -517
[    1.199672] devfreq devfreq0: Couldn't update frequency transition information.
(<<------- here, OK)
[    2.482327] s5p_mfc_load_firmware:73: Firmware is not present in the /lib/firmware directory nor compiled in kernel
[    2.659931] OF: graph: no port node found in /soc/hdmi@14530000
e2fsck 1.44.1 (24-Mar-2018)
rootfs: clean, 172196/475136 files, 1317338/1875456 blocks
[FAILED] Failed to start Load Kernel Modules.
See 'systemctl status systemd-modules-load.service' for details.
[  OK  ] Started Remount Root and Kernel File Systems.
         Starting Load/Save Random Seed...
         Starting Apply Kernel Variables...
         Mounting Kernel Configuration File System...
         Starting Create Static Device Nodes in /dev...
[  OK  ] Started Set the console keyboard layout.
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Started Apply Kernel Variables.

I'm planning to look into it more and make a patch the end of this week.

[joy]

Hi b0b0tiken,
Here are some patches.

[ 1 ] Add ramp_delay for DCDC
This is a patch which can activate ramp_delay for buck1~4 and 6
and I've confirmed this fixes abnormal booting with quiet option.

Code: Select all

diff --git a/arch/arm/boot/dts/exynos5422-odroid-core.dtsi b/arch/arm/boot/dts/exynos5422-odroid-core.dtsi
index 6277562..78f8c44 100644
--- a/arch/arm/boot/dts/exynos5422-odroid-core.dtsi
+++ b/arch/arm/boot/dts/exynos5422-odroid-core.dtsi
@@ -307,6 +307,7 @@
 				regulator-max-microvolt = <1300000>;
 				regulator-always-on;
 				regulator-boot-on;
+				regulator-ramp-delay = <12500>;
 			};
 
 			buck2_reg: BUCK2 {
@@ -315,6 +316,7 @@
 				regulator-max-microvolt = <1500000>;
 				regulator-always-on;
 				regulator-boot-on;
+				regulator-ramp-delay = <12500>;
 			};
 
 			buck3_reg: BUCK3 {
@@ -323,6 +325,7 @@
 				regulator-max-microvolt = <1400000>;
 				regulator-always-on;
 				regulator-boot-on;
+				regulator-ramp-delay = <12500>;
 			};
 
 			buck4_reg: BUCK4 {
@@ -331,6 +334,7 @@
 				regulator-max-microvolt = <1400000>;
 				regulator-always-on;
 				regulator-boot-on;
+				regulator-ramp-delay = <12500>;
 			};
 
 			buck5_reg: BUCK5 {
@@ -347,6 +351,7 @@
 				regulator-max-microvolt = <1500000>;
 				regulator-always-on;
 				regulator-boot-on;
+				regulator-ramp-delay = <12500>;
 			};
 
 			buck7_reg: BUCK7 {

[ 2 ] Add enable_time
Actually, based on my first approach using "mdelay", I thought that also regulator "enable_time" can be a solution, not "ramp_delay".
A bit different approach.
https://www.kernel.org/doc/html/v4.11/d ... lator.html

And this approach also can fix abnormal booting issue.

Code: Select all

#define regulator_desc_s2mps11_ldo(num, step) {		\
	.name		= "LDO"#num,			\
	.id		= S2MPS11_LDO##num,		\
	.ops		= &s2mps11_ldo_ops,		\
	.type		= REGULATOR_VOLTAGE,		\
	.owner		= THIS_MODULE,			\
	.ramp_delay	= RAMP_DELAY_12_MVUS,		\
	.min_uV		= MIN_800_MV,			\
	.uV_step	= step,				\
	.n_voltages	= S2MPS11_LDO_N_VOLTAGES,	\
	.vsel_reg	= S2MPS11_REG_L1CTRL + num - 1,	\
	.vsel_mask	= S2MPS11_LDO_VSEL_MASK,	\
	.enable_reg	= S2MPS11_REG_L1CTRL + num - 1,	\
	.enable_mask	= S2MPS11_ENABLE_MASK,		\
	.enable_time	= 10				\
}

#define regulator_desc_s2mps11_buck1_4(num) {			\
	.name		= "BUCK"#num,				\
	.id		= S2MPS11_BUCK##num,			\
	.ops		= &s2mps11_buck_ops,			\
	.type		= REGULATOR_VOLTAGE,			\
	.owner		= THIS_MODULE,				\
	.min_uV		= MIN_600_MV,				\
	.uV_step	= STEP_6_25_MV,				\
	.n_voltages	= S2MPS11_BUCK_N_VOLTAGES,		\
	.ramp_delay	= S2MPS11_RAMP_DELAY,			\
	.vsel_reg	= S2MPS11_REG_B1CTRL2 + (num - 1) * 2,	\
	.vsel_mask	= S2MPS11_BUCK_VSEL_MASK,		\
	.enable_reg	= S2MPS11_REG_B1CTRL1 + (num - 1) * 2,	\
	.enable_mask	= S2MPS11_ENABLE_MASK			\
}

Code: Select all

diff --git a/drivers/regulator/s2mps11.c b/drivers/regulator/s2mps11.c
index 9e3f078..0a3d9e4 100755
--- a/drivers/regulator/s2mps11.c
+++ b/drivers/regulator/s2mps11.c
@@ -275,7 +275,8 @@
 	.vsel_reg	= S2MPS11_REG_L1CTRL + num - 1,	\
 	.vsel_mask	= S2MPS11_LDO_VSEL_MASK,	\
 	.enable_reg	= S2MPS11_REG_L1CTRL + num - 1,	\
-	.enable_mask	= S2MPS11_ENABLE_MASK		\
+	.enable_mask	= S2MPS11_ENABLE_MASK,		\
+	.enable_time	= 10				\
 }
 
 #define regulator_desc_s2mps11_buck1_4(num) {			\

But in this case, adding enable_time for BUCK has no benefit.
Only enable_time for LDO works to remove this issue.

If you have available time, could you test these patches?
I attaches patch files.

Both of two approaches can fix this issue.
To narrow the cause and pick a better one, let's look into more and confirm it.

[nicolasvila]

Hi, I'm facing the issue and potentially some other parameters are causing issues. Adding "quiet" makes reboot impossible and only cold boot works. Adding "loglevel=3" prevents from rebooting too, but cold boot also work. I don't know if it is related or not.
Is there a way to update the PKGBUILD?