odroid xu4 won't boot with kernel parameter 'quiet'

[b0b0tiken]

Hello everyone,

We noticed about half of our emmcs not booting if the kernel parameter 'quiet' is passed as a boot argument.

I also upgraded to the latest kernel (ubuntu image) and the same issue was present.

To replicate the bug:

in /medit/boot/boot.ini

setenv bootargs ... <--- add quiet, sync and reboot

In the serial output, I get:

Starting kernel ...

[ 0.148403] CPU4: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[ 0.168373] CPU5: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[ 0.176713] CPU6: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[ 0.184697] CPU7: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[ 1.167349] exynos-hdmi 14530000.hdmi: Failed to get supply 'vdd': -517
[ 1.182343] devfreq devfreq0: Couldn't update frequency transition information.


And that's it, board hangs in there; fan is always on, blue led is turned off.


My guess this is some kind of a timing issue masked by kernel delays during the boot logging ? Any clues on how to fix this ?

Cheers,

b0b0

[b0b0tiken]

More updates:

Turns out, with the upgraded kernel, on a cold boot it works. However, the system hangs during a reboot (u-boot / kernel loads and starts but hangs) in all the cases.

Very strange bug.

[odroid]

It is really weird.

[joy]

Hi b0b0tiken,
I've tried this issue and can easily reproduce it with "quiet" option.

I think it could be related to tiny DELAY caused by debug logs during booting and it can affect drivers' init process.
And I found this booting lock-up is caused by abnormal initialization of regulator driver during booting (especially reboot).

1. Kernel code : linux/drivers/regulator/core.c
https://github.com/hardkernel/linux/blo ... ore.c#L912

Code: Select all

		if (target_min != current_uV || target_max != current_uV) {
			rdev_info(rdev, "Bringing %duV into %d-%duV\n",
				  current_uV, target_min, target_max)
			ret = _regulator_do_set_voltage(
				rdev, target_min, target_max);
			if (ret < 0) {
				rdev_err(rdev,
					"failed to apply %d-%duV constraint(%d)\n",
					target_min, target_max, ret);
				return ret;
			}
		}

2. Booting log

Code: Select all

[    1.300233] i2c /dev entries driver
[    1.315758] vdd_ldo9: Bringing 3300000uV into 3000000-3000000uV
[    1.333584] vdd_sd: Bringing 3300000uV into 2800000-2800000uV
[    1.353212] vdd_ldo30: Bringing 1800000uV into 3300000-3300000uV
[    1.381239] vdd_1.8v_ldo: Bringing 1850000uV into 1500000-1500000uV

To verify my assumption, I tried the following temporary ways.
(1) I modified that log before _regulator_do_set_voltage as rdev_err that shows logs via serial console ( make some delay )
or
(2) added some delay at the line, mdelay(10)

Code: Select all

		if (target_min != current_uV || target_max != current_uV) {
			rdev_info(rdev, "Bringing %duV into %d-%duV\n",
				  current_uV, target_min, target_max)
			/* just for test */
			mdelay(10);

and then I can confirm booting works OK with "quiet" option for both of cold and warm booting.

It looks that some ramp_delay for regulator_enable_delay may be needed to run XU4 more stably.
As I checked the logic so far, there is no ramp_delay for regulator driver of exynos5422 XU4.

Code: Select all

Starting kernel ...

[    0.148390] CPU4: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[    0.168368] CPU5: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[    0.176687] CPU6: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[    0.184677] CPU7: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[    1.184757] exynos-hdmi 14530000.hdmi: Failed to get supply 'vdd': -517
[    1.199672] devfreq devfreq0: Couldn't update frequency transition information.
(<<------- here, OK)
[    2.482327] s5p_mfc_load_firmware:73: Firmware is not present in the /lib/firmware directory nor compiled in kernel
[    2.659931] OF: graph: no port node found in /soc/hdmi@14530000
e2fsck 1.44.1 (24-Mar-2018)
rootfs: clean, 172196/475136 files, 1317338/1875456 blocks
[FAILED] Failed to start Load Kernel Modules.
See 'systemctl status systemd-modules-load.service' for details.
[  OK  ] Started Remount Root and Kernel File Systems.
         Starting Load/Save Random Seed...
         Starting Apply Kernel Variables...
         Mounting Kernel Configuration File System...
         Starting Create Static Device Nodes in /dev...
[  OK  ] Started Set the console keyboard layout.
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Started Apply Kernel Variables.

I'm planning to look into it more and make a patch the end of this week.

[joy]

Hi b0b0tiken,
Here are some patches.

[ 1 ] Add ramp_delay for DCDC
This is a patch which can activate ramp_delay for buck1~4 and 6
and I've confirmed this fixes abnormal booting with quiet option.

Code: Select all

diff --git a/arch/arm/boot/dts/exynos5422-odroid-core.dtsi b/arch/arm/boot/dts/exynos5422-odroid-core.dtsi
index 6277562..78f8c44 100644
--- a/arch/arm/boot/dts/exynos5422-odroid-core.dtsi
+++ b/arch/arm/boot/dts/exynos5422-odroid-core.dtsi
@@ -307,6 +307,7 @@
 				regulator-max-microvolt = <1300000>;
 				regulator-always-on;
 				regulator-boot-on;
+				regulator-ramp-delay = <12500>;
 			};
 
 			buck2_reg: BUCK2 {
@@ -315,6 +316,7 @@
 				regulator-max-microvolt = <1500000>;
 				regulator-always-on;
 				regulator-boot-on;
+				regulator-ramp-delay = <12500>;
 			};
 
 			buck3_reg: BUCK3 {
@@ -323,6 +325,7 @@
 				regulator-max-microvolt = <1400000>;
 				regulator-always-on;
 				regulator-boot-on;
+				regulator-ramp-delay = <12500>;
 			};
 
 			buck4_reg: BUCK4 {
@@ -331,6 +334,7 @@
 				regulator-max-microvolt = <1400000>;
 				regulator-always-on;
 				regulator-boot-on;
+				regulator-ramp-delay = <12500>;
 			};
 
 			buck5_reg: BUCK5 {
@@ -347,6 +351,7 @@
 				regulator-max-microvolt = <1500000>;
 				regulator-always-on;
 				regulator-boot-on;
+				regulator-ramp-delay = <12500>;
 			};
 
 			buck7_reg: BUCK7 {

[ 2 ] Add enable_time
Actually, based on my first approach using "mdelay", I thought that also regulator "enable_time" can be a solution, not "ramp_delay".
A bit different approach.
https://www.kernel.org/doc/html/v4.11/d ... lator.html

And this approach also can fix abnormal booting issue.

Code: Select all

#define regulator_desc_s2mps11_ldo(num, step) {		\
	.name		= "LDO"#num,			\
	.id		= S2MPS11_LDO##num,		\
	.ops		= &s2mps11_ldo_ops,		\
	.type		= REGULATOR_VOLTAGE,		\
	.owner		= THIS_MODULE,			\
	.ramp_delay	= RAMP_DELAY_12_MVUS,		\
	.min_uV		= MIN_800_MV,			\
	.uV_step	= step,				\
	.n_voltages	= S2MPS11_LDO_N_VOLTAGES,	\
	.vsel_reg	= S2MPS11_REG_L1CTRL + num - 1,	\
	.vsel_mask	= S2MPS11_LDO_VSEL_MASK,	\
	.enable_reg	= S2MPS11_REG_L1CTRL + num - 1,	\
	.enable_mask	= S2MPS11_ENABLE_MASK,		\
	.enable_time	= 10				\
}

#define regulator_desc_s2mps11_buck1_4(num) {			\
	.name		= "BUCK"#num,				\
	.id		= S2MPS11_BUCK##num,			\
	.ops		= &s2mps11_buck_ops,			\
	.type		= REGULATOR_VOLTAGE,			\
	.owner		= THIS_MODULE,				\
	.min_uV		= MIN_600_MV,				\
	.uV_step	= STEP_6_25_MV,				\
	.n_voltages	= S2MPS11_BUCK_N_VOLTAGES,		\
	.ramp_delay	= S2MPS11_RAMP_DELAY,			\
	.vsel_reg	= S2MPS11_REG_B1CTRL2 + (num - 1) * 2,	\
	.vsel_mask	= S2MPS11_BUCK_VSEL_MASK,		\
	.enable_reg	= S2MPS11_REG_B1CTRL1 + (num - 1) * 2,	\
	.enable_mask	= S2MPS11_ENABLE_MASK			\
}

Code: Select all

diff --git a/drivers/regulator/s2mps11.c b/drivers/regulator/s2mps11.c
index 9e3f078..0a3d9e4 100755
--- a/drivers/regulator/s2mps11.c
+++ b/drivers/regulator/s2mps11.c
@@ -275,7 +275,8 @@
 	.vsel_reg	= S2MPS11_REG_L1CTRL + num - 1,	\
 	.vsel_mask	= S2MPS11_LDO_VSEL_MASK,	\
 	.enable_reg	= S2MPS11_REG_L1CTRL + num - 1,	\
-	.enable_mask	= S2MPS11_ENABLE_MASK		\
+	.enable_mask	= S2MPS11_ENABLE_MASK,		\
+	.enable_time	= 10				\
 }
 
 #define regulator_desc_s2mps11_buck1_4(num) {			\

But in this case, adding enable_time for BUCK has no benefit.
Only enable_time for LDO works to remove this issue.

If you have available time, could you test these patches?
I attaches patch files.

Both of two approaches can fix this issue.
To narrow the cause and pick a better one, let's look into more and confirm it.

[nicolasvila]

Hi, I'm facing the issue and potentially some other parameters are causing issues. Adding "quiet" makes reboot impossible and only cold boot works. Adding "loglevel=3" prevents from rebooting too, but cold boot also work. I don't know if it is related or not.
Is there a way to update the PKGBUILD?

[Priidu]

Same problem here.

Has anyone found a workaround that doesn't require manual kernel patching?

[phaseshifter]

if you have another board try the media on that board you may have a hardware fault

quiet should not be doing what fault your describing or flash another sd-card or what ever media you are using..your image may be broken some where...

pss dont have your machine on the web when flashing..nothing on the web is 100 % secure...

several cards leads me to think of a faulty flash image to start with...or maybe your writing system somewhere...electronics can be quite finicky at times..go to another place to down load your flashing image

in the first place in my o/p there should be no need for the patches...there is an underlying issue..i dont know exactly but the patches should not be necessary..although need in this case like i stated in the latter ..on this sentence///

only keaves 2 options hard ware or software..look out for spikes and so on maybe a un filterd power supply try changing for one with better regulation and mains filtering etc..etc..

[Priidu]

Hi @phaseshifter!

Unfortunately, this is a reproducible issue on the XU4, not just a one-off (e.g. bad PSU). You could probably try it yourself.

In the end, I applied the kernel patch from above and now everything seems to work fine.