odroid xu4 won't boot with kernel parameter 'quiet'

Test and fix the Kernel 4.14 features
Post Reply
b0b0tiken
Posts: 6
Joined: Wed Feb 08, 2017 7:05 am
languages_spoken: english
ODROIDs: XU4
Has thanked: 0
Been thanked: 0
Contact:

odroid xu4 won't boot with kernel parameter 'quiet'

Post by b0b0tiken »

Hello everyone,

We noticed about half of our emmcs not booting if the kernel parameter 'quiet' is passed as a boot argument.

I also upgraded to the latest kernel (ubuntu image) and the same issue was present.

To replicate the bug:

in /medit/boot/boot.ini

setenv bootargs ... <--- add quiet, sync and reboot

In the serial output, I get:

Starting kernel ...

[ 0.148403] CPU4: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[ 0.168373] CPU5: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[ 0.176713] CPU6: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[ 0.184697] CPU7: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[ 1.167349] exynos-hdmi 14530000.hdmi: Failed to get supply 'vdd': -517
[ 1.182343] devfreq devfreq0: Couldn't update frequency transition information.


And that's it, board hangs in there; fan is always on, blue led is turned off.


My guess this is some kind of a timing issue masked by kernel delays during the boot logging ? Any clues on how to fix this ?

Cheers,

b0b0

b0b0tiken
Posts: 6
Joined: Wed Feb 08, 2017 7:05 am
languages_spoken: english
ODROIDs: XU4
Has thanked: 0
Been thanked: 0
Contact:

Re: odroid xu4 won't boot with kernel parameter 'quiet'

Post by b0b0tiken »

More updates:

Turns out, with the upgraded kernel, on a cold boot it works. However, the system hangs during a reboot (u-boot / kernel loads and starts but hangs) in all the cases.

Very strange bug.

User avatar
odroid
Site Admin
Posts: 40102
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 2854 times
Been thanked: 1605 times
Contact:

Re: odroid xu4 won't boot with kernel parameter 'quiet'

Post by odroid »

It is really weird. :o

joy
Posts: 1584
Joined: Fri Oct 02, 2015 1:44 pm
languages_spoken: english
ODROIDs: .
Has thanked: 179 times
Been thanked: 211 times
Contact:

Re: odroid xu4 won't boot with kernel parameter 'quiet'

Post by joy »

Hi b0b0tiken,
I've tried this issue and can easily reproduce it with "quiet" option.

I think it could be related to tiny DELAY caused by debug logs during booting and it can affect drivers' init process.
And I found this booting lock-up is caused by abnormal initialization of regulator driver during booting (especially reboot).

1. Kernel code : linux/drivers/regulator/core.c
https://github.com/hardkernel/linux/blo ... ore.c#L912

Code: Select all

		if (target_min != current_uV || target_max != current_uV) {
			rdev_info(rdev, "Bringing %duV into %d-%duV\n",
				  current_uV, target_min, target_max)
			ret = _regulator_do_set_voltage(
				rdev, target_min, target_max);
			if (ret < 0) {
				rdev_err(rdev,
					"failed to apply %d-%duV constraint(%d)\n",
					target_min, target_max, ret);
				return ret;
			}
		}
2. Booting log

Code: Select all

[    1.300233] i2c /dev entries driver
[    1.315758] vdd_ldo9: Bringing 3300000uV into 3000000-3000000uV
[    1.333584] vdd_sd: Bringing 3300000uV into 2800000-2800000uV
[    1.353212] vdd_ldo30: Bringing 1800000uV into 3300000-3300000uV
[    1.381239] vdd_1.8v_ldo: Bringing 1850000uV into 1500000-1500000uV
To verify my assumption, I tried the following temporary ways.
(1) I modified that log before _regulator_do_set_voltage as rdev_err that shows logs via serial console ( make some delay )
or
(2) added some delay at the line, mdelay(10)

Code: Select all

		if (target_min != current_uV || target_max != current_uV) {
			rdev_info(rdev, "Bringing %duV into %d-%duV\n",
				  current_uV, target_min, target_max)
			/* just for test */
			mdelay(10);
and then I can confirm booting works OK with "quiet" option for both of cold and warm booting.

It looks that some ramp_delay for regulator_enable_delay may be needed to run XU4 more stably.
As I checked the logic so far, there is no ramp_delay for regulator driver of exynos5422 XU4.

Code: Select all

Starting kernel ...

[    0.148390] CPU4: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[    0.168368] CPU5: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[    0.176687] CPU6: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[    0.184677] CPU7: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[    1.184757] exynos-hdmi 14530000.hdmi: Failed to get supply 'vdd': -517
[    1.199672] devfreq devfreq0: Couldn't update frequency transition information.
(<<------- here, OK)
[    2.482327] s5p_mfc_load_firmware:73: Firmware is not present in the /lib/firmware directory nor compiled in kernel
[    2.659931] OF: graph: no port node found in /soc/hdmi@14530000
e2fsck 1.44.1 (24-Mar-2018)
rootfs: clean, 172196/475136 files, 1317338/1875456 blocks
[FAILED] Failed to start Load Kernel Modules.
See 'systemctl status systemd-modules-load.service' for details.
[  OK  ] Started Remount Root and Kernel File Systems.
         Starting Load/Save Random Seed...
         Starting Apply Kernel Variables...
         Mounting Kernel Configuration File System...
         Starting Create Static Device Nodes in /dev...
[  OK  ] Started Set the console keyboard layout.
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Started Apply Kernel Variables.
I'm planning to look into it more and make a patch the end of this week.
These users thanked the author joy for the post:
odroid (Tue Sep 03, 2019 2:44 pm)

joy
Posts: 1584
Joined: Fri Oct 02, 2015 1:44 pm
languages_spoken: english
ODROIDs: .
Has thanked: 179 times
Been thanked: 211 times
Contact:

Re: odroid xu4 won't boot with kernel parameter 'quiet'

Post by joy »

Hi b0b0tiken,
Here are some patches.

[ 1 ] Add ramp_delay for DCDC
This is a patch which can activate ramp_delay for buck1~4 and 6
and I've confirmed this fixes abnormal booting with quiet option.

Code: Select all

diff --git a/arch/arm/boot/dts/exynos5422-odroid-core.dtsi b/arch/arm/boot/dts/exynos5422-odroid-core.dtsi
index 6277562..78f8c44 100644
--- a/arch/arm/boot/dts/exynos5422-odroid-core.dtsi
+++ b/arch/arm/boot/dts/exynos5422-odroid-core.dtsi
@@ -307,6 +307,7 @@
 				regulator-max-microvolt = <1300000>;
 				regulator-always-on;
 				regulator-boot-on;
+				regulator-ramp-delay = <12500>;
 			};
 
 			buck2_reg: BUCK2 {
@@ -315,6 +316,7 @@
 				regulator-max-microvolt = <1500000>;
 				regulator-always-on;
 				regulator-boot-on;
+				regulator-ramp-delay = <12500>;
 			};
 
 			buck3_reg: BUCK3 {
@@ -323,6 +325,7 @@
 				regulator-max-microvolt = <1400000>;
 				regulator-always-on;
 				regulator-boot-on;
+				regulator-ramp-delay = <12500>;
 			};
 
 			buck4_reg: BUCK4 {
@@ -331,6 +334,7 @@
 				regulator-max-microvolt = <1400000>;
 				regulator-always-on;
 				regulator-boot-on;
+				regulator-ramp-delay = <12500>;
 			};
 
 			buck5_reg: BUCK5 {
@@ -347,6 +351,7 @@
 				regulator-max-microvolt = <1500000>;
 				regulator-always-on;
 				regulator-boot-on;
+				regulator-ramp-delay = <12500>;
 			};
 
 			buck7_reg: BUCK7 {
[ 2 ] Add enable_time
Actually, based on my first approach using "mdelay", I thought that also regulator "enable_time" can be a solution, not "ramp_delay".
A bit different approach.
https://www.kernel.org/doc/html/v4.11/d ... lator.html

And this approach also can fix abnormal booting issue.

Code: Select all

#define regulator_desc_s2mps11_ldo(num, step) {		\
	.name		= "LDO"#num,			\
	.id		= S2MPS11_LDO##num,		\
	.ops		= &s2mps11_ldo_ops,		\
	.type		= REGULATOR_VOLTAGE,		\
	.owner		= THIS_MODULE,			\
	.ramp_delay	= RAMP_DELAY_12_MVUS,		\
	.min_uV		= MIN_800_MV,			\
	.uV_step	= step,				\
	.n_voltages	= S2MPS11_LDO_N_VOLTAGES,	\
	.vsel_reg	= S2MPS11_REG_L1CTRL + num - 1,	\
	.vsel_mask	= S2MPS11_LDO_VSEL_MASK,	\
	.enable_reg	= S2MPS11_REG_L1CTRL + num - 1,	\
	.enable_mask	= S2MPS11_ENABLE_MASK,		\
	.enable_time	= 10				\
}

#define regulator_desc_s2mps11_buck1_4(num) {			\
	.name		= "BUCK"#num,				\
	.id		= S2MPS11_BUCK##num,			\
	.ops		= &s2mps11_buck_ops,			\
	.type		= REGULATOR_VOLTAGE,			\
	.owner		= THIS_MODULE,				\
	.min_uV		= MIN_600_MV,				\
	.uV_step	= STEP_6_25_MV,				\
	.n_voltages	= S2MPS11_BUCK_N_VOLTAGES,		\
	.ramp_delay	= S2MPS11_RAMP_DELAY,			\
	.vsel_reg	= S2MPS11_REG_B1CTRL2 + (num - 1) * 2,	\
	.vsel_mask	= S2MPS11_BUCK_VSEL_MASK,		\
	.enable_reg	= S2MPS11_REG_B1CTRL1 + (num - 1) * 2,	\
	.enable_mask	= S2MPS11_ENABLE_MASK			\
}

Code: Select all

diff --git a/drivers/regulator/s2mps11.c b/drivers/regulator/s2mps11.c
index 9e3f078..0a3d9e4 100755
--- a/drivers/regulator/s2mps11.c
+++ b/drivers/regulator/s2mps11.c
@@ -275,7 +275,8 @@
 	.vsel_reg	= S2MPS11_REG_L1CTRL + num - 1,	\
 	.vsel_mask	= S2MPS11_LDO_VSEL_MASK,	\
 	.enable_reg	= S2MPS11_REG_L1CTRL + num - 1,	\
-	.enable_mask	= S2MPS11_ENABLE_MASK		\
+	.enable_mask	= S2MPS11_ENABLE_MASK,		\
+	.enable_time	= 10				\
 }
 
 #define regulator_desc_s2mps11_buck1_4(num) {			\
But in this case, adding enable_time for BUCK has no benefit.
Only enable_time for LDO works to remove this issue.

If you have available time, could you test these patches?
I attaches patch files.

Both of two approaches can fix this issue.
To narrow the cause and pick a better one, let's look into more and confirm it. :)
Attachments
0002_xu4_add_enable_time_for_ldo.diff.zip
(664 Bytes) Downloaded 207 times
0001_xu4_add_ramp_delay.diff.zip
(661 Bytes) Downloaded 205 times

User avatar
nicolasvila
Posts: 7
Joined: Thu Aug 04, 2016 11:23 pm
languages_spoken: french, english
ODROIDs: U3, XU4
Location: Toulouse, France
Has thanked: 0
Been thanked: 0
Contact:

Re: odroid xu4 won't boot with kernel parameter 'quiet'

Post by nicolasvila »

Hi, I'm facing the issue and potentially some other parameters are causing issues. Adding "quiet" makes reboot impossible and only cold boot works. Adding "loglevel=3" prevents from rebooting too, but cold boot also work. I don't know if it is related or not.
Is there a way to update the PKGBUILD?

Priidu
Posts: 30
Joined: Sat Apr 23, 2016 10:56 am
languages_spoken: English, Spanish, Estonian
ODROIDs: 5 x ODROID XU4
Location: Tallinn, Estonia
Has thanked: 0
Been thanked: 0
Contact:

Re: odroid xu4 won't boot with kernel parameter 'quiet'

Post by Priidu »

Same problem here.

Has anyone found a workaround that doesn't require manual kernel patching?

phaseshifter
Posts: 3807
Joined: Fri May 08, 2015 9:12 am
languages_spoken: english
ODROIDs: U-2,U3+,,XU-3,,XU3-LITE,,XU-4
C1+,,C-2,,,n2+2G and n2 4G
cloudshell I and shell II
N-1,,N-2,...other odroid acc`s as well..vu7 etc..all sorts of sbc`s these days
Has thanked: 69 times
Been thanked: 52 times
Contact:

Re: odroid xu4 won't boot with kernel parameter 'quiet'

Post by phaseshifter »

if you have another board try the media on that board you may have a hardware fault

quiet should not be doing what fault your describing or flash another sd-card or what ever media you are using..your image may be broken some where...

pss dont have your machine on the web when flashing..nothing on the web is 100 % secure...

several cards leads me to think of a faulty flash image to start with...or maybe your writing system somewhere...electronics can be quite finicky at times..go to another place to down load your flashing image

in the first place in my o/p there should be no need for the patches...there is an underlying issue..i dont know exactly but the patches should not be necessary..although need in this case like i stated in the latter ..on this sentence///

only keaves 2 options hard ware or software..look out for spikes and so on maybe a un filterd power supply try changing for one with better regulation and mains filtering etc..etc..
necessity the mother of all creation and invention..!!!..check out the world ...long live rock and roll.....

Post Reply

Return to “Linux Kernel 4.14 Debugging Party”

Who is online

Users browsing this forum: No registered users and 2 guests