Playing with upstream (Exynos4412)

Share here your ideas for new projects

Moderators: odroid, meveric, mdrjr

Re: Playing with upstream (Exynos4412)

Unread postby frol » Mon Nov 21, 2016 5:18 pm

LiquidAcid wrote:The only reboot issue that I know of is gargabe in the UART FIFO, but that only affects u-boot, i.e. interrupts auto-boot and drops you to the prompt. See the corresponding patch in my 'stable' u-boot branch.

I forgot to ask whether this means that you have U3 which runs your kernel fine.

In any case, could you please comment on the following investigation?
... the problem is that the PMIC does not get reset on the U3.
I reported the issue on samsung ml a while ago, and I was not the only one having the issue. @mdrjr also confirmed and actually had a patch to reset pmic on the 4.8 kernel (which he deleted before making it public :D).
Afaik 4.8 should still have the issue. I tried many things, and sometimes it looked like it was fixed, I could reboot sometimes, just to see the issue back after a while, it was pretty random. If the PMIC will not reset, it will keep whatever voltage it has on the next reboot, so it's actually pretty nasty - maybe the mali issue comes from that as well? But test the power source too.
Try changing the DT to have only one frequency available, e.g. lock it at 1Ghz, since that's the frequency uboot uses too. Also try disabling frequency scaling in kernel config, it will get your board to behave better, although you will still have issues.

if you know what you're doing, try to look at the reset code from the HK kernel (in the platform files), and copy the sequence (2 lines of code) to the new kernel (in the new generic reset driver).

For me, locking at 1Ghz is simply not an option, and since I am not a kernel hacker, I don't think I will succeed in providing something valuable without a very detailed guidance (I don't even know how to get logs when kernel panics, so I only observe the "alive" LED changes).
frol
 
Posts: 11
Joined: Fri Sep 05, 2014 2:07 pm
languages_spoken: english, russian, ukrainian
ODROIDs: U3

Re: Playing with upstream (Exynos4412)

Unread postby Panzerknacker » Mon Nov 21, 2016 6:03 pm

memeka wrote:EDIT: if you know what you're doing, try to look at the reset code from the HK kernel (in the platform files), and copy the sequence (2 lines of code) to the new kernel (in the new generic reset driver).


Hi memeka,

could you share these two lines of code?
I will test them and prepare a patch for ML.

Thanks.
User avatar
Panzerknacker
 
Posts: 237
Joined: Sat Feb 22, 2014 10:08 pm
languages_spoken: German, English
ODROIDs: U3, X, XU, XU3, XU4, W

Re: Playing with upstream (Exynos4412)

Unread postby LiquidAcid » Tue Nov 22, 2016 4:25 am

frol wrote:I forgot to ask whether this means that you have U3 which runs your kernel fine.

The only ARM board I have is the X2.

frol wrote:In any case, could you please comment on the following investigation?
<snip>

There is a number of outdated or simply wrong information here. First of all, the PMIC issue has been fixed upstream since 2015 (see commit 1605b60ad064c7019db8ade07f0b7bdc8c197b93). cpufreq-dt switches to the state marked with 'opp-suspend' when the system goes into suspend or does shutdown. That restores the regulator settings for the first-stage bootloader to work.

Another misconception is that this has something to do with u-boot. It's not u-boot that fails here on reboot, but the BL0/BL1/BL2.

More misinformation in that post:
- locking cpufreq to 1GHz does nothing in that situation, since cpufreq-dt switches to the suspend OPP anyway
- there is no PMIC reset happening in the platform code of Hardkernel's odroid-3.8.y
- Mali platform glue switches G3D regulator on before enabling clocks, so in particular it doesn't rely on the regulator state set by the bootloader
- Nicolas Dufresne has reported some reboot issue some time ago (http://www.spinics.net/lists/linux-sams ... 54371.html)

However (!) the issue reported by Nicolas is related to emergency reboot/shutdown triggered by SysRq. In that case the kernel omits calling the reboot notifier chain, which in particular also triggers eMMC hw reset or the aforementioned switch to the suspend OPP. This is by design (emergency == do things quick) and is not likely to change.
Like the combination watchdog+eMMC, this feature is going to remain broken due to the flaws in ODROID board design.

However this does _not_ affect the default (non-emergency) reboot/shutdown.


The only thing that does come to my mind with respect to regulator voltage, are MIF and INT, which are adjusted through exynos-bus. AFAIK there is no mechanism in place to return these values to their defaults on suspend. But there is no difference between upstream and odroid-4.8.y here. So, if you don't have any of these issues on the upstream kernel (like you say above), I doubt it's related to MIF/INT.
LiquidAcid
 
Posts: 1046
Joined: Fri Oct 11, 2013 11:07 pm
languages_spoken: english
ODROIDs: X2

Re: Playing with upstream (Exynos4412)

Unread postby Panzerknacker » Tue Nov 22, 2016 4:37 am

I have these issues with upstream and current -next, too.

@frol:
Could you try
# echo performance > /sys/class/devfreq/bus_leftbus/governor
# echo performance > /sys/class/devfreq/bus_dmc/governor
before reboot
and report back?
Last edited by Panzerknacker on Tue Nov 22, 2016 4:42 am, edited 1 time in total.
User avatar
Panzerknacker
 
Posts: 237
Joined: Sat Feb 22, 2014 10:08 pm
languages_spoken: German, English
ODROIDs: U3, X, XU, XU3, XU4, W

Re: Playing with upstream (Exynos4412)

Unread postby LiquidAcid » Tue Nov 22, 2016 4:41 am

Well, apparantly the issues frol does describe don't appear upstream. Hence I have to assume that your reboot issues are a different thing.
LiquidAcid
 
Posts: 1046
Joined: Fri Oct 11, 2013 11:07 pm
languages_spoken: english
ODROIDs: X2

Re: Playing with upstream (Exynos4412)

Unread postby memeka » Tue Nov 22, 2016 5:25 am

Panzerknacker wrote:
memeka wrote:EDIT: if you know what you're doing, try to look at the reset code from the HK kernel (in the platform files), and copy the sequence (2 lines of code) to the new kernel (in the new generic reset driver).


Hi memeka,

could you share these two lines of code?
I will test them and prepare a patch for ML.

Thanks.


Ask @mdrjr about that, like I said, I did not fix the issue but @mdrjr said he did.

LiquidAcid wrote:There is a number of outdated or simply wrong information here. First of all, the PMIC issue has been fixed upstream since 2015 (see commit 1605b60ad064c7019db8ade07f0b7bdc8c197b93). cpufreq-dt switches to the state marked with 'opp-suspend' when the system goes into suspend or does shutdown. That restores the regulator settings for the first-stage bootloader to work.

Another misconception is that this has something to do with u-boot. It's not u-boot that fails here on reboot, but the BL0/BL1/BL2.

More misinformation in that post:
- locking cpufreq to 1GHz does nothing in that situation, since cpufreq-dt switches to the suspend OPP anyway
- there is no PMIC reset happening in the platform code of Hardkernel's odroid-3.8.y
- Mali platform glue switches G3D regulator on before enabling clocks, so in particular it doesn't rely on the regulator state set by the bootloader
- Nicolas Dufresne has reported some reboot issue some time ago (http://www.spinics.net/lists/linux-sams ... 54371.html)


Read that thread from Nicolas again:

Nicolas:
Well for me both method hangs. On an extra note, reset inside Uboot
works, but reboot in Linux hang. I tried adding a trace
in syscon_restart_handle(), but the trace is never displayed.


So it's the reboot that he's complaining about, not emergency restart.

The response from Krzysztof (from samsung):
The problem is: reset is a software reset of AP which means that clocks
go to default values. If cpufreq is active and CPU regulator voltage is
low, then it will fail to boot because clock frequency is high but
voltage is low, too low.

Possible solutions would be:
1. Reset PMIC values. This does not happen on software reset. I tried
also hardware reset through watchdog - also nothing.

2. Set regulator to safe value (same as for suspend, opp-suspend). This
cannot be used in emergency path because it is a sleeping, long
operation. I even wrote simpler cpufreq restart handler but still this
is not accepted sleeping operation. Also on Odroid U3 the CPU regulator
voltage cannot be controlled through GPIO/DVS because... they are wired
to VDD.

3. Extend emergency reboot path with execution of some new
syscore_emergency_shutdown() op. That is quite intrusive and requires
changing in many archs... Not really applicable.

4. Change the minimum voltage of buck2 to 1.0V on Odroid. Set is as
constrain, always. Not nice, but should help... Is it worth the effort?


his patch was to increase the regulator voltage (#4).
i was proposing to lower the clock => by lowering the clock, you won't have the issue of high freq, low voltage => no reboot.
but my solution did not work all the time (although it was better).

Now, about the HK platform code.
Newer kernel uses the syscon restart handler. Let's compare the 2:

POWEROFF (which works):
4.8 kernel syscon dt: https://github.com/tobiasjakobi/linux-o ... t.dtsi#L13
offset = <0x330C>; /* PS_HOLD_CONTROL */
mask = <0x5200>; /* reset value */
3.8 HK platform file: https://github.com/hardkernel/linux/blo ... 412.c#L659
writel(0x5200, S5P_PS_HOLD_CONTROL);

so for poweroff they do the same thing.

REBOOT:
4.8 kernel: https://github.com/tobiasjakobi/linux-o ... t.dtsi#L20
offset = <0x0400>; /* SWRESET */
mask = <0x1>;
3.8 HK platform file: https://github.com/hardkernel/linux/blo ... 412.c#L669

Not the same anymore.

In any case, I hope @mdrjr can share his code :)
User avatar
memeka
 
Posts: 3262
Joined: Mon May 20, 2013 10:22 am
languages_spoken: english
ODROIDs: XU rev2 + eMMC + UART
U3 + eMMC + IO Shield + UART

Re: Playing with upstream (Exynos4412)

Unread postby frol » Tue Nov 22, 2016 5:31 am

@LiquidAcid Thank you for the detailed response!

I have found out that the kernel which I thought was the pure mainline, in fact, wasn't pure. It is a mainline with 13 patches: https://github.com/archlinuxarm/PKGBUIL ... inux-armv7

And one of the patches is: https://github.com/archlinuxarm/PKGBUIL ... ator.patch

Code: Select all
From cf306f4ec83a1f645d01e01d1d18c555e6b766e5 Mon Sep 17 00:00:00 2001
From: Kevin Mihelich <kevin@archlinuxarm.org>
Date: Thu, 11 Aug 2016 00:42:37 -0600
Subject: [PATCH 10/13] exynos4412-odroid: set higher minimum buck2 regulator
 voltage

Set a higher minimum voltage to help reboot issue.
http://www.spinics.net/lists/linux-samsung-soc/msg54434.html

Signed-off-by: Kevin Mihelich <kevin@archlinuxarm.org>
---
 arch/arm/boot/dts/exynos4412-odroid-common.dtsi | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm/boot/dts/exynos4412-odroid-common.dtsi b/arch/arm/boot/dts/exynos4412-odroid-common.dtsi
index 58ad48e7..f7c5571 100644
--- a/arch/arm/boot/dts/exynos4412-odroid-common.dtsi
+++ b/arch/arm/boot/dts/exynos4412-odroid-common.dtsi
@@ -423,7 +423,7 @@
 
          buck2_reg: BUCK2 {
             regulator-name = "vdd_arm";
-            regulator-min-microvolt = <900000>;
+            regulator-min-microvolt = <1100000>;
             regulator-max-microvolt = <1350000>;
             regulator-always-on;
             regulator-boot-on;
--
2.10.2

There is a link to the discussion on a mailing list. I will give this patch a try and will report back.
frol
 
Posts: 11
Joined: Fri Sep 05, 2014 2:07 pm
languages_spoken: english, russian, ukrainian
ODROIDs: U3

Re: Playing with upstream (Exynos4412)

Unread postby LiquidAcid » Tue Nov 22, 2016 6:19 am

That's Krzysztof workaround for emergency reboot. I already covered that.
LiquidAcid
 
Posts: 1046
Joined: Fri Oct 11, 2013 11:07 pm
languages_spoken: english
ODROIDs: X2

Re: Playing with upstream (Exynos4412)

Unread postby frol » Tue Nov 22, 2016 7:30 am

The patch hasn't made any difference.

I am building a mainline kernel currently (using the config from the ALARM repo as a starting point), and will iterate from there. The Arch Linux linux-armv7 kernel still works fine, so my board is sane.

I have also tried to compile CEC kernel module and it compiled just fine (/dev/cec0 device is available after `modprobe s5p-cec`), but I cannot test it (once I start Kodi on this kernel, the board goes dead).
frol
 
Posts: 11
Joined: Fri Sep 05, 2014 2:07 pm
languages_spoken: english, russian, ukrainian
ODROIDs: U3

Re: Playing with upstream (Exynos4412)

Unread postby memeka » Tue Nov 22, 2016 8:54 am

Please understand, neither mainline nor any existing tree will have reboot working on the U3.
User avatar
memeka
 
Posts: 3262
Joined: Mon May 20, 2013 10:22 am
languages_spoken: english
ODROIDs: XU rev2 + eMMC + UART
U3 + eMMC + IO Shield + UART

Re: Playing with upstream (Exynos4412)

Unread postby frol » Tue Nov 22, 2016 3:43 pm

@memeka The kernel from ALARM (4.8.8 with 13 patches and a quite different generic config) works absolutely fine on my U3! (the only thing I miss there is Mali) https://github.com/archlinuxarm/PKGBUIL ... inux-armv7

I have booted and rebooted into that kernel over a dozen times with no issues at all.
frol
 
Posts: 11
Joined: Fri Sep 05, 2014 2:07 pm
languages_spoken: english, russian, ukrainian
ODROIDs: U3

Re: Playing with upstream (Exynos4412)

Unread postby memeka » Tue Nov 22, 2016 4:37 pm

frol wrote:@memeka The kernel from ALARM (4.8.8 with 13 patches and a quite different generic config) works absolutely fine on my U3! (the only thing I miss there is Mali) https://github.com/archlinuxarm/PKGBUIL ... inux-armv7

I have booted and rebooted into that kernel over a dozen times with no issues at all.


can you reboot from the UART console?
User avatar
memeka
 
Posts: 3262
Joined: Mon May 20, 2013 10:22 am
languages_spoken: english
ODROIDs: XU rev2 + eMMC + UART
U3 + eMMC + IO Shield + UART

Re: Playing with upstream (Exynos4412)

Unread postby frol » Tue Nov 22, 2016 5:00 pm

@memeka I think, I cannot. I don't have tools for the low-level operations. I can provide any information I can get via SSH. In the attachment you can find two dmesgs (one from ALARM kernel and another from @LiquidAcid's successfully running kernel (I don't know how to extract a failing dmesg, though)), and their diff.
Attachments
dmesg.diff.txt
(28.39 KiB) Downloaded 25 times
dmesg-LiquidAcid.txt
(23.2 KiB) Downloaded 24 times
dmesg-alarm.txt
(31.12 KiB) Downloaded 18 times
frol
 
Posts: 11
Joined: Fri Sep 05, 2014 2:07 pm
languages_spoken: english, russian, ukrainian
ODROIDs: U3

Re: Playing with upstream (Exynos4412)

Unread postby frol » Wed Nov 23, 2016 9:16 pm

Here are the experiments I have conducted so far:

  1. Mainline 4.8.9 kernel with ALARM patches and their default config (it took over 5 hours to compile on my U3):
    • + boots and reboots 100% fine
    • + operates normally (my board has never died running this kernel)
    • - the only missing part is missing Mali GPU support (this is "by design")
  2. @LuquidAcid's 4.8.9 kernel with his vanila-4.8 config:
    • - boots only after an abrupt power disconnection during a previous boot before it dies itself
    • - normal reboot doesn't work, so the manual actions should be performed
    • - dies after 10-20 minutes
    • - `maligpu` driver exists, but fails, though I have seen `/dev/mali` once
  3. Mainline 4.8.9 kernel with ALARM patches and @LuquidAcid's vanila-4.8 config
    • + cold boot is fine
    • + operates normally (my board has never died running this kernel; another kernel compilation succeeded just fine)
    • - normal reboot just powers off my board (thus, cold boot is required, which works)
    • - no Mali GPU "by design"

NOTE: @ztombol's kernel config, which is based on @LiquidAcid's vanila-4.8 produced the same results as the vanila-4.8-based kernels.

I am currently compiling the @LiquidAcid's kernel with the ALARM config. If it will not make a difference, I will start bisecting @LiquidAcid's patches, otherwise, I will bisect the configs diff.
frol
 
Posts: 11
Joined: Fri Sep 05, 2014 2:07 pm
languages_spoken: english, russian, ukrainian
ODROIDs: U3

Re: Playing with upstream (Exynos4412)

Unread postby frol » Fri Nov 25, 2016 6:03 am

    4. @LiquidAcid's 4.8.10 kernel with an ALARM config (the missing configs, mostly around Mali GPU support, were set as they are set in the vanila-4.8.conf)
    • operates normally unless I do `rmmod/modprobe maligpu`
    • - cold boot fails sometimes (though I couldn't get a successful cold boot on the @LiquidAcid's kernel + his vanila-4.8 at all, so it is already an improvement)
    • - reboot doesn't work
    • - `maligpu` driver gets loaded, but `mali-utgard: probe of 13000000.gpu failed with error -14` error seems to prevent its operation

I have no clue what is wrong with the Mali driver, but it seems to be the key factor for kernel instability. I still cannot get a sense out of the messages Mali driver writes to dmesg:

Code: Select all
[    6.111539] mali-utgard 13000000.gpu: Mali platform: G3D clock rate = 24 MHz
[    6.113097] mali-utgard 13000000.gpu: Mali platform: initial G3D core clock rate = 160 MHz
[    6.138396] Mali: ERR: drivers/gpu/arm/mali/common/mali_gp.c
[    6.138515]            mali_gp_reset_wait() 178
                          Mali GP: Failed to reset core Mali_GP, rawstat: 0x00000000
[    6.150605] Mali: ERR: drivers/gpu/arm/mali/common/mali_kernel_core.c
[    6.157109]            mali_create_group() 315
[    6.167268] Mali: ERR: drivers/gpu/arm/mali/linux/mali_kernel_linux.c
[    6.172740]            mali_probe() 490
                          mali_probe(): Failed to initialize Mali device driver.
[    6.182372] mali-utgard: probe of 13000000.gpu failed with error -14
[    6.188896] Mali: Mali device driver loaded


I am going to compile the kernel with MALI_DEBUG options set to hopefully get a more detailed output.
frol
 
Posts: 11
Joined: Fri Sep 05, 2014 2:07 pm
languages_spoken: english, russian, ukrainian
ODROIDs: U3

Re: Playing with upstream (Exynos4412)

Unread postby LiquidAcid » Thu Jan 12, 2017 2:59 am

First status update in 2017:
- odroid-4.7.y removed
- odroid-4.8.y is EOL, last rebase onto 4.8.17, branch locked
- odroid-4.9.y rebased on stable/linux-4.9.2
- work on odroid-4.10.y started, only compile tested atm, not pushed yet

Minimal work done on odroid-4.9.y, mostly cosmetic. First/preliminary patchset applied which should mitigate the devfreq issues pointed out by Panzerknacker. I hope to find some time to work on the patchset to get it upstream for the 4.11.y merge window.

I also did some analysis on the garbage-on-UART issue on my X2 (when nothing is connected). Turns out that the issue, which I had only identified in u-boot, also persisted in the kernel. E.g. when enabling all SysRq combinations by default on boot, the garbage data would very likely trigger a SysRq+k during init, which then left the system in an unuseable state. Manually setting some pullup on the corresponding pins seems to help. Needs further testing.
LiquidAcid
 
Posts: 1046
Joined: Fri Oct 11, 2013 11:07 pm
languages_spoken: english
ODROIDs: X2

Re: Playing with upstream (Exynos4412)

Unread postby ztombol » Thu Jan 12, 2017 7:50 pm

I updated my package for Arch Linux ARM to 4.9.2 if anyone wants to test.
ztombol
 
Posts: 7
Joined: Wed Nov 09, 2016 11:16 pm
languages_spoken: English, Japanese, Hungarian
ODROIDs: U2

Previous

Return to The Ideas

Who is online

Users browsing this forum: No registered users and 1 guest