KVM

Post Reply
crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

KVM

Unread post by crashoverride » Tue Mar 01, 2016 4:53 am

Continuation of KVM related discussion from http://forum.odroid.com/posting.php?mod ... 38&t=18735.

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Tue Mar 01, 2016 5:07 am

Summary thus far:

1) We need to fix our device tree GIC entry.
https://github.com/hardkernel/linux/blo ... #L226-L234
As noted here:
http://lxr.free-electrons.com/source/Do ... v=3.14#L65
we need two additional "reg" entries.
Additional regions specifying the base physical address and
size of the VGIC registers. The first additional region is the GIC
virtual interface control register base and size. The 2nd additional
region is the GIC virtual cpu interface register base and size.
2) We need to fix our virtual timer code.
https://github.com/hardkernel/linux/blo ... #L241-L245
The current timer assumes an armv7 or armv8 timer. Instead, we have a meson timer.
https://github.com/hardkernel/linux/blo ... i#L47-L106
We either need to patch the current code to support this timer or write a new driver to support it.

I do not have enough information to do either #1[*] or #2 above. So it will be up to HardKernel/Amlogic to take this any further.

[*] I did guess at what #1 may be:

Code: Select all

   gic: interrupt-controller@2c001000 {
      compatible = "arm,cortex-a15-gic", "arm,cortex-a9-gic";
      #interrupt-cells = <3>;
      #address-cells = <0>;
      interrupt-controller;
      reg = <0x0 0xc4301000 0 0x1000>,
            <0x0 0xc4302000 0 0x0100>,
           <0x0 0xc4304000 0 0x2000>,
           <0x0 0xc4306000 0 0x2000>;
      interrupts = <GIC_PPI 9 0xf04>;
   };

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Tue Mar 01, 2016 12:11 pm

Made some more changes to device tree:

Code: Select all

[    0.453097] kvm [1]: Using HYP init bounce page @5c4a4000
[    0.453230] kvm [1]: interrupt-controller@c4304000 IRQ25
[    0.453415] kvm [1]: timer IRQ27
[    0.453429] kvm [1]: Hyp mode initialized successfully
However, libvirt still faults:

Code: Select all

[    8.611084] libvirtd[549]: undefined instruction: pc=0000007f9f991f38
[    8.611090] Code: d2800002 1e602008 54000701 d5033fdf (d53be040)
[    9.068067] fb: osd[0] enable: 1 (Xorg)
[    9.070808] libvirtd[670]: undefined instruction: pc=0000007f8dad1f38
[    9.070814] Code: d2800002 1e602008 54000701 d5033fdf (d53be040)
[    9.287850] libvirtd[690]: undefined instruction: pc=0000007f96c1df38
[    9.287863] Code: d2800002 1e602008 54000701 d5033fdf (d53be040)
[    9.492243] libvirtd[711]: undefined instruction: pc=0000007f777adf38
[    9.492256] Code: d2800002 1e602008 54000701 d5033fdf (d53be040)
[    9.945703] libvirtd[747]: undefined instruction: pc=0000007f7d42bf38
[    9.945717] Code: d2800002 1e602008 54000701 d5033fdf (d53be040)
For those interested, my completely made up change to the timer device tree entry is:

Code: Select all

	timer{
		compatible = "arm, meson-timer","arm,armv8-timer";
		#address-cells = <2>;
		#size-cells = <2>;
		ranges;
		reg=   <0x0 0xc1109990 0x0 0x4 0x0 0xc1109988 0x0 0x4>;
		interrupts = <GIC_PPI 13 0xff01>,
					 <GIC_PPI 14 0xff01>,
			         <GIC_PPI 11 0xff01>,
			         <GIC_PPI 10 0xff01>;

		timer_f:timer-f{
			timer_name="MESON TIMER-F";
			clockevent-rating=<300>;
			clockevent-shift=<20>;
			clockevent-features=<0x03>;
			interrupts = <0 60 1>;
			bit_enable=<16>;
			bit_mode=<12>;
			bit_resolution=<0>;
			reg=   <0x0 0xc1109994 0x0 0x4>;
		};
		timer_g:timer-g{
			timer_name="MESON TIMER-G";
			clockevent-rating=<300>;
			clockevent-shift=<20>;
			clockevent-features=<0x03>;
			interrupts = <0 61 1>;
			bit_enable=<17>;
			bit_mode=<13>;
			bit_resolution=<2>;
			reg=<0x0 0xc1109998 0x0 0x4>;
		};
		timer_h:timer-h{
			timer_name="MESON TIMER-H";
			clockevent-rating=<300>;
			clockevent-shift=<20>;
			clockevent-features=<0x03>;
			interrupts = <0 62 1>;
			bit_enable=<18>;
			bit_mode=<14>;
			bit_resolution=<4>;
			reg=<0x0 0xc110999c 0x0 0x4>;
		};
		timer_i:timer-i{
			timer_name="MESON TIMER-I";
			clockevent-rating=<300>;
			clockevent-shift=<20>;
			clockevent-features=<0x03>;
			interrupts = <0 63 1>;
			bit_enable=<19>;
			bit_mode=<15>;
			bit_resolution=<6>;
			reg=<0x0 0xc11099a0 0x0 0x4>;
		};
	};
[edit]
Related:

Code: Select all

odroid@odroid64:~$ qemu-system-arm
Illegal instruction (core dumped)

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Tue Mar 01, 2016 3:49 pm

I tested on @XeoSal 's 64bit Debian Jessie image and it does not produce an illegal instruction fault. However, qemu crashes when trying to initialize the timer (as expected due to my fake timer device tree entry.)

Code: Select all

[ 6258.309021] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[ 6258.311452] pgd = ffffffc04bc71000
[ 6258.314814] [00000000] *pgd=0000000052bc0003, *pmd=0000000000000000
[ 6258.321028] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[ 6258.326546] Modules linked in:
[ 6258.329566] CPU: 1 PID: 4096 Comm: qemu-system-aar Not tainted 3.14.29+ #10
[ 6258.336465] task: ffffffc04bc89000 ti: ffffffc04bd10000 task.ti: ffffffc04bd10000
[ 6258.343889] PC is at kvm_timer_init+0x38/0x60
[ 6258.348196] LR is at kvm_arch_init_vm+0x74/0x90

Code: Select all

[ 6259.526798] Call trace:
[ 6259.529218] [<ffffffc0010aa940>] kvm_timer_init+0x38/0x60
[ 6259.534564] [<ffffffc00109e2b8>] kvm_arch_init_vm+0x70/0x90
[ 6259.540083] [<ffffffc00109a52c>] kvm_dev_ioctl+0xf4/0x478
[ 6259.545432] [<ffffffc0011e8770>] do_vfs_ioctl+0x338/0x598
[ 6259.550778] [<ffffffc0011e8a60>] SyS_ioctl+0x90/0xa8

nobe
Posts: 129
Joined: Sun Feb 07, 2016 9:52 pm
languages_spoken: english, french
ODROIDs: Odroid-C2
Contact:

Re: KVM

Unread post by nobe » Tue Mar 01, 2016 11:04 pm

It seems Andreas Faerber has worked (or is still working) on enabling virtualization for a S905 TV Box called vegas s95.
There may be good things to try from his commits :
https://github.com/afaerber/linux/tree/ ... rt/kvm/arm
https://github.com/afaerber/linux/tree/ ... ts/amlogic
HTH

Edit:
as stated in cnx-software.com comments, Andreas has submitted his code to mainline the s95 box.
http://www.cnx-software.com/2016/02/29/ ... ent-523156

so there may be useful information from the discussion with kernel maintainers
for example :
http://lists.infradead.org/pipermail/li ... 12736.html

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Wed Mar 02, 2016 1:51 am

I took a look at it and its pretty much what I am already testing with:
http://lists.infradead.org/pipermail/li ... 12616.html
That covers #1 in the OP and is the same thing I guessed.

http://lists.infradead.org/pipermail/li ... 12620.html
That covers #2 and effectively is the same "made up" entry I used.

The important part is here:
http://lists.infradead.org/pipermail/li ... 12738.html
> Do you know if the firmware allows the kernel to be entered in EL2
> (which is the arm64 name for HYP)?
> So can we run kvm?
> If you have a booted kernel, can you grep for "EL2" and "kvm" in the dmesg?

I do not have a rootfs yet (MMC v5 patches by Carlo are still waiting
for review), but with this change the KVM driver initializes okay - the
purpose of this patch!
Its a theoretical patch, not a tested/working patch. I achieved the same result. KVM initializes okay. However, it crashes if you actually try to use it.

mlinuxguy
Posts: 840
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, N1, USB-IO
Contact:

Re: KVM

Unread post by mlinuxguy » Wed Mar 02, 2016 2:49 am

This is an interesting patch set that might give ideas
Though perhaps you are past the GIC issues
arm/arm64: KVM: Detect vGIC presence at runtime

Code: Select all

Before commit 662d9715840aef44dcb573b0f9fab9e8319c868a ("arm/arm64: KVM: Kill CONFIG_KVM_ARM_{VGIC,TIMER}") is was possible to compile the kernel without vGIC and vTimer support. Commit message says about possibility to detect vGIC support in runtime, but this has never been implemented. This patch introduces runtime check, restoring the lost functionality. It again allows to use KVM on hardware without vGIC. Interrupt controller has to be emulated in userspace in this case. -ENODEV return code from probe function means there's no GIC at all. -ENXIO happens when, for example, there is GIC node in the device tree, but it does not specify vGIC resources. Any other error code is still treated as full stop because it might mean some really serious problems. 
https://git.kernel.org/cgit/linux/kerne ... eb9208cae7

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Wed Mar 02, 2016 3:09 am

The vGIC should be for boards without an ARM GIC (like RPi). We have a GIC, and the device tree entries that are missing are for the virtualized registers. The terminology is very confusing but if our hardware GIC has virtualization support (and it should) then everything is good. vGIC is the contingency plan where a hardware GIC and its virtualized registers are emulated in software. We should only need it if there is something preventing our hardware GIC's virtulization support from operating properly.

I think at this point if we can get the timer issue resolved (#2 in OP), then KVM should be up and running. This should also allow XEN to be brought up after incorporating the changes.

nobe
Posts: 129
Joined: Sun Feb 07, 2016 9:52 pm
languages_spoken: english, french
ODROIDs: Odroid-C2
Contact:

Re: KVM

Unread post by nobe » Wed Mar 02, 2016 3:31 am

Do you think this timer DT node implementation could prevent qemu from crashing during timer init phase ?
http://lists.infradead.org/pipermail/li ... 12620.html

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Wed Mar 02, 2016 3:56 am

There can be only one timer node in the device tree (I did try adding a separate entry for armv8-timer). To compensate, what I did was merge it with the meson-timer entry. The entry I created is functionally equivalent to what is presented in that patch:

Code: Select all

   timer{
      compatible = "arm, meson-timer","arm,armv8-timer";
      #address-cells = <2>;
      #size-cells = <2>;
      ranges;
      reg=   <0x0 0xc1109990 0x0 0x4 0x0 0xc1109988 0x0 0x4>;
      interrupts = <GIC_PPI 13 0xff01>,
                <GIC_PPI 14 0xff01>,
                  <GIC_PPI 11 0xff01>,
                  <GIC_PPI 10 0xff01>;
[edit]
If you look at the device tree, the meson-timer is per-cpu using IRQ 60, 61, 62, 63. The armv8-timer uses PPI with the IRQ's having a different meaning: they are IRQ per exception level. The arch-timer (virtual timer) uses the 3rd entry (IRQ 11) which is for virtualization. My interpretation is that we need to re-write arch-timer to use 4 different IRQs (for each CPU) rather than the current single shared (IRQ 11) among all cpu's. I may be completely wrong about this which is why someone with more expertise in how the board is wired internally will need to address it. As noted in the first post, I have pretty much done all I can do.

[edit 2]
Following the earlier example with vGIC, what we may need is vTimer. A driver that emulates armv8-timer using meson-timer.

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Wed Mar 02, 2016 5:25 am

Based on this:
http://infocenter.arm.com/help/index.js ... 02s01.html
I am not even sure our GIC device table entry is correct.

Code: Select all

   gic: interrupt-controller@2c001000 {
      compatible = "arm,cortex-a15-gic", "arm,cortex-a9-gic";
      #interrupt-cells = <3>;
      #address-cells = <0>;
      interrupt-controller;
      reg = <0x0 0xc4301000 0 0x1000>,
            <0x0 0xc4302000 0 0x0100>,
           <0x0 0xc4304000 0 0x2000>,
           <0x0 0xc4306000 0 0x2000>;
      interrupts = <GIC_PPI 9 0xf04>;
   };
1) note the base address of interrupt-controller@2c001000 differs from 0xc4301000.
2) the reg entries are of by an order of magnitued
0x00000-0x01FFF CPU Interface
0xc4301000 0 0x1000

0x10000-0x10FFF Virtual Interface Control
0xc4302000 0 0x0100

0x20000-0x21FFF Virtual CPU Interface
0xc4304000 0 0x2000

0x2F000-0x30FFF Alias of Virtual CPU Interface
0xc4306000 0 0x2000

nobe
Posts: 129
Joined: Sun Feb 07, 2016 9:52 pm
languages_spoken: english, french
ODROIDs: Odroid-C2
Contact:

Re: KVM

Unread post by nobe » Wed Mar 02, 2016 8:56 am

This might be a bad idea, but i still look at Andreas's repo

his arch_timer.c file has 14 more commits applied than ours.
https://github.com/afaerber/linux/commi ... ch_timer.c
This particular commit looks interesting : https://github.com/afaerber/linux/commi ... b56eef2201

Code: Select all

To address this issue, move the architected timers initialization to the
time when we run a VCPU for the first time, and then only initialize
(and enable) the architected timers if we have a properly created and
initialized in-kernel VGIC.
Could this be related to the crash you demonstrated ?
If yes, then i wonder if switching back to xen could be a better idea than "backporting" kvm code.

About the gic DT node, Andreas's one looks better than ours.
https://github.com/afaerber/linux/blob/ ... #L136-L147

Code: Select all

		gic: interrupt-controller@c4301000 {
			compatible = "arm,cortex-a15-gic", "arm,cortex-a9-gic";
			reg = <0x0 0xc4301000 0 0x1000>,
			      <0x0 0xc4302000 0 0x0100>,
			      <0x0 0xc4304000 0 0x2000>,
			      <0x0 0xc4306000 0 0x2000>;
			interrupt-controller;
			interrupts = <GIC_PPI 9
				(GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_HIGH)>;
			#interrupt-cells = <3>;
			#address-cells = <0>;
		};
And in the kernel discussion http://lists.infradead.org/pipermail/li ... 12757.html , they also talk about :
- using a size of 2000 in reg 2nd entry
- using some u-boot code to determine if it is gic-400 or not

PS: i now realize there are so many things i need to learn...

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Wed Mar 02, 2016 9:14 am

Thanks for the info. I will look at it further when I have free time again.
nobe wrote:i wonder if switching back to xen could be a better idea than "backporting" kvm code.
Xen has the same requirements: a GIC and timer. For now, I am sticking with KVM because its easier to get diagnostic information out of it since the rest of the kernel works even if KVM doesnt. If at a future point XEN looks easier, I have no issue with focusing on that instead.
nobe wrote:PS: i now realize there are so many things i need to learn...
I am in the exact same situation with this. ;)

nobe
Posts: 129
Joined: Sun Feb 07, 2016 9:52 pm
languages_spoken: english, french
ODROIDs: Odroid-C2
Contact:

Re: KVM

Unread post by nobe » Wed Mar 02, 2016 3:30 pm

just for information :
http://forum.odroid.com/viewtopic.php?p=127300#p127300
odroid wrote:we are focusing on the x11 Mali 64bit/32bit driver development and the XEN/KVM implementation at this moment.
Once the Kodi runs on Ubuntu 64bit, we may assign our possible resources for developing Kernel 4.4. or mainlining from May time frame.

nobe
Posts: 129
Joined: Sun Feb 07, 2016 9:52 pm
languages_spoken: english, french
ODROIDs: Odroid-C2
Contact:

Re: KVM

Unread post by nobe » Wed Mar 09, 2016 3:36 am

@crashoverride
did you get those libvirt 'illegal instruction' errors from dmesg ?


I received my C2 yesterday and i can finally start to play with it :)
Though i don't know what consequences there could be, i still tried those changes based on Andreas Farber's dts file.
Here is what it looks like :

Code: Select all

diff --git a/arch/arm64/boot/dts/amlogic/mesongxbb.dtsi b/arch/arm64/boot/dts/amlogic/mesongxbb.dtsi
index 05200a08..d0c4ab0 100644
--- a/arch/arm64/boot/dts/amlogic/mesongxbb.dtsi
+++ b/arch/arm64/boot/dts/amlogic/mesongxbb.dtsi
@@ -15,7 +15,6 @@
 			compatible = "arm,cortex-a53","arm,armv8";
 			reg = <0x0 0x0>;
 			enable-method = "psci";
-			timer=<&timer_f>;
 		};
 
 		cpu@1 {
@@ -23,14 +22,12 @@
 			compatible = "arm,cortex-a53","arm,armv8";
 			reg = <0x0 0x1>;
 			enable-method = "psci";
-			timer=<&timer_g>;
 		};
 		cpu@2 {
 			device_type = "cpu";
 			compatible = "arm,cortex-a53","arm,armv8";
 			reg = <0x0 0x2>;
 			enable-method = "psci";
-			timer=<&timer_h>;
 		};
 
 		cpu@3 {
@@ -38,72 +35,19 @@
 			compatible = "arm,cortex-a53","arm,armv8";
 			reg = <0x0 0x3>;
 			enable-method = "psci";
-			timer=<&timer_i>;
 		};
 
 
 	};
 
 	timer{
-		compatible = "arm, meson-timer";
-		#address-cells = <2>;
-		#size-cells = <2>;
-		ranges;
-		reg=   <0x0 0xc1109990 0x0 0x4 0x0 0xc1109988 0x0 0x4>;
-		timer_f:timer-f{
-			timer_name="MESON TIMER-F";
-			clockevent-rating=<300>;
-			clockevent-shift=<20>;
-			clockevent-features=<0x03>;
-			interrupts = <0 60 1>;
-			bit_enable=<16>;
-			bit_mode=<12>;
-			bit_resolution=<0>;
-			reg=   <0x0 0xc1109994 0x0 0x4>;
-		};
-		timer_g:timer-g{
-			timer_name="MESON TIMER-G";
-			clockevent-rating=<300>;
-			clockevent-shift=<20>;
-			clockevent-features=<0x03>;
-			interrupts = <0 61 1>;
-			bit_enable=<17>;
-			bit_mode=<13>;
-			bit_resolution=<2>;
-			reg=<0x0 0xc1109998 0x0 0x4>;
-		};
-		timer_h:timer-h{
-			timer_name="MESON TIMER-H";
-			clockevent-rating=<300>;
-			clockevent-shift=<20>;
-			clockevent-features=<0x03>;
-			interrupts = <0 62 1>;
-			bit_enable=<18>;
-			bit_mode=<14>;
-			bit_resolution=<4>;
-			reg=<0x0 0xc110999c 0x0 0x4>;
-		};
-		timer_i:timer-i{
-			timer_name="MESON TIMER-I";
-			clockevent-rating=<300>;
-			clockevent-shift=<20>;
-			clockevent-features=<0x03>;
-			interrupts = <0 63 1>;
-			bit_enable=<19>;
-			bit_mode=<15>;
-			bit_resolution=<6>;
-			reg=<0x0 0xc11099a0 0x0 0x4>;
-		};
-	};
-/*
-	timer {
 		compatible = "arm,armv8-timer";
-		interrupts = <GIC_PPI 13 0xff01>,
-				<GIC_PPI 14 0xff01>,
-			        <GIC_PPI 11 0xff01>,
-			        <GIC_PPI 10 0xff01>;
+		interrupts = <GIC_PPI 13 (GIC_CPU_MASK_RAW(0xff) | IRQ_TYPE_EDGE_RISING)>,
+			     	 <GIC_PPI 14 (GIC_CPU_MASK_RAW(0xff) | IRQ_TYPE_EDGE_RISING)>,
+			     	 <GIC_PPI 11 (GIC_CPU_MASK_RAW(0xff) | IRQ_TYPE_EDGE_RISING)>,
+			     	 <GIC_PPI 10 (GIC_CPU_MASK_RAW(0xff) | IRQ_TYPE_EDGE_RISING)>;
 	};
-*/
+
 	arm_pmu {
 		compatible = "arm,armv8-pmuv3";
 		interrupts = <0 137 4>,
@@ -223,13 +167,15 @@
 		  };
 
 	};
-	gic: interrupt-controller@2c001000 {
-		compatible = "arm,cortex-a15-gic", "arm,cortex-a9-gic";
+	gic: interrupt-controller@c4301000 {
+		compatible = "arm,gic-400", "arm,cortex-a15-gic";
 		#interrupt-cells = <3>;
 		#address-cells = <0>;
 		interrupt-controller;
 		reg = <0x0 0xc4301000 0 0x1000>,
-		      <0x0 0xc4302000 0 0x0100>;
+		      <0x0 0xc4302000 0 0x2000>,
+		      <0x0 0xc4304000 0 0x2000>,
+		      <0x0 0xc4306000 0 0x2000>;
 		interrupts = <GIC_PPI 9 0xf04>;
 	};
After compiling the dtb file and copying it over ssh to /media/boot , the kernel boots successfully and hyp mode is enabled.
You already had that working with your own changes so there is no real added value so far... that's why i'd like to see how you generated your libvirt errors to check how my current settings are.

I already installed libvirt-bin and its dependencies, virsh command doesn't crash so far :

Code: Select all

root@odroid64:~# virsh list 
setlocale: No such file or directory
 Id    Name                           State
----------------------------------------------------

root@odroid64:~# 
the qemu-system-aarch64 doesn't crash either :

Code: Select all

root@odroid64:~# qemu-system-aarch64 -machine list
qemu-system-aarch64: -machine list: Unsupported machine type
Use -machine help to list supported machines!
root@odroid64:~# qemu-system-aarch64 -machine help
Supported machines are:
lm3s811evb           Stellaris LM3S811EVB
canon-a1100          Canon PowerShot A1100 IS
vexpress-a15         ARM Versatile Express for Cortex-A15
vexpress-a9          ARM Versatile Express for Cortex-A9
xilinx-zynq-a9       Xilinx Zynq Platform Baseboard for Cortex-A9
connex               Gumstix Connex (PXA255)
n800                 Nokia N800 tablet aka. RX-34 (OMAP2420)
lm3s6965evb          Stellaris LM3S6965EVB
versatileab          ARM Versatile/AB (ARM926EJ-S)
borzoi               Borzoi PDA (PXA270)
tosa                 Tosa PDA (PXA255)
cheetah              Palm Tungsten|E aka. Cheetah PDA (OMAP310)
midway               Calxeda Midway (ECX-2000)
mainstone            Mainstone II (PXA27x)
n810                 Nokia N810 tablet aka. RX-44 (OMAP2420)
terrier              Terrier PDA (PXA270)
highbank             Calxeda Highbank (ECX-1000)
cubieboard           cubietech cubieboard
sx1-v1               Siemens SX1 (OMAP310) V1
sx1                  Siemens SX1 (OMAP310) V2
realview-eb-mpcore   ARM RealView Emulation Baseboard (ARM11MPCore)
kzm                  ARM KZM Emulation Baseboard (ARM1136)
akita                Akita PDA (PXA270)
z2                   Zipit Z2 (PXA27x)
musicpal             Marvell 88w8618 / MusicPal (ARM926EJ-S)
realview-pb-a8       ARM RealView Platform Baseboard for Cortex-A8
versatilepb          ARM Versatile/PB (ARM926EJ-S)
realview-eb          ARM RealView Emulation Baseboard (ARM926EJ-S)
realview-pbx-a9      ARM RealView Platform Baseboard Explore for Cortex-A9
spitz                Spitz PDA (PXA270)
none                 empty machine
virt                 ARM Virtual Machine
collie               Collie PDA (SA-1110)
smdkc210             Samsung SMDKC210 board (Exynos4210)
verdex               Gumstix Verdex (PXA270)
nuri                 Samsung NURI board (Exynos4210)
integratorcp         ARM Integrator/CP (ARM926EJ-S)
root@odroid64:~# 

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Wed Mar 09, 2016 7:24 am

nobe wrote:did you get those libvirt 'illegal instruction' errors from dmesg ?
the one for libvirt-bin shows up in dmesg. the one for qemu-system-aarch64 shows up on the console and in dmesg

Code: Select all

odroid@odroid64:~$ sudo service libvirt-bin start
Job for libvirt-bin.service failed because a fatal signal was delivered to the control process. See "systemctl status libvirt-bin.service" and "journalctl -xe" for details.

Code: Select all

odroid@odroid64:~$ qemu-system-aarch64
Illegal instruction

Code: Select all

[41383.862399] libvirtd[6757]: undefined instruction: pc=0000007f7ea38f38
[41383.862411] Code: d2800002 1e602008 54000701 d5033fdf (d53be040)
[41409.397927] qemu-system-aar[6784]: undefined instruction: pc=0000007f85b19f38
[41409.397940] Code: d2800002 1e602008 54000701 d5033fdf (d53be040)
My ubuntu version is non-current and I have a kernel modification application to support 64bit debugging.

mlinuxguy
Posts: 840
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, N1, USB-IO
Contact:

Re: KVM

Unread post by mlinuxguy » Wed Mar 09, 2016 2:01 pm

I'm deep into patch research for the crash utility fix and ran across this patch
arm64: Add support for hooks to handle undefined instructions
https://git.kernel.org/cgit/linux/kerne ... c3122809bb

Might help, dunno...
BTW, there is a whole series of patches dealing with instruction emulation around that time-period

nobe
Posts: 129
Joined: Sun Feb 07, 2016 9:52 pm
languages_spoken: english, french
ODROIDs: Odroid-C2
Contact:

Re: KVM

Unread post by nobe » Wed Mar 09, 2016 3:17 pm

OK, then this DT patch (see the 1st code block from my previous post) is definitely worth investigating.
libvirtd is starts up and no error appears
Btw, i use debian 64 bit from xeosal (got emmc 8GB so i need to save some space).

Code: Select all

root@odroid64:~# systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled)
   Active: active (running) since Tue 2016-03-08 23:00:41 MST; 11min ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 9778 (libvirtd)
   CGroup: /system.slice/libvirtd.service
           └─9778 /usr/sbin/libvirtd

Mar 08 23:00:41 odroid64 systemd[1]: Started Virtualization daemon.
root@odroid64:~# dmesg | grep -i libvirt
root@odroid64:~# 
now i guess my next step is to learn how to create a kvm powered vm using qemu-system-aarch64.
but i think at this point kvm backporting will be needed and this might too hard for a beginner like me :oops:

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Wed Mar 09, 2016 3:46 pm

nobe wrote:Btw, i use debian 64 bit from xeosal
It does work on debian. The issue is only on Ubuntu. I posted that info somewhere. Its probably on the other thread.

[Edit]
By "it does work" I mean that it does not fault with an illegal instruction. It still can not actually run a VM.

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Fri Mar 18, 2016 11:01 pm

With some device tree modifications, I did manage to get a VM to run and start booting before crashing:

Remove the meson-timer and use armv8-timer instead.

Code: Select all

	cpus:cpus {
		#address-cells = <2>;
		#size-cells = <0>;
		#cooling-cells = <2>; /* min followed by max */

		cpu@0 {
			device_type = "cpu";
			compatible = "arm,cortex-a53","arm,armv8";
			reg = <0x0 0x0>;
			enable-method = "psci";
			//timer=<&timer_f>;
		};

		cpu@1 {
			device_type = "cpu";
			compatible = "arm,cortex-a53","arm,armv8";
			reg = <0x0 0x1>;
			enable-method = "psci";
			//timer=<&timer_g>;
		};
		cpu@2 {
			device_type = "cpu";
			compatible = "arm,cortex-a53","arm,armv8";
			reg = <0x0 0x2>;
			enable-method = "psci";
			//timer=<&timer_h>;
		};

		cpu@3 {
			device_type = "cpu";
			compatible = "arm,cortex-a53","arm,armv8";
			reg = <0x0 0x3>;
			enable-method = "psci";
			//timer=<&timer_i>;
		};


	};

	/*
	timer{
		compatible = "arm, meson-timer","arm,armv8-timer";
		#address-cells = <2>;
		#size-cells = <2>;
		ranges;
		reg=   <0x0 0xc1109990 0x0 0x4 0x0 0xc1109988 0x0 0x4>;
		interrupts = <GIC_PPI 13 0xff08>,
                <GIC_PPI 14 0xff08>,
                <GIC_PPI 11 0xff08>,
                <GIC_PPI 10 0xff08>;
		timer_f:timer-f{
			timer_name="MESON TIMER-F";
			clockevent-rating=<300>;
			clockevent-shift=<20>;
			clockevent-features=<0x03>;
			interrupts = <0 60 1>;
			bit_enable=<16>;
			bit_mode=<12>;
			bit_resolution=<0>;
			reg=   <0x0 0xc1109994 0x0 0x4>;
		};
		timer_g:timer-g{
			timer_name="MESON TIMER-G";
			clockevent-rating=<300>;
			clockevent-shift=<20>;
			clockevent-features=<0x03>;
			interrupts = <0 61 1>;
			bit_enable=<17>;
			bit_mode=<13>;
			bit_resolution=<2>;
			reg=<0x0 0xc1109998 0x0 0x4>;
		};
		timer_h:timer-h{
			timer_name="MESON TIMER-H";
			clockevent-rating=<300>;
			clockevent-shift=<20>;
			clockevent-features=<0x03>;
			interrupts = <0 62 1>;
			bit_enable=<18>;
			bit_mode=<14>;
			bit_resolution=<4>;
			reg=<0x0 0xc110999c 0x0 0x4>;
		};
		timer_i:timer-i{
			timer_name="MESON TIMER-I";
			clockevent-rating=<300>;
			clockevent-shift=<20>;
			clockevent-features=<0x03>;
			interrupts = <0 63 1>;
			bit_enable=<19>;
			bit_mode=<15>;
			bit_resolution=<6>;
			reg=<0x0 0xc11099a0 0x0 0x4>;
		};
	};
	*/

	timer {
        compatible = "arm,armv8-timer";
		interrupt-parent = <&gic>;
        interrupts = <GIC_PPI 13 0xf08>,
                     <GIC_PPI 14 0xf08>,
                     <GIC_PPI 11 0xf08>,
                     <GIC_PPI 10 0xf08>;
		
    };
Also using modified GIC entry

Code: Select all

	gic: interrupt-controller@c4300000 {
      compatible = "arm,gic-400", "arm,cortex-a15-gic", "arm,cortex-a9-gic";
      #interrupt-cells = <3>;
      #address-cells = <0>;
      interrupt-controller;
      reg = <0x0 0xc4301000 0 0x1000>,
            <0x0 0xc4302000 0 0x2000>,
            <0x0 0xc4304000 0 0x2000>,
            <0x0 0xc4306000 0 0x2000>;
      interrupts = <GIC_PPI 9 0xf04>;
   };

The fault is:

Code: Select all

[  759.625076] virbr0: port 2(vnet0) entered listening state
[  759.625107] virbr0: port 2(vnet0) entered listening state
[  761.624728] virbr0: port 2(vnet0) entered learning state
[  762.638035] Unhandled fault: alignment fault (0x96000021) at 0xffffffc0386d37bc
[  762.639713] Internal error: : 96000021 [#1] PREEMPT SMP
[  762.644872] Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables fuse autofs4
[  762.668247] CPU: 0 PID: 4257 Comm: qemu-system-aar Not tainted 3.14.29+ #10
[  762.675145] task: ffffffc038618000 ti: ffffffc04a3dc000 task.ti: ffffffc04a3dc000
[  762.682568] PC is at set_bit+0x14/0x30
[  762.686273] LR is at vgic_v2_sync_lr_elrsr+0x38/0x48
[  762.691186] pc : [<ffffffc0013572f4>] lr : [<ffffffc0010a9ae8>] pstate: 40000145
[  762.698516] sp : ffffffc04a3dfc50
[  762.701794] x29: ffffffc04a3dfc50 x28: 0000000000000001
[  762.707055] x27: 0000000000000000 x26: 0000000000000021
[  762.712316] x25: 0000000000000021 x24: 0000000000000004
[  762.717577] x23: ffffffc04a3dfd18 x22: 0000000000000000
[  762.722838] x21: ffffffc0386d3000 x20: 0000000000000000
[  762.728100] x19: 0000000000000021 x18: 0000000000000000
[  762.733361] x17: ffffffc001083000 x16: 000000ff440c0400
[  762.738622] x15: 0000007fa01999c8 x14: 0000000000000000
[  762.743883] x13: 0000000000000000 x12: 0000000056000000
[  762.749145] x11: 00000032b5193519 x10: 0000000001ffa000
[  762.754406] x9 : 12b600005ba8e000 x8 : 000000003b9aca00
[  762.759667] x7 : 00000004468ee358 x6 : 0002cf349fe2fe04
[  762.764928] x5 : 0000000000000000 x4 : 0000000000000000
[  762.770190] x3 : 0000000000000001 x2 : 0000000000000001
[  762.775451] x1 : ffffffc0386d37bc x0 : 0000000000000000
[  762.780713]
               PC: 0xffffffc001357274:
[  762.785629] 7274  54fff140 52800015 17fffdab 12001403 4a030000 d2800022 8b400c21 9ac32043
[  762.793736] 7294  c85f7c22 ca030042 c8007c22 35ffffa0 d65f03c0 d503201f d503201f 12001403
[  762.801843] 72b4  4a030000 d2800022 8b400c21 9ac32043 c85f7c22 8a230042 c8007c22 35ffffa0
[  762.809951] 72d4  d65f03c0 d503201f d503201f 12001403 4a030000 d2800022 8b400c21 9ac32043
[  762.818058] 72f4  c85f7c22 aa030042 c8007c22 35ffffa0 d65f03c0 d503201f d503201f 12001403
[  762.826166] 7314  4a030000 d2800022 8b400c21 9ac32044 c85f7c22 9ac32440 ca040042 c805fc22
[  762.834273] 7334  35ffff85 d5033bbf 92400000 d65f03c0 d503201f d503201f d503201f 12001403
[  762.842381] 7354  4a030000 d2800022 8b400c21 9ac32044 c85f7c22 9ac32440 8a240042 c805fc22
[  762.850489]
               LR: 0xffffffc0010a9a68:
[  762.855404] 9a68  2a010041 2a000020 b907ae80 a94153f3 a8c27bfd d65f03c0 a9be7bfd 910003fd
[  762.863512] 9a88  f9000bf3 aa0003f3 aa1e03e0 d503201f b907ae7f 52800020 b907aa60 f9400bf3
[  762.871619] 9aa8  a8c27bfd d65f03c0 a9bd7bfd 910003fd a90153f3 f90013f5 aa0003f5 aa1e03e0
[  762.879727] 9ac8  2a0203f3 2a0103f4 d503201f f268067f 54000081 911ef2a1 2a1403e0 940ab5ff
[  762.887834] 9ae8  a94153f3 f94013f5 a8c37bfd d65f03c0 a9b57bfd 910003fd a90153f3 a9025bf5
[  762.895942] 9b08  a90363f7 aa0003f3 aa1e03e0 aa0103f7 aa0203f6 f0006175 d503201f 911042b4
[  762.904049] 9b28  aa1303e0 52800001 9413025a b9001680 34000f20 910103a2 aa1303e0 52800041
[  762.912157] 9b48  94130038 2a0003f8 35000d60 aa1303e0 52800041 9413007d f9000e80 b4000ba0
[  762.920265]
               SP: 0xffffffc04a3dfbd0:
[  762.925181] fbd0  00000000 00000000 386d3000 ffffffc0 00000000 00000000 4a3dfd18 ffffffc0
[  762.933288] fbf0  00000004 00000000 00000021 00000000 00000021 00000000 00000000 00000000
[  762.941396] fc10  00000001 00000000 4a3dfc50 ffffffc0 010a9ae8 ffffffc0 4a3dfc50 ffffffc0
[  762.949503] fc30  013572f4 ffffffc0 40000145 00000000 4a3dfc60 ffffffc0 01861680 ffffffc0
[  762.957611] fc50  4a3dfc80 ffffffc0 010a8008 ffffffc0 386d3000 ffffffc0 01cd8000 ffffffc0
[  762.965718] fc70  01cd8400 ffffffc0 01cd8000 ffffffc0 4a3dfd20 ffffffc0 0109e8c4 ffffffc0
[  762.973826] fc90  386d3000 ffffffc0 00000001 00000000 4a3dc000 ffffffc0 01cd29f0 ffffffc0
[  762.981933] fcb0  01c13000 ffffffc0 5c672000 ffffffc0 4a3dfda0 ffffffc0 01cd8380 ffffffc0
[  762.990042]
               X1: 0xffffffc0386d373c:
[  762.994957] 373c  00000000 00000000 00000000 00000000 00000000 80786623 00000000 93540006
[  763.003064] 375c  00000000 00018040 ffffff80 00090000 00000000 00000000 00000000 741b3e50
[  763.011172] 377c  ffffffc0 5bdc4a00 ffffffc0 00000000 00000000 52195700 ffffffc0 00000001
[  763.019279] 379c  00000000 00000004 00000000 00000001 f04c0001 00000001 00000001 00000000
[  763.027387] 37bc  0000000e 00000000 00000000 00000021 0000001b 00000000 00000000 00000000
[  763.035494] 37dc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  763.043602] 37fc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  763.051709] 381c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  763.059821]
               X17: 0xffffffc001082f80:
[  763.064819] 2f80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  763.072927] 2fa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  763.081034] 2fc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  763.089142] 2fe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  763.097249] 3000  14000279 d503201f d503201f d503201f d503201f d503201f d503201f d503201f
[  763.105357] 3020  d503201f d503201f d503201f d503201f d503201f d503201f d503201f d503201f
[  763.113464] 3040  d503201f d503201f d503201f d503201f d503201f d503201f d503201f d503201f
[  763.121572] 3060  d503201f d503201f d503201f d503201f d503201f d503201f d503201f d503201f
[  763.129681]
               X21: 0xffffffc0386d2f80:
[  763.134682] 2f80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  763.142789] 2fa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  763.150897] 2fc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  763.159004] 2fe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  763.167112] 3000  4a235400 ffffffc0 00000000 00000000 386182e8 ffffffc0 01cd8318 ffffffc0
[  763.175219] 3020  00000000 00000000 00000000 00000000 00000040 00000000 00000000 00000000
[  763.183327] 3040  00000000 00000000 386d3048 ffffffc0 386d3048 ffffffc0 38618000 ffffffc0
[  763.191434] 3060  00000000 00000000 5c672000 ffffffc0 00000000 00000000 00000000 00000000
[  763.199543]
               X23: 0xffffffc04a3dfc98:
[  763.204544] fc98  00000001 00000000 4a3dc000 ffffffc0 01cd29f0 ffffffc0 01c13000 ffffffc0
[  763.212652] fcb8  5c672000 ffffffc0 4a3dfda0 ffffffc0 01cd8380 ffffffc0 01cd8368 ffffffc0
[  763.220759] fcd8  00000000 00000000 01a27478 ffffffc0 4a235690 ffffffc0 00000001 ffffffc0
[  763.228867] fcf8  00000001 00000000 4a235400 ffffffc0 01cd29f0 00000000 01c13000 ffffffc0
[  763.236974] fd18  00000001 00000000 4a3dfdb0 ffffffc0 01099d58 ffffffc0 0000ae80 00000000
[  763.245082] fd38  386d3000 ffffffc0 4a3dc000 ffffffc0 00000000 00000000 5c783600 ffffffc0
[  763.253189] fd58  00000000 00000000 00000117 00000000 0000001d 00000000 01c11000 ffffffc0
[  763.261297] fd78  4a3dc000 ffffffc0 7ffbfeff fffffffe 010997e8 ffffffc0 0000ae80 00000000
[  763.269406]
               X29: 0xffffffc04a3dfbd0:
[  763.274407] fbd0  00000000 00000000 386d3000 ffffffc0 00000000 00000000 4a3dfd18 ffffffc0
[  763.282514] fbf0  00000004 00000000 00000021 00000000 00000021 00000000 00000000 00000000
[  763.290622] fc10  00000001 00000000 4a3dfc50 ffffffc0 010a9ae8 ffffffc0 4a3dfc50 ffffffc0
[  763.298729] fc30  013572f4 ffffffc0 40000145 00000000 4a3dfc60 ffffffc0 01861680 ffffffc0
[  763.306837] fc50  4a3dfc80 ffffffc0 010a8008 ffffffc0 386d3000 ffffffc0 01cd8000 ffffffc0
[  763.314944] fc70  01cd8400 ffffffc0 01cd8000 ffffffc0 4a3dfd20 ffffffc0 0109e8c4 ffffffc0
[  763.323052] fc90  386d3000 ffffffc0 00000001 00000000 4a3dc000 ffffffc0 01cd29f0 ffffffc0
[  763.331159] fcb0  01c13000 ffffffc0 5c672000 ffffffc0 4a3dfda0 ffffffc0 01cd8380 ffffffc0

[  763.340735] Process qemu-system-aar (pid: 4257, stack limit = 0xffffffc04a3dc058)
[  763.348152] Stack: (0xffffffc04a3dfc50 to 0xffffffc04a3e0000)
[  763.353846] fc40:                                     4a3dfc80 ffffffc0 010a8008 ffffffc0
[  763.361954] fc60: 386d3000 ffffffc0 01cd8000 ffffffc0 01cd8400 ffffffc0 01cd8000 ffffffc0
[  763.370062] fc80: 4a3dfd20 ffffffc0 0109e8c4 ffffffc0 386d3000 ffffffc0 00000001 00000000
[  763.378169] fca0: 4a3dc000 ffffffc0 01cd29f0 ffffffc0 01c13000 ffffffc0 5c672000 ffffffc0
[  763.386277] fcc0: 4a3dfda0 ffffffc0 01cd8380 ffffffc0 01cd8368 ffffffc0 00000000 00000000
[  763.394384] fce0: 01a27478 ffffffc0 4a235690 ffffffc0 00000001 ffffffc0 00000001 00000000
[  763.402492] fd00: 4a235400 ffffffc0 01cd29f0 00000000 01c13000 ffffffc0 00000001 00000000
[  763.410599] fd20: 4a3dfdb0 ffffffc0 01099d58 ffffffc0 0000ae80 00000000 386d3000 ffffffc0
[  763.418707] fd40: 4a3dc000 ffffffc0 00000000 00000000 5c783600 ffffffc0 00000000 00000000
[  763.426814] fd60: 00000117 00000000 0000001d 00000000 01c11000 ffffffc0 4a3dc000 ffffffc0
[  763.434922] fd80: 7ffbfeff fffffffe 010997e8 ffffffc0 0000ae80 00000000 386d3000 ffffffc0
[  763.443029] fda0: 4a3dc000 ffffffc0 00000000 00000000 4a3dfe10 ffffffc0 011e8774 ffffffc0
[  763.451137] fdc0: 0000ae80 00000000 00000000 00000000 5c783600 ffffffc0 5c849200 ffffffc0
[  763.459244] fde0: 00000010 00000000 013afb38 ffffffc0 4a3dfe50 ffffffc0 011d52e0 ffffffc0
[  763.467352] fe00: 4a3dfe20 ffffffc0 011f3b68 ffffffc0 4a3dfe90 ffffffc0 011e8a64 ffffffc0
[  763.475459] fe20: 00000000 00000000 5c783601 ffffffc0 5c783600 ffffffc0 00000010 00000000
[  763.483567] fe40: 0000ae80 00000000 4a3dc000 ffffffc0 4a3dfe60 ffffffc0 012e5360 ffffffc0
[  763.491674] fe60: 4a3dfe90 ffffffc0 011e8a1c ffffffc0 00000000 00000000 5c783601 ffffffc0
[  763.499782] fe80: 5c783600 ffffffc0 4a3dfed0 ffffffc0 843882a0 0000007f 0108425c ffffffc0
[  763.507889] fea0: 00000000 00000000 00fb3260 00000000 ffffffff ffffffff 8c65436c 0000007f
[  763.515997] fec0: 40000000 00000000 00000015 00000000 00000010 00000000 0000ae80 00000000
[  763.524104] fee0: 00000000 00000000 ffffff80 00000000 ffffffd0 00000000 843883a0 0000007f
[  763.532212] ff00: 00000000 00000000 00000001 00000000 0000001d 00000000 00000800 00000000
[  763.540319] ff20: 01194ab0 00000000 843882c8 0000007f 00000001 00000000 a913fb86 ffffffff
[  763.548427] ff40: 2a000000 0022e6b9 00000000 003b9aca 009a9740 00000000 8c654360 0000007f
[  763.556534] ff60: 00ece010 00000000 00a14000 00000000 00fb3260 00000000 00fbb260 00000000
[  763.564642] ff80: 00000001 00000000 00000000 00000000 83b8a000 0000007f 84388c70 0000007f
[  763.572749] ffa0: f73802f8 0000007f 8c702000 0000007f 00000001 00000000 843882a0 0000007f
[  763.580857] ffc0: 004a9658 00000000 843882a0 0000007f 8c65436c 0000007f 40000000 00000000
[  763.588964] ffe0: 00000010 00000000 0000001d 00000000 7139a72d 020f3620 5a845757 4bda8a9b
[  763.597068] Call trace:
[  763.599487] [<ffffffc0013572f4>] set_bit+0x14/0x30
[  763.604230] [<ffffffc0010a8004>] kvm_vgic_sync_hwstate+0x2dc/0x450
[  763.610355] [<ffffffc00109e8c0>] kvm_arch_vcpu_ioctl_run+0x158/0x478
[  763.616649] [<ffffffc001099d54>] kvm_vcpu_ioctl+0x5cc/0x6d0
[  763.622171] [<ffffffc0011e8770>] do_vfs_ioctl+0x338/0x598
[  763.627516] [<ffffffc0011e8a60>] SyS_ioctl+0x90/0xa8
[  763.632433] Code: 4a030000 d2800022 8b400c21 9ac32043 (c85f7c22)
[  763.638845] virbr0: topology change detected, propagating
[  763.643857] virbr0: port 2(vnet0) entered forwarding state
[  763.649441] ---[ end trace be62a7e43f3e022a ]---
Its an alignment fault related to GICv2. I was really excited to finally see a kernel boot.

[edit]
One line patch for issue is here
http://www.spinics.net/lists/kvm-arm/msg11336.html
KVM boots up a VM without crashing now.

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Sat Mar 19, 2016 12:10 am

Booting unmodified Ubuntu Server 14.04 cloud image works. (Ironically, its kernel is 3.13)

The issue now seems to be related to permissions with virt-manager + network manger to get internet access.

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Sat Mar 19, 2016 1:34 am

Boot log from unmodifed Ubuntu 16.04 kernel 4.4

Code: Select all

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 4.4.0-13-generic (buildd@bos01-arm64-011) (gcc version 5.3.1 20160225 (Ubuntu/Linaro 5.3.1-10ubuntu2) ) #29-Ubuntu SMP Fri Mar 11 19:30:56 UTC 2016 (Ubuntu 4.4.0-13.29-generic 4.4.5)
[    0.000000] Boot CPU: AArch64 Processor [410fd034]
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv0.2 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] PERCPU: Embedded 17 pages/cpu @ffff80001ffcc000 s31128 r8192 d30312 u69632
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: enabling workaround for ARM erratum 845719
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 129024
[    0.000000] Kernel command line: console=ttyAMA0 root=/dev/vda1
[    0.000000] PID hash table entries: 2048 (order: 2, 16384 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.000000] software IO TLB [mem 0x5ac00000-0x5ec00000] (64MB) mapped at [ffff80001ac00000-ffff80001ebfffff]
[    0.000000] Memory: 405032K/524288K available (8612K kernel code, 1013K rwdata, 3748K rodata, 752K init, 785K bss, 119256K reserved, 0K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vmalloc : 0xffff000000000000 - 0xffff7dffbfff0000   (129022 GB)
[    0.000000]     vmemmap : 0xffff7dffc0000000 - 0xffff7fffc0000000   (  2048 GB maximum)
[    0.000000]               0xffff7dffc0000000 - 0xffff7dffc0800000   (     8 MB actual)
[    0.000000]     fixed   : 0xffff7ffffa7fd000 - 0xffff7ffffac00000   (  4108 KB)
[    0.000000]     PCI I/O : 0xffff7ffffae00000 - 0xffff7ffffbe00000   (    16 MB)
[    0.000000]     modules : 0xffff7ffffc000000 - 0xffff800000000000   (    64 MB)
[    0.000000]     memory  : 0xffff800000000000 - 0xffff800020000000   (   512 MB)
[    0.000000]       .init : 0xffff800000c94000 - 0xffff800000d50000   (   752 KB)
[    0.000000]       .text : 0xffff800000080000 - 0xffff800000c94000   ( 12368 KB)
[    0.000000]       .data : 0xffff800000d61000 - 0xffff800000e5e600   (  1014 KB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	Build-time adjustment of leaf fanout to 64.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=128 to nr_cpu_ids=2.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=2
[    0.000000] NR_IRQS:64 nr_irqs:64 0
[    0.000000] GICv2m: range[0x8020000:0x8020fff], SPI[80:144]
[    0.000000] Architected cp15 timer(s) running at 24.00MHz (virt).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x588fe9dc0, max_idle_ns: 440795202592 ns
[    0.000003] sched_clock: 56 bits at 24MHz, resolution 41ns, wraps every 4398046511097ns
[    0.000354] Console: colour dummy device 80x25
[    0.000418] Calibrating delay loop (skipped), value calculated using timer frequency.. 48.00 BogoMIPS (lpj=96000)
[    0.000427] pid_max: default: 32768 minimum: 301
[    0.000515] Security Framework initialized
[    0.000547] Yama: becoming mindful.
[    0.000796] AppArmor: AppArmor initialized
[    0.001009] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes)
[    0.001014] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes)
[    0.002588] Initializing cgroup subsys io
[    0.002627] Initializing cgroup subsys memory
[    0.002679] Initializing cgroup subsys devices
[    0.002698] Initializing cgroup subsys freezer
[    0.002715] Initializing cgroup subsys net_cls
[    0.002732] Initializing cgroup subsys perf_event
[    0.002740] Initializing cgroup subsys net_prio
[    0.002747] Initializing cgroup subsys hugetlb
[    0.002869] Initializing cgroup subsys pids
[    0.002918] ftrace: allocating 30045 entries in 118 pages
[    0.075302] EFI services will not be available.
[    0.075360] ASID allocator initialised with 65536 entries
[    0.088255] Detected VIPT I-cache on CPU1
[    0.088387] CPU1: Booted secondary processor [410fd034]
[    0.088591] Brought up 2 CPUs
[    0.088604] SMP: Total of 2 processors activated.
[    0.088613] CPU: All CPU(s) started at EL1
[    0.088653] alternatives: patching kernel code
[    0.091896] devtmpfs: initialized
[    0.093803] evm: security.selinux
[    0.093807] evm: security.SMACK64
[    0.093810] evm: security.SMACK64EXEC
[    0.093812] evm: security.SMACK64TRANSMUTE
[    0.093815] evm: security.SMACK64MMAP
[    0.093818] evm: security.ima
[    0.093820] evm: security.capability
[    0.095295] DMI not present or invalid.
[    0.095684] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.096006] pinctrl core: initialized pinctrl subsystem
[    0.097374] NET: Registered protocol family 16
[    0.148063] cpuidle: using governor ladder
[    0.217815] cpuidle: using governor menu
[    0.217934] vdso: 2 pages (1 code @ ffff800000d69000, 1 data @ ffff800000d68000)
[    0.217965] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.218984] DMA: preallocated 256 KiB pool for atomic allocations
[    0.219255] Serial: AMBA PL011 UART driver
[    0.223958] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 38, base_baud = 0) is a PL011 rev1
[    0.401118] console [ttyAMA0] enabled
[    0.479946] ACPI: Interpreter disabled.
[    0.481290] vgaarb: loaded
[    0.483306] SCSI subsystem initialized
[    0.484449] usbcore: registered new interface driver usbfs
[    0.485596] usbcore: registered new interface driver hub
[    0.486637] usbcore: registered new device driver usb
[    0.494862] dmi: Firmware registration failed.
[    0.500664] NetLabel: Initializing
[    0.501950] NetLabel:  domain hash size = 128
[    0.503344] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.505049] NetLabel:  unlabeled traffic allowed by default
[    0.510602] clocksource: Switched to clocksource arch_sys_counter
[    0.561025] AppArmor: AppArmor Filesystem Enabled
[    0.573058] pnp: PnP ACPI: disabled
[    0.590127] NET: Registered protocol family 2
[    0.604483] TCP established hash table entries: 4096 (order: 3, 32768 bytes)
[    0.607958] TCP bind hash table entries: 4096 (order: 4, 65536 bytes)
[    0.610179] TCP: Hash tables configured (established 4096 bind 4096)
[    0.612235] UDP hash table entries: 256 (order: 1, 8192 bytes)
[    0.614033] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    0.616040] NET: Registered protocol family 1
[    0.617648] Trying to unpack rootfs image as initramfs...
[    9.139467] Freeing initrd memory: 20840K (ffff800008000000 - ffff80000945a000)
[    9.142280] kvm [1]: HYP mode not available
[    9.144297] futex hash table entries: 512 (order: 4, 65536 bytes)
[    9.146885] audit: initializing netlink subsys (disabled)
[    9.148983] audit: type=2000 audit(8.860:1): initialized
[    9.151383] Initialise system trusted keyring
[    9.153640] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    9.161087] zbud: loaded
[    9.163456] VFS: Disk quotas dquot_6.6.0
[    9.165026] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    9.169650] fuse init (API version 7.23)
[    9.171816] Key type big_key registered
[    9.175789] Key type asymmetric registered
[    9.177459] Asymmetric key parser 'x509' registered
[    9.181470] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
[    9.184574] io scheduler noop registered
[    9.186150] io scheduler deadline registered (default)
[    9.188433] io scheduler cfq registered
[    9.190925] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    9.192923] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    9.195538] PCI host bridge /pcie@10000000 ranges:
[    9.197052]    IO 0x3eff0000..0x3effffff -> 0x00000000
[    9.198881]   MEM 0x10000000..0x3efeffff -> 0x10000000
[    9.200848]   MEM 0x8000000000..0xffffffffff -> 0x8000000000
[    9.203683] pci-host-generic 3f000000.pcie: PCI host bridge to bus 0000:00
[    9.206700] pci_bus 0000:00: root bus resource [bus 00-0f]
[    9.209413] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    9.211722] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]
[    9.214258] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]
[    9.218068] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    9.224730] pci 0000:01:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    9.240324] pci 0000:00:01.0: BAR 14: assigned [mem 0x10000000-0x100fffff]
[    9.247157] pci 0000:01:01.0: BAR 0: assigned [mem 0x10000000-0x100000ff 64bit]
[    9.257057] pci 0000:01:01.0: PCI bridge to [bus 02]
[    9.265509] pci 0000:00:01.0: PCI bridge to [bus 01-02]
[    9.269414] pci 0000:00:01.0:   bridge window [mem 0x10000000-0x100fffff]
[    9.289970] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[    9.320489] msm_serial: driver initialized
[    9.323285] Unable to detect cache hierarchy from DT for CPU 0
[    9.343931] brd: module loaded
[    9.357625] loop: module loaded
[    9.364388]  vda: vda1
[    9.367971] libphy: Fixed MDIO Bus: probed
[    9.370134] tun: Universal TUN/TAP device driver, 1.6
[    9.372657] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    9.380742] PPP generic driver version 2.4.2
[    9.386653] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    9.395654] ehci-pci: EHCI PCI platform driver
[    9.407808] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    9.415301] ohci-pci: OHCI PCI platform driver
[    9.417048] uhci_hcd: USB Universal Host Controller Interface driver
[    9.421280] mousedev: PS/2 mouse device common for all mice
[    9.424703] i2c /dev entries driver
[    9.427406] device-mapper: uevent: version 1.0.3
[    9.429891] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com
[    9.433010] ledtrig-cpu: registered to indicate activity on CPUs
[    9.435943] NET: Registered protocol family 10
[    9.438763] NET: Registered protocol family 17
[    9.440478] Key type dns_resolver registered
[    9.442128] Registered cp15_barrier emulation handler
[    9.444816] Registered setend emulation handler
[    9.447486] registered taskstats version 1
[    9.449004] Loading compiled-in X.509 certificates
[    9.452931] Loaded X.509 cert 'Build time autogenerated kernel key: 168907e71dae1fccbc216b487b0cc43f22ef1818'
[    9.456251] zswap: loaded using pool lzo/zbud
[    9.465995] Key type trusted registered
[    9.492847] Key type encrypted registered
[    9.494327] AppArmor: AppArmor sha1 policy hashing enabled
[    9.496060] ima: No TPM chip found, activating TPM-bypass!
[    9.498045] evm: HMAC attrs: 0x1
[    9.499962] hctosys: unable to open rtc device (rtc0)
[    9.509064] uart-pl011 9000000.pl011: no DMA platform data
[    9.514057] Freeing unused kernel memory: 752K (ffff800000c94000 - ffff800000d50000)
[    9.517556] Freeing alternatives memory: 48K (ffff800000d50000 - ffff800000d5c000)
Loading, please wait...
starting version 229
[    9.578480] random: systemd-udevd urandom read with 1 bits of entropy available
[    9.733048] scsi host0: Virtio SCSI HBA
Begin: Loading essential drivers ... [   11.195495] md: linear personality registered for level -1
[   11.213774] md: multipath personality registered for level -4
[   11.224135] md: raid0 personality registered for level 0
[   11.236487] md: raid1 personality registered for level 1
[   11.327557] raid6: int64x1  gen()   522 MB/s
[   11.407539] raid6: int64x1  xor()   535 MB/s
[   11.487629] raid6: int64x2  gen()   861 MB/s
[   11.567619] raid6: int64x2  xor()   787 MB/s
[   11.658839] raid6: int64x4  gen()  1635 MB/s
[   11.757584] raid6: int64x4  xor()  1041 MB/s
[   11.837518] raid6: int64x8  gen()  1501 MB/s
[   11.917596] raid6: int64x8  xor()  1017 MB/s
[   11.997637] raid6: neonx1   gen()   915 MB/s
[   12.077500] raid6: neonx1   xor()   971 MB/s
[   12.157496] raid6: neonx2   gen()  1548 MB/s
[   12.237590] raid6: neonx2   xor()  1438 MB/s
[   12.317624] raid6: neonx4   gen()  2058 MB/s
[   12.397850] raid6: neonx4   xor()  1663 MB/s
[   12.487606] raid6: neonx8   gen()  2437 MB/s
[   12.567679] raid6: neonx8   xor()  1698 MB/s
[   12.569409] raid6: using algorithm neonx8 gen() 2437 MB/s
[   12.571242] raid6: .... xor() 1698 MB/s, rmw enabled
[   12.572901] raid6: using intx1 recovery algorithm
[   12.577857] xor: measuring software checksum speed
[   12.618616]    8regs     :  2832.000 MB/sec
[   12.670607]    8regs_prefetch:  2605.000 MB/sec
[   12.710648]    32regs    :  3423.000 MB/sec
[   12.750659]    32regs_prefetch:  2909.000 MB/sec
[   12.752325] xor: using function: 32regs (3423.000 MB/sec)
[   12.757419] async_tx: api initialized (async)
[   12.782008] md: raid6 personality registered for level 6
[   12.784223] md: raid5 personality registered for level 5
[   12.786441] md: raid4 personality registered for level 4
[   12.805355] md: raid10 personality registered for level 10
done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... [   12.915827] Btrfs loaded
Scanning for Btrfs filesystems
done.
Warning: fsck not present, so skipping root file system
[   13.117284] EXT4-fs (vda1): INFO: recovery required on readonly filesystem
[   13.133622] EXT4-fs (vda1): write access will be enabled during recovery
[   13.333286] EXT4-fs (vda1): recovery complete
[   13.336780] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)
done.
Begin: Running /scripts/local-bottom ... done.
Begin: Running /scripts/init-bottom ... Warning: overlayroot: debug is busted
done.
[   13.662267] systemd[1]: System time before build time, advancing clock.
[   13.717666] systemd[1]: systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN)
[   13.724263] systemd[1]: Detected virtualization qemu.
[   13.726275] systemd[1]: Detected architecture arm64.

Welcome to Ubuntu 16.04!


User avatar
odroid
Site Admin
Posts: 30028
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Contact:

Re: KVM

Unread post by odroid » Sat Mar 19, 2016 10:51 am

:o Ubuntu 16.04 kernel 4.4 on the KVM :o

nobe
Posts: 129
Joined: Sun Feb 07, 2016 9:52 pm
languages_spoken: english, french
ODROIDs: Odroid-C2
Contact:

Re: KVM

Unread post by nobe » Sat Mar 19, 2016 3:48 pm

awesome news, well done crashoverride
i'll need to find some time to play again with this :)

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Sat Mar 19, 2016 11:03 pm

I hit another milestone. I have successfully passed through a USB device from the C2 host to a VM running stock linux kernel 4.4. This is accomplished with 'usbip'.

Code: Select all

[   31.885688] usbip_core: USB/IP Core v1.0.0
[   61.279908] vhci_hcd vhci_hcd: USB/IP Virtual Host Controller
[   61.279922] vhci_hcd vhci_hcd: new USB bus registered, assigned bus number 2
[   61.287665] hub 2-0:1.0: USB hub found
[   61.287698] hub 2-0:1.0: 8 ports detected
[   61.288645] vhci_hcd: USB/IP 'Virtual' Host Controller (VHCI) Driver v1.0.0
[  105.283126] random: nonblocking pool is initialized
[  160.428115] vhci_hcd vhci_hcd: rhport(0) sockfd(3) devid(65539) speed(3) speed_str(high-speed)
[  160.714625] usb 2-1: new high-speed USB device number 2 using vhci_hcd
[  161.014628] usb 2-1: new high-speed USB device number 3 using vhci_hcd
[  161.304617] usb 2-1: new high-speed USB device number 4 using vhci_hcd
[  161.304641] usb 2-1: SetAddress Request (4) to port 0
[  161.387745] Linux video capture interface: v2.00
[  161.414809] em28xx: New device  USB 2875 Device @ 480 Mbps (1b80:e34c, interface 0, class 0)
[  161.414824] em28xx: DVB interface 0 found: bulk
[  161.415872] em28xx: chip ID is em2874
[  162.023411] em2874 #0: EEPROM ID = 26 00 01 00, EEPROM hash = 0x5d3e97ab
[  162.023423] em2874 #0: EEPROM info:
[  162.023427] em2874 #0: 	microcode start address = 0x0004, boot configuration = 0x01
[  162.060253] em2874 #0: 	AC97 audio (5 sample rates)
[  162.060269] em2874 #0: 	500mA max power
[  162.060274] em2874 #0: 	Table at offset 0x24, strings=0x206a, 0x048a, 0x0000
[  162.060466] em2874 #0: Identified as KWorld USB ATSC TV Stick UB435-Q V3 (card=93)
[  162.060473] em2874 #0: dvb set to bulk mode.
[  162.060626] usbcore: registered new interface driver em28xx
[  162.091974] em2874 #0: Binding DVB extension
[  162.131900] tda18212 1-0060: NXP TDA18212HN/M successfully identified
[  162.131953] DVB: registering new adapter (em2874 #0)
[  162.131968] usb 2-1: DVB: registering adapter 0 frontend 0 (LG Electronics LGDT3305 VSB/QAM Frontend)...
[  162.132864] em2874 #0: DVB extension successfully initialized
[  162.132876] em28xx: Registered (Em28xx dvb Extension) extension
This shows the Linux 4.4 kernel running in a VM using my USB TV tuner

blah666
Posts: 1
Joined: Mon Mar 07, 2016 10:41 pm
languages_spoken: english
ODROIDs: C2
Contact:

Re: KVM

Unread post by blah666 » Mon Mar 21, 2016 1:54 pm

thanks to all of you for the pointers and clues... I've also gotten KVM running with an Ubuntu cloud image at kernel v4.4.

cmetz
Posts: 4
Joined: Mon Apr 25, 2016 9:52 am
languages_spoken: english
Contact:

Re: KVM

Unread post by cmetz » Mon Apr 25, 2016 10:05 am

crashoverride, I'm trying to get KVM working with your pre-merged kernel:

https://github.com/OtherCrashOverride/l ... rge-staged

dmesg says:

[ 0.381719] kvm [1]: Using HYP init bounce page @5bebc000
[ 0.381843] kvm [1]: Cannot obtain GICH resource
[ 0.381866] kvm [1]: error initializing Hyp mode: -22

I looked into the dts/dtb and did not find the changes you posted earlier in this thread, but when I tried to make them I ended up with a non-functional kernel.

How would you recommend going about getting to a working KVM now that you've done this KVM work and your more general kernel cherry-pick and merge work?

Also, crashoverride - thank you for all of your great work!

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Mon Apr 25, 2016 10:34 am

cmetz wrote:[ 0.381843] kvm [1]: Cannot obtain GICH resource
That means the device tree does not contain the required information.

I have not posted the device tree patch on github yet because its still "evolving". The current one I am using is mentioned in this thread:
http://forum.odroid.com/viewtopic.php?f ... 46#p136192

Code: Select all

commit fedcd409e69074f500dd3eb709b6bf6740a02179
Author: OtherCrashOverride <OtherCrashOverride@github.com>
Date:   Thu Apr 14 11:06:37 2016 -0400

    ODROID-C2: Use armv8 timer.  Add compatible entries to GIC for KVM support.

diff --git a/arch/arm64/boot/dts/amlogic/mesongxbb.dtsi b/arch/arm64/boot/dts/amlogic/mesongxbb.dtsi
old mode 100644
new mode 100755
index aa21680..2036895
--- a/arch/arm64/boot/dts/amlogic/mesongxbb.dtsi
+++ b/arch/arm64/boot/dts/amlogic/mesongxbb.dtsi
@@ -15,7 +15,7 @@
                        compatible = "arm,cortex-a53","arm,armv8";
                        reg = <0x0 0x0>;
                        enable-method = "psci";
-                       timer=<&timer_f>;
+                       //timer=<&timer_f>;
                };

                cpu@1 {
@@ -23,14 +23,14 @@
                        compatible = "arm,cortex-a53","arm,armv8";
                        reg = <0x0 0x1>;
                        enable-method = "psci";
-                       timer=<&timer_g>;
+                       //timer=<&timer_g>;
                };
                cpu@2 {
                        device_type = "cpu";
                        compatible = "arm,cortex-a53","arm,armv8";
                        reg = <0x0 0x2>;
                        enable-method = "psci";
-                       timer=<&timer_h>;
+                       //timer=<&timer_h>;
                };

                cpu@3 {
@@ -38,13 +38,12 @@
                        compatible = "arm,cortex-a53","arm,armv8";
                        reg = <0x0 0x3>;
                        enable-method = "psci";
-                       timer=<&timer_i>;
+                       //timer=<&timer_i>;
                };


        };
-
-       timer{
+/*     timer{
                compatible = "arm, meson-timer";
                #address-cells = <2>;
                #size-cells = <2>;
@@ -94,29 +93,27 @@
                        bit_resolution=<6>;
                        reg=<0x0 0xc11099a0 0x0 0x4>;
                };
-       };
-/*
+       }; */
        timer {
                compatible = "arm,armv8-timer";
                interrupts = <GIC_PPI 13 0xff01>,
-                               <GIC_PPI 14 0xff01>,
-                               <GIC_PPI 11 0xff01>,
-                               <GIC_PPI 10 0xff01>;
+                                        <GIC_PPI 14 0xff01>,
+                                <GIC_PPI 11 0xff01>,
+                                <GIC_PPI 10 0xff01>;
        };
-*/
        arm_pmu {
                compatible = "arm,armv8-pmuv3";
                interrupts = <0 137 4>,
                             <0 138 4>,
                             <0 153 4>,
                             <0 154 4>;
-       };
+       };
        meson_suspend:pm{
                   compatible = "amlogic, pm";
                   gxbaby-suspend;
                   reg = <0x0 0xc81000a8 0x0 0x4
                                0x0 0xc810023c 0x0 0x4>;
-       };
+       };
        gpu:mali@d00c0000{
                #cooling-cells = <2>; /* min followed by max */
                compatible = "arm,mali-450";
@@ -223,8 +220,9 @@
                  };

        };
+
        gic: interrupt-controller@c4301000{
-               compatible = "arm,gic-400";
+               compatible = "arm,gic-400", "arm,cortex-a15-gic", "arm,cortex-a9-gic";
                #interrupt-cells = <3>;
                #address-cells = <0>;
                interrupt-controller;
The long term solution I would like to see would be in the form of a boot.ini option such as "virtualization = 1". This should either load a different device tree or dynamically patch the current one at boot time using uboot's device tree support. I think providing a different device tree would be the more stable/robust approach and least likely to break. I am not sure what HardKernel's plans are in this area so I am awaiting the Ubuntu 16.04 release (not-beta) in May.

User avatar
Brian.K
Posts: 246
Joined: Tue Sep 15, 2015 7:30 pm
languages_spoken: English, Korean
ODROIDs: XU4, C1+, C2
Location: South Korea
Contact:

Re: KVM

Unread post by Brian.K » Tue Apr 26, 2016 10:04 am

crashoverride wrote:
cmetz wrote:[ 0.381843] kvm [1]: Cannot obtain GICH resource
That means the device tree does not contain the required information.

I have not posted the device tree patch on github yet because its still "evolving". The current one I am using is mentioned in this thread:
http://forum.odroid.com/viewtopic.php?f ... 46#p136192

Code: Select all

commit fedcd409e69074f500dd3eb709b6bf6740a02179
Author: OtherCrashOverride <OtherCrashOverride@github.com>
Date:   Thu Apr 14 11:06:37 2016 -0400

    ODROID-C2: Use armv8 timer.  Add compatible entries to GIC for KVM support.

diff --git a/arch/arm64/boot/dts/amlogic/mesongxbb.dtsi b/arch/arm64/boot/dts/amlogic/mesongxbb.dtsi
old mode 100644
new mode 100755
index aa21680..2036895
--- a/arch/arm64/boot/dts/amlogic/mesongxbb.dtsi
+++ b/arch/arm64/boot/dts/amlogic/mesongxbb.dtsi
@@ -15,7 +15,7 @@
                        compatible = "arm,cortex-a53","arm,armv8";
                        reg = <0x0 0x0>;
                        enable-method = "psci";
-                       timer=<&timer_f>;
+                       //timer=<&timer_f>;
                };

                cpu@1 {
@@ -23,14 +23,14 @@
                        compatible = "arm,cortex-a53","arm,armv8";
                        reg = <0x0 0x1>;
                        enable-method = "psci";
-                       timer=<&timer_g>;
+                       //timer=<&timer_g>;
                };
                cpu@2 {
                        device_type = "cpu";
                        compatible = "arm,cortex-a53","arm,armv8";
                        reg = <0x0 0x2>;
                        enable-method = "psci";
-                       timer=<&timer_h>;
+                       //timer=<&timer_h>;
                };

                cpu@3 {
@@ -38,13 +38,12 @@
                        compatible = "arm,cortex-a53","arm,armv8";
                        reg = <0x0 0x3>;
                        enable-method = "psci";
-                       timer=<&timer_i>;
+                       //timer=<&timer_i>;
                };


        };
-
-       timer{
+/*     timer{
                compatible = "arm, meson-timer";
                #address-cells = <2>;
                #size-cells = <2>;
@@ -94,29 +93,27 @@
                        bit_resolution=<6>;
                        reg=<0x0 0xc11099a0 0x0 0x4>;
                };
-       };
-/*
+       }; */
        timer {
                compatible = "arm,armv8-timer";
                interrupts = <GIC_PPI 13 0xff01>,
-                               <GIC_PPI 14 0xff01>,
-                               <GIC_PPI 11 0xff01>,
-                               <GIC_PPI 10 0xff01>;
+                                        <GIC_PPI 14 0xff01>,
+                                <GIC_PPI 11 0xff01>,
+                                <GIC_PPI 10 0xff01>;
        };
-*/
        arm_pmu {
                compatible = "arm,armv8-pmuv3";
                interrupts = <0 137 4>,
                             <0 138 4>,
                             <0 153 4>,
                             <0 154 4>;
-       };
+       };
        meson_suspend:pm{
                   compatible = "amlogic, pm";
                   gxbaby-suspend;
                   reg = <0x0 0xc81000a8 0x0 0x4
                                0x0 0xc810023c 0x0 0x4>;
-       };
+       };
        gpu:mali@d00c0000{
                #cooling-cells = <2>; /* min followed by max */
                compatible = "arm,mali-450";
@@ -223,8 +220,9 @@
                  };

        };
+
        gic: interrupt-controller@c4301000{
-               compatible = "arm,gic-400";
+               compatible = "arm,gic-400", "arm,cortex-a15-gic", "arm,cortex-a9-gic";
                #interrupt-cells = <3>;
                #address-cells = <0>;
                interrupt-controller;
The long term solution I would like to see would be in the form of a boot.ini option such as "virtualization = 1". This should either load a different device tree or dynamically patch the current one at boot time using uboot's device tree support. I think providing a different device tree would be the more stable/robust approach and least likely to break. I am not sure what HardKernel's plans are in this area so I am awaiting the Ubuntu 16.04 release (not-beta) in May.
I agree the 'virtualization' option in boot.ini file.
Could you send the patch by email to me? or Could I modify the patch something?

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Tue Apr 26, 2016 10:47 am

Right now I am tied up with business trips. I will get back to you on this later in the week.

cmetz
Posts: 4
Joined: Mon Apr 25, 2016 9:52 am
languages_spoken: english
Contact:

Re: KVM

Unread post by cmetz » Wed Apr 27, 2016 8:23 am

crashoverride,

The patch you linked to has an unmodified line that says:

gic: interrupt-controller@c4301000{

My checkout of your tree instead has:

gic: interrupt-controller@2c001000 {

The register entries are:

reg = <0x0 0xc4301000 0 0x1000>,
<0x0 0xc4302000 0 0x0100>;

Which leads me to believe that the former is correct and not the latter. Is that right?

cmetz
Posts: 4
Joined: Mon Apr 25, 2016 9:52 am
languages_spoken: english
Contact:

Re: KVM

Unread post by cmetz » Wed Apr 27, 2016 8:42 am

[Update: solution below]
Last edited by cmetz on Wed Apr 27, 2016 8:50 am, edited 1 time in total.

cmetz
Posts: 4
Joined: Mon Apr 25, 2016 9:52 am
languages_spoken: english
Contact:

Re: KVM

Unread post by cmetz » Wed Apr 27, 2016 8:49 am

crashoverride wrote: Also using modified GIC entry

Code: Select all

	gic: interrupt-controller@c4300000 {
      compatible = "arm,gic-400", "arm,cortex-a15-gic", "arm,cortex-a9-gic";
      #interrupt-cells = <3>;
      #address-cells = <0>;
      interrupt-controller;
      reg = <0x0 0xc4301000 0 0x1000>,
            <0x0 0xc4302000 0 0x2000>,
            <0x0 0xc4304000 0 0x2000>,
            <0x0 0xc4306000 0 0x2000>;
      interrupts = <GIC_PPI 9 0xf04>;
   };
When I used this entry exactly, that resolved both errors. That is, I had to change the address from 2c001000 to c4301000 and change the reg entries to match.

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Wed Apr 27, 2016 2:43 pm

cmetz wrote:The patch you linked to has an unmodified line that says:

gic: interrupt-controller@c4301000{

My checkout of your tree instead has:

gic: interrupt-controller@2c001000 {
Yes, the patch is against the HardKernel tree. That line was modified by the following commit:
https://github.com/hardkernel/linux/com ... 5c54b573ae

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Sat Apr 30, 2016 6:00 pm

@Brian.K,
The following kernel patch is required to use KVM:
https://github.com/OtherCrashOverride/l ... fd843eb609

I have already posted the device tree patch in this thread. I have included the complete file with patch applied for reference here:
https://gist.github.com/OtherCrashOverr ... 86e25c4e8b

Next, there needs to be a decision made to either include two (2) device trees [one using meson-timer, the other using armv8-timer] or to dynamically patch a single device tree using uboot commands. After that, we can add "virtualization = 1" as an option to boot.ini.

User avatar
Brian.K
Posts: 246
Joined: Tue Sep 15, 2015 7:30 pm
languages_spoken: English, Korean
ODROIDs: XU4, C1+, C2
Location: South Korea
Contact:

Re: KVM

Unread post by Brian.K » Tue May 17, 2016 5:25 pm

@crashoverride
I'm sorry for late response.

https://github.com/OtherCrashOverride/l ... fd843eb609
This patch looks good for me. I will cherry-pick it soon.
crashoverride wrote:

Code: Select all

commit fedcd409e69074f500dd3eb709b6bf6740a02179
Author: OtherCrashOverride <OtherCrashOverride@github.com>
Date:   Thu Apr 14 11:06:37 2016 -0400

    ODROID-C2: Use armv8 timer.  Add compatible entries to GIC for KVM support.

diff --git a/arch/arm64/boot/dts/amlogic/mesongxbb.dtsi b/arch/arm64/boot/dts/amlogic/mesongxbb.dtsi
old mode 100644
new mode 100755
index aa21680..2036895
--- a/arch/arm64/boot/dts/amlogic/mesongxbb.dtsi
+++ b/arch/arm64/boot/dts/amlogic/mesongxbb.dtsi
@@ -15,7 +15,7 @@
                        compatible = "arm,cortex-a53","arm,armv8";
                        reg = <0x0 0x0>;
                        enable-method = "psci";
-                       timer=<&timer_f>;
+                       //timer=<&timer_f>;
                };

                cpu@1 {
@@ -23,14 +23,14 @@
                        compatible = "arm,cortex-a53","arm,armv8";
                        reg = <0x0 0x1>;
                        enable-method = "psci";
-                       timer=<&timer_g>;
+                       //timer=<&timer_g>;
                };
                cpu@2 {
                        device_type = "cpu";
                        compatible = "arm,cortex-a53","arm,armv8";
                        reg = <0x0 0x2>;
                        enable-method = "psci";
-                       timer=<&timer_h>;
+                       //timer=<&timer_h>;
                };

                cpu@3 {
@@ -38,13 +38,12 @@
                        compatible = "arm,cortex-a53","arm,armv8";
                        reg = <0x0 0x3>;
                        enable-method = "psci";
-                       timer=<&timer_i>;
+                       //timer=<&timer_i>;
                };


        };
-
-       timer{
+/*     timer{
                compatible = "arm, meson-timer";
                #address-cells = <2>;
                #size-cells = <2>;
@@ -94,29 +93,27 @@
                        bit_resolution=<6>;
                        reg=<0x0 0xc11099a0 0x0 0x4>;
                };
-       };
-/*
+       }; */
        timer {
                compatible = "arm,armv8-timer";
                interrupts = <GIC_PPI 13 0xff01>,
-                               <GIC_PPI 14 0xff01>,
-                               <GIC_PPI 11 0xff01>,
-                               <GIC_PPI 10 0xff01>;
+                                        <GIC_PPI 14 0xff01>,
+                                <GIC_PPI 11 0xff01>,
+                                <GIC_PPI 10 0xff01>;
        };
-*/
        arm_pmu {
                compatible = "arm,armv8-pmuv3";
                interrupts = <0 137 4>,
                             <0 138 4>,
                             <0 153 4>,
                             <0 154 4>;
-       };
+       };
        meson_suspend:pm{
                   compatible = "amlogic, pm";
                   gxbaby-suspend;
                   reg = <0x0 0xc81000a8 0x0 0x4
                                0x0 0xc810023c 0x0 0x4>;
-       };
+       };
        gpu:mali@d00c0000{
                #cooling-cells = <2>; /* min followed by max */
                compatible = "arm,mali-450";
@@ -223,8 +220,9 @@
                  };

        };
+
        gic: interrupt-controller@c4301000{
-               compatible = "arm,gic-400";
+               compatible = "arm,gic-400", "arm,cortex-a15-gic", "arm,cortex-a9-gic";
                #interrupt-cells = <3>;
                #address-cells = <0>;
                interrupt-controller;
I do not agree using comments to prevent code in the patch files. So, please just remove unnecessary routines on your patch.
The dts file can override and include. So, I think removing the timer attribute in cpu node and moving timer node to each dts files are correct approach. (a.k.a> "meson-timer" -> meson64_odroidc2.dts, "armv8-timer" -> meson64_kvm_odroidc2.dts)

User avatar
Brian.K
Posts: 246
Joined: Tue Sep 15, 2015 7:30 pm
languages_spoken: English, Korean
ODROIDs: XU4, C1+, C2
Location: South Korea
Contact:

Re: KVM

Unread post by Brian.K » Mon May 30, 2016 2:31 pm

@crashoverride, thank you for the your work!
Ubuntu 16.04 Cloud image(unmodified) is working perfectly well now on the official Ubuntu v2 image! :-o
I made the Running Ubuntu 16.04 Cloud image using KVM/ARM wiki page for ODROIDian who has interested in this work.

xufour
Posts: 10
Joined: Wed May 11, 2016 7:24 pm
languages_spoken: english
ODROIDs: xu4, c2
Contact:

Re: KVM

Unread post by xufour » Tue Jun 14, 2016 9:35 pm

Dear Brian Kim,

Thank you very much for your post http://odroid.com/dokuwiki/doku.php?id= ... untu_cloud
It works almost nicely to me until I call qemu-system-aarch64 to instantiate the VM. Two things:
1. There is no cloud.img
2. When I ignore the line with cloud.img, I was stuck at the shell command prompt "UEFI Interactive Shell v2.1". I try several workaround (eg, this one: http://askubuntu.com/questions/566315/v ... tive-shell), however I still stuck there.

Can you please help me solve the issue, or maybe make it clearer about the parameters of your command?

Best regards,

User avatar
Brian.K
Posts: 246
Joined: Tue Sep 15, 2015 7:30 pm
languages_spoken: English, Korean
ODROIDs: XU4, C1+, C2
Location: South Korea
Contact:

Re: KVM

Unread post by Brian.K » Wed Jun 15, 2016 9:53 am

xufour wrote:Dear Brian Kim,

Thank you very much for your post http://odroid.com/dokuwiki/doku.php?id= ... untu_cloud
It works almost nicely to me until I call qemu-system-aarch64 to instantiate the VM. Two things:
1. There is no cloud.img
2. When I ignore the line with cloud.img, I was stuck at the shell command prompt "UEFI Interactive Shell v2.1". I try several workaround (eg, this one: http://askubuntu.com/questions/566315/v ... tive-shell), however I still stuck there.

Can you please help me solve the issue, or maybe make it clearer about the parameters of your command?

Best regards,
cloud.img is generated by cloud-localds tool.

Code: Select all

$ sudo apt install cloud-image-utils
$ cat > cloud.txt <<EOF
#cloud-config
password: odroid
chpasswd: { expire: False }
ssh_pwauth: True
EOF
$ cloud-localds cloud.img cloud.txt 

xufour
Posts: 10
Joined: Wed May 11, 2016 7:24 pm
languages_spoken: english
ODROIDs: xu4, c2
Contact:

Re: KVM

Unread post by xufour » Wed Jun 15, 2016 4:18 pm

Dear Brian and all,

That's correct! I made few small mistakes. But it's all fine now.
It works well by following the steps in Brian's tutorial. Thank you very much for that!
Best regards,

mudroid
Posts: 2
Joined: Sun Sep 11, 2016 9:28 am
languages_spoken: english
Contact:

Re: KVM

Unread post by mudroid » Mon Oct 24, 2016 11:28 am

Hey everyone!

I'm trying to put together a updated tutorial for running virtualized OS on C2 with KVM. I've managed to get KVM running with Ubuntu cloud and a traditional Ubuntu server installation, and there are definitely some specific things to do. (For example, there no arm64 virt-tools support if you're running a Debian host OS on C2.) But I am figuring things out! Brian's tutorial helped a lot, thanks! :D

Now, I'm trying to run Android through KVM. I feel like it should be possible, due to this thread (KVM on XU3) and the November 2014 issue that had pre-built android images.

So before I try to figure this out, does anyone have any tips or success stories booting Android through KVM on C2? Would it be possible to adapt the hardkernel android images for KVM? Thank you!

crashoverride
Posts: 4316
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: KVM

Unread post by crashoverride » Mon Oct 24, 2016 1:19 pm

The issue I ran into before was that there are no AArch64 virtual machine models that include any graphics. Libvirt/virsh also introduces lots of issue. While QEMU/KVM may support things, libvirt/virsh may make it impossible to activate/select/use. For best results, use the QEMU command line directly.
mudroid wrote:Would it be possible to adapt the hardkernel android images for KVM?
There is no emulated Amlogic/Samsung hardware for QEMUs device model (DM).

In the 32bit world, ARM Versatile Express boards are supported by QEMU's DM. This combined with a version of Android for ARM Versatile Express is likely what others are using. The challenge is getting a 64bit ARM reference board emulated with graphics. Its been quite some time since I checked on the status of this so I do not know if any progress was made by QEMU/KVM upstream.

Post Reply

Return to “Other OS”

Who is online

Users browsing this forum: No registered users and 2 guests