Proxmox

Post Reply
User avatar
n2qcn
Posts: 34
Joined: Sat Dec 01, 2018 12:38 pm
languages_spoken: english
ODROIDs: H2 N2
Has thanked: 0
Been thanked: 3 times
Contact:

Proxmox

Unread post by n2qcn » Sat Dec 01, 2018 12:51 pm

Screen Shot 2018-11-30 at 10.41.21 PM.png
Screen Shot 2018-11-30 at 10.41.21 PM.png (205.47 KiB) Viewed 5398 times
yup, https://www.proxmox.com/en/proxmox-ve works but it takes some effort. Its booting from nvme0n1 with ZFS and sees the other sata ports (eMMC untested) and the two ethernets are in bond-mode 802.3ad

so, may I buy two more pls!
Last edited by n2qcn on Sat Dec 01, 2018 11:08 pm, edited 2 times in total.
These users thanked the author n2qcn for the post:
NigelM (Wed Jul 17, 2019 11:24 pm)

User avatar
n2qcn
Posts: 34
Joined: Sat Dec 01, 2018 12:38 pm
languages_spoken: english
ODROIDs: H2 N2
Has thanked: 0
Been thanked: 3 times
Contact:

Re: Proxmox

Unread post by n2qcn » Sat Dec 01, 2018 2:06 pm

The Proxmox install kinda really wants legacy boot. And if you want to boot ZFS on modern hardware (NVMe), they aren't enumerated in legacy bios. So the process used was:

1) On another machine with legacy boot and NVMe, install with ZFS onto a USB stick in legacy mode.
2) zpool attach the NVMe drive
3) format a uEFI partition on NVMe
4) reboot in uEFI mode
4) zpool detach stick and online -e NVMe to get all the space
5) move NVMe to H2
6) downgrade grub2 (the fun part)

Let ZFS partition the NVMe and format the Reserved partition as uEFI

Code: Select all

zpool attach rpool sda2 nvme0n1
apt-get install parted
sgdisk -t9:EF00 -c9:uEFI-Reserved.ZFS /dev/nvme0n1
parted -ls
mkdosfs -n BOOT_EFI /dev/nvme0n1p9
echo /dev/disk/by-label/BOOT_EFI /boot/efi vfat defaults 0 1 >> /etc/fstab
mount /boot/efi
grub-install -d /usr/lib/grub/x86_64-efi /dev/nvme0n1 
grub-install will spit out errors that there are no EFI variable because we booted legacy. Put the http://www.rodsbooks.com/refind/getting.html USB image on a stick and boot it while turning off legacy and on uEFI.

to confirm no EFI errors and remove the need for rEFIND

Code: Select all

grub-install /dev/nvme0n1
zpool detach rpool sda2
zpool online -e rpool nvme0n1
These are not required on the boot machine, but are included here so you can play with EFI boot

Code: Select all

efibootmgr -v # to see the EFI Boot strings
efibootmgr -qBb 1 # to remove the first one
efibootmgr -c -d /dev/nvme0n1 -p 9 -L nvme0n1 -l "\EFI\proxmox\grubx64.efi"   # put a boot string in EFI
efibootmgr -o 0,1 # boot order of 0 then 1
Unfortunately the grub2 used by Proxmox does not work on the H2, so I copied /usr/lib/grub/x86_64-efi/* from the H2 ubuntu install and overwrite it on the Proxmox disk. Then grub-install -d /usr/lib/grub/x86_64-efi complained about "grub-install: error: relocation 0x4 is not implemented yet." so I overwrote /usr/lib/grub/x86_64-efi/kernel.img from the pve install. Then it would install /boot/efi/EFI/proxmox/grubx64.efi and /boot/efi/EFI/proxmox/grub.cfg but the $prefix EFI var still had ubuntu in it. The workaround is to copy everything in the proxmox dir to ubuntu. This is ok as these files don't change and all the kernel references are in /boot/grub/grub.cfg in ZFS

Code: Select all

root@pve:/boot/efi/EFI# pwd
/boot/efi/EFI
root@pve:/boot/efi/EFI# ls -lR
.:
total 8
drwxr-xr-x 2 root root 4096 Nov 30 21:45 proxmox
drwxr-xr-x 2 root root 4096 Nov 30 21:54 ubuntu

./proxmox:
total 1084
-rwxr-xr-x 1 root root     109 Nov 30 21:45 grub.cfg
-rwxr-xr-x 1 root root 1105784 Nov 30 21:45 grubx64.efi

./ubuntu:
total 1084
-rwxr-xr-x 1 root root     109 Nov 30 21:54 grub.cfg
-rwxr-xr-x 1 root root 1105784 Nov 30 21:54 grubx64.efi
root@pve:/boot/efi/EFI# cat ubuntu/grub.cfg 
search.fs_uuid 6bca05803e66869e root 
set prefix=($root)'/ROOT/pve-1@/boot/grub'
configfile $prefix/grub.cfg
root@pve:/boot/efi/EFI# df -h . /
Filesystem        Size  Used Avail Use% Mounted on
/dev/nvme0n1p9    8.0M  2.2M  5.9M  27% /boot/efi
rpool/ROOT/pve-1  444G  1.3G  443G   1% /

chao42
Posts: 1
Joined: Fri Dec 07, 2018 9:15 pm
languages_spoken: english
ODROIDs: ORDOID-H2
Has thanked: 0
Been thanked: 0
Contact:

Re: Proxmox

Unread post by chao42 » Fri Dec 07, 2018 9:18 pm

HI,
I also want to setup Proxmox VE on H2 but I have the Problem that my USB does not boot. I have tried the medium on another PC - there it works.
Only on H2 it only shows me a black screen with the dash in the upper left corner.

So can you please give me the advice how you setup your install medium?
Thank you!

Regards

User avatar
n2qcn
Posts: 34
Joined: Sat Dec 01, 2018 12:38 pm
languages_spoken: english
ODROIDs: H2 N2
Has thanked: 0
Been thanked: 3 times
Contact:

Re: Proxmox

Unread post by n2qcn » Sat Dec 08, 2018 3:49 pm

Well, the right way to distribute this would be a respun install ISO with a H2 compatible grub2, but its not as easy as first thought. (the boot loader directly loads linux26 and initrd.img) While I figure this out, a quick fix would be to torrent

Code: Select all

magnet:?xt=urn:btih:733501933dc5e554948db11fe19af96916abc78e&dn=ProxmoxH2.iso.7z
which is a H2 bootable ZFS boot image of a 16G stick. It's 11.5G compressed, so it will take awhile to download. use dd or etcher to image. The root password is hardkernel

If your seeding Proxmox VE 5.3 ISO

Code: Select all

https://www.proxmox.com/en/downloads?task=callelement&format=raw&item_id=423&element=f85c494b-2b32-4109-b8c1-083cca2b7db6&method=download&args[0]=a8201d0dada6f939cb378435baea0404
it will be easy to find the DHT hash.

User avatar
n2qcn
Posts: 34
Joined: Sat Dec 01, 2018 12:38 pm
languages_spoken: english
ODROIDs: H2 N2
Has thanked: 0
Been thanked: 3 times
Contact:

Re: Proxmox

Unread post by n2qcn » Sun Dec 09, 2018 3:25 pm

http://rob.com/porteus/ProxmoxH2.iso is a 642Meg Proxmox VE 5.3 install image with a H2 compliant grub2. It boots and runs normally, but installs a broken grub2.

A few notes that effect all Proxmox installs unrelated to the grub2 issue:

* If you get a "can't find cdrom" try a higher quality USB stick.
* It won't partition a NVMe disk with ZFS, but does for LVM / ext4
* It makes a (broken) esp/EFI partition for LVM/ext4 but not ZFS

Basic workflow:

* Download iso
* burn it to a USB stick with dd or https://www.balena.io/etcher/
* hit esc to enter H2's setup
* at the bottom of Save&Exit select the stick
* select Install and do your thing then reboot
* esc back into H2's setup and select the stick again
* select Rescue Boot
* if one used ZFS on a slow USB disk (to latter zpool attach NVMe disk) one might need to

Code: Select all

zpool import rpool ; exit
* login and do basic maintenance

Code: Select all

echo stty -ixon >> ~/.profile 
source ~/.profile
echo deb http://download.proxmox.com/debian/pve stretch pve-no-subscription > /etc/apt/sources.list.d/pve-enterprise.list
echo ZFS_INITRD_POST_MODPROBE_SLEEP='2' >> /etc/default/zfs  # Only for slow USB disks
apt-get update
apt-get -y dist-upgrade
apt-get -y install parted
* Identify the install disk and booted disk, for ZFS:

Code: Select all

# zpool status ; parted -ls ; lsblk -o name,size,label,mountpoint,uuid
 
  pool: rpool
 state: ONLINE
  scan: none requested
config:

NAME        STATE     READ WRITE CKSUM
rpool       ONLINE       0     0     0   
  sdb3      ONLINE       0     0     0

errors: No known data errors
Warning: Not all of the space available to /dev/sda appears to be used, you can fix the GPT to use all of the space (an extra 29891004 blocks) or continue with the current setting?
Model: MX MXUB3SESU-16G (scsi)
Disk /dev/sda: 16.0GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name                Flags
 1      32.8kB  276kB   244kB                Gap0                hidden, msftdata
 2      276kB   3226kB  2949kB               EFI boot partition  boot, hidden, esp
 3      3226kB  673MB   669MB   hfs+         HFSPLUS             hidden
 4      673MB   673MB   307kB                Gap1                hidden, msftdata

 
Model: SSD G2 s eries 64GB (scsi)
Disk /dev/sdb: 60.0GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt 
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      17.4kB  1049kB  1031kB                     bios_grub
 2      1049kB  538MB   537MB                      boot, esp
 3      538MB   60.0GB  59.5GB  zfs


NAME     SIZE LABEL MOUNTPOINT UUID
sda     14.9G
├─sda1   238K
├─sda2   2.8M                  2D50-96C4
├─sda3 638.4M PVE
└─sda4   300K
sdb     55.9G rpool            17677783431408352291
├─sdb1  1007K rpool            17677783431408352291
├─sdb2   512M rpool            17677783431408352291
└─sdb3  55.4G rpool            17677783431408352291
or LVM

Code: Select all

Model: MX MXUB3SESU-16G (scsi)
Disk /dev/sda: 16.0GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
  
Number  Start   End     Size    File system  Name                Flags
 1      32.8kB  276kB   244kB                Gap0                hidden, msftdata
 2      276kB   3226kB  2949kB               EFI boot partition  boot, hidden, esp
 3      3226kB  673MB   669MB   hfs+         HFSPLUS             hidden
 4      673MB   673MB   307kB                Gap1                hidden, msftdata


Error: /dev/mapper/pve-data: unrecognised disk label
Model: Linux device-mapper (thin-pool) (dm)
Disk /dev/mapper/pve-data: 364GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Error: /dev/mapper/pve-data_tdata: unrecognised disk label
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/pve-data_tdata: 364GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Error: /dev/mapper/pve-data_tmeta: unrecognised disk label
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/pve-data_tmeta: 3720MB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/pve-swap: 7516MB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system     Flags
 1      0.00B  7516MB  7516MB  linux-swap(v1)


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/pve-root: 103GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End    Size   File system  Flags
 1      0.00B  103GB  103GB  ext4


Model: Unknown (unknown)
Disk /dev/nvme0n1: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      17.4kB  1049kB  1031kB                     bios_grub
 2      1049kB  538MB   537MB   fat32              boot, esp
 3      538MB   500GB   500GB                      lvm


NAME                 SIZE LABEL MOUNTPOINT UUID
sda                 14.9G PVE              2018-12-03-12-15-47-00
├─sda1               238K PVE              2018-12-03-12-15-47-00
├─sda2               2.8M PVE   /mnt       2D50-96C4
├─sda3             638.4M PVE              2018-12-03-12-15-47-00
└─sda4               300K PVE              2018-12-03-12-15-47-00
nvme0n1            465.8G
├─nvme0n1p1         1007K
├─nvme0n1p2          512M       /boot/efi  5A2C-DF73
└─nvme0n1p3        465.3G                  C4PNAc-dAhI-Mvyl-iekI-XKOr-nata-3JycxL
  ├─pve-swap           7G       [SWAP]     c3f5ba52-7e9d-40d8-87df-e9f2fe0752ac
  ├─pve-root          96G       /          12a0f88d-61b6-4485-a927-336b32819fe9
  ├─pve-data_tmeta   3.5G
  │ └─pve-data     339.3G
  └─pve-data_tdata 339.3G
    └─pve-data     339.3G
* for this example the install disk is=sda and the booted disk is=sdb but yours might be swapped.
* for ZFS build a esp/EFI partition

Code: Select all

mkdosfs -n esp /dev/sdb2      
echo /dev/disk/by-label/esp /boot/efi vfat defaults 0 1 >> /etc/fstab
mount /boot/efi
grub-install -d /usr/lib/grub/x86_64-efi /dev/sdb
cd /boot/efi/EFI
mkdir BOOT
* if LVM

Code: Select all

cd /boot/efi/EFI
* mount the install disk and copy its grub2 to the boot disk

Code: Select all

mount /dev/sda2 /mnt
cp /mnt/efi/boot/BOOTx64.EFI BOOT/
cp /mnt/efi/boot/BOOTx64.EFI proxmox/grubx64.efi
mkdir ubuntu
cp /mnt/efi/ubuntu/grub.cfg.zfs ubuntu/grub.cfg
* if you use LVM change the last line to

Code: Select all

cp /mnt/efi/ubuntu/grub.cfg.lvm ubuntu/grub.cfg
* reboot and be happy!

User avatar
rooted
Posts: 6697
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 165 times
Been thanked: 33 times
Contact:

Re: Proxmox

Unread post by rooted » Sun Dec 09, 2018 3:38 pm

I don't use proxmox but I appreciate the detailed information and work you have put into this.

How is the H2 working for your needs on this?

PigLover
Posts: 26
Joined: Wed Nov 15, 2017 3:55 am
languages_spoken: english
ODROIDs: C2, XU4, HC1, MC1
Has thanked: 0
Been thanked: 0
Contact:

Re: Proxmox

Unread post by PigLover » Mon Dec 10, 2018 4:24 am

Nice to see that Proxmox works (almost) out of the box.

Have you done any tests to see if upgrading (“#apt update && apt dist-upgrade -y”) overwrites your Grub replacement and blows it up? Or does it remain stable?

I’m really interested in seeing if Proxmox can boot to eMMC. Ideal for me would be 5 nodes, eMMC boot and NVMe dedicated to Ceph for the cluster. When all is done this is probably not the most cost effective way to go but its a great environment for experimenting with clusters.

Now if HK could get get some love from Intel to re-stock the store...

User avatar
n2qcn
Posts: 34
Joined: Sat Dec 01, 2018 12:38 pm
languages_spoken: english
ODROIDs: H2 N2
Has thanked: 0
Been thanked: 3 times
Contact:

Re: Proxmox

Unread post by n2qcn » Mon Dec 10, 2018 5:39 am

PigLover wrote:Have you done any tests to see if upgrading (“#apt update && apt dist-upgrade -y”) overwrites your Grub replacement and blows it up? Or does it remain stable?
it lives though those and

Code: Select all

update-grub ; grub-install ; update-initramfs -u 
but not

Code: Select all

grub-install --removable
as this hack uses the fallback loader ($prefix/efi/boot/bootx64.efi) to take pri from the proxmox loader. (ie one sets the auto generated uEFI boot string as the first entry in the boot order, before "proxmox")

Its a good bet the eMMC will work, but I don't have one because the local MicroCenter is selling 480G SSDs for 44USD. Faster and cheaper than eMMC. (I'll buy one from HK when I can order more H2s)

Rooted: it will work better when I have a few H2s for the CephFS cluster.

User avatar
n2qcn
Posts: 34
Joined: Sat Dec 01, 2018 12:38 pm
languages_spoken: english
ODROIDs: H2 N2
Has thanked: 0
Been thanked: 3 times
Contact:

Re: Proxmox

Unread post by n2qcn » Thu Jan 03, 2019 3:49 am

Bought an eMMC. /dev/mmcblk0 can be partitioned and booted, including Proxmox and Porteus. But /dev/mmcblk0boot0 and /dev/mmcblk0boot1 are a mystery and it works best to ignore them. From a Porteus install:

Code: Select all

lsblk -o name,size,label,mountpoint,uuid
NAME           SIZE LABEL MOUNTPOINT UUID
loop2         35.1M                  
loop3         43.8M                  
loop4         59.4M                  
loop5          1.1M                  
loop6          1.3M                  
loop7          1.2M                  
loop8          600K                  
loop9          368K                  
mmcblk0       29.1G                  
|-mmcblk0p1     40M Boot             53EC-876B
|-mmcblk0p2    900M Kiosk            2019-01-01-23-15-12-89
`-mmcblk0p4   28.2G                  47f36073-edf5-4610-a8f2-f9b4fd2737b0
mmcblk0boot0     4M                  
mmcblk0boot1     4M                  
zram0          1.4G       [SWAP]     
nvme0n1      465.8G                  
|-nvme0n1p1  465.8G rpool            8305907072045084449
`-nvme0n1p9      8M esp              058D-E5F5
root@HomePorteus:~# 

User avatar
odroid
Site Admin
Posts: 32355
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 145 times
Been thanked: 325 times
Contact:

Re: Proxmox

Unread post by odroid » Thu Jan 03, 2019 9:17 am

Glad to hear we can install Proxmox into eMMC. :D
If you want to write something to the hidden boot blocks in the eMMC, you need to unlock it first probably. We tested it on other ARM boards only.

Code: Select all

sudo su -
# echo 0 > /sys/block/mmcblk0boot0/force_ro 
# dd if=/root/boot0.img of=/dev/mmcblk0boot0
BTW, can you please write an article about Proxmox introduction for our Magazine.
If yes, I will connect you to our chief editor.

xauser
Posts: 50
Joined: Mon Apr 01, 2013 6:49 pm
languages_spoken: english
ODROIDs: U2,U3,XU3
Has thanked: 0
Been thanked: 0
Contact:

Re: Proxmox

Unread post by xauser » Mon Apr 22, 2019 5:50 pm

I tried to install proxmox on h2 today and failed :(

I used the image provided by n2qcn and could boot from usb stick to proxmox installer. (The official image won't boot)
I have no sata drives attached and wanted to install to mmc. The installer rejects to install on 8gb mmc. With a 16gb mmc
I get "proxmox unable to get device for partition". So no luck for me. Official proxmox forum says that installing on mmc/usb
is not supported as there will be a lot of log writes and such media would wear out fast.

Metteus
Posts: 4
Joined: Sun Jul 07, 2019 4:23 pm
languages_spoken: Italian, English
ODROIDs: H2
Has thanked: 0
Been thanked: 0
Contact:

Re: Proxmox

Unread post by Metteus » Sun Jul 07, 2019 4:43 pm

You can install the latest Debian version on H2 (even on eMMC) and then convert it to Proxmox (https://pve.proxmox.com/wiki/Install_Pr ... an_Stretch).
I suppose this is the easiest way to install proxmox for now.

@n2qcn do you also have these errors in the dmesg ?
ucsi_acpi USBC000:00: Failed to reset PPM! Trying again..
ucsi_acpi USBC000:00: failed to reset PPM!
ucsi_acpi USBC000:00: PPM init failed (-110)

@n2qcn which are the advantages to install proxmox on ZFS ?

Thanks

User avatar
n2qcn
Posts: 34
Joined: Sat Dec 01, 2018 12:38 pm
languages_spoken: english
ODROIDs: H2 N2
Has thanked: 0
Been thanked: 3 times
Contact:

Re: Proxmox

Unread post by n2qcn » Tue Jul 09, 2019 7:19 am

> advantages to install proxmox on ZFS ?

The advantages are:

1) replication (every 15mins) of containers between nodes
2) snapshot based vmdumps
3) several boot configs for rollbacks
4) sparse VM disk images.
5) no mounting required to cd;ls around a container's root dir
6) arc cache for fast reads
7) zil for grouping writes
8) lz4 disk compression to speed up spinning rust storage
9) full disk encryption. (made easier with zfs-0.8.0)

off the top of my head, but what's even cooler is any old fs for root and put everything on ceph!

filou59
Posts: 3
Joined: Sat Jan 12, 2019 6:18 am
languages_spoken: english
ODROIDs: HC1/HC2/XU4
Has thanked: 0
Been thanked: 3 times
Contact:

Re: Proxmox

Unread post by filou59 » Wed Jul 17, 2019 1:03 pm

Hello

New Proxmox VE 6.0.4 is out

Seem's to be OK on H2, Install with ZFS on NVME with the official ISO
These users thanked the author filou59 for the post (total 2):
NigelM (Wed Jul 17, 2019 11:24 pm) • powerful owl (Thu Oct 03, 2019 2:48 pm)

User avatar
n2qcn
Posts: 34
Joined: Sat Dec 01, 2018 12:38 pm
languages_spoken: english
ODROIDs: H2 N2
Has thanked: 0
Been thanked: 3 times
Contact:

Re: Proxmox

Unread post by n2qcn » Thu Jul 18, 2019 3:01 am

systemd-boot is the new ESP booting and sync tool. read about it https://pve.proxmox.com/pve-docs/pve-ad ... ml#sysboot

powerful owl
Posts: 100
Joined: Thu Mar 28, 2019 8:57 pm
languages_spoken: english
ODROIDs: 6 x HC1, N2, H2
Has thanked: 19 times
Been thanked: 8 times
Contact:

Re: Proxmox

Unread post by powerful owl » Thu Oct 03, 2019 2:53 pm

filou59 wrote:
Wed Jul 17, 2019 1:03 pm
New Proxmox VE 6.0.4 is out

Seem's to be OK on H2, Install with ZFS on NVME with the official ISO
Hi all, I just wanted to emphasize this in case anyone was put off by the installation requirements listed above for earlier versions, but Proxmox VE 6 installs on H2 with no hiccups or problems, really easy. (i.e. if I can do it anyone can!)

Since I have only one H2, I made a two-node cluster with a NUC, machines just migrate back and forth with no problem, it's pretty awesome :)

If you want more stats/graphs than Proxmox provides, install netdata on the host https://www.netdata.cloud/ (NB use their one-line installer, don't use instructions on the web because that will bork it and you will have to start again, how do I know this...)

jimmycrackernel
Posts: 4
Joined: Thu Oct 10, 2019 7:17 am
languages_spoken: english
ODROIDs: Odroid H2
Has thanked: 0
Been thanked: 2 times
Contact:

Re: Proxmox

Unread post by jimmycrackernel » Thu Oct 10, 2019 9:23 am

Is anybody else having MASSIVE trouble with the r8169 driver? The NICs are so unstable the H2 is effectively unusable.

User avatar
n2qcn
Posts: 34
Joined: Sat Dec 01, 2018 12:38 pm
languages_spoken: english
ODROIDs: H2 N2
Has thanked: 0
Been thanked: 3 times
Contact:

Re: Proxmox

Unread post by n2qcn » Thu Oct 10, 2019 11:16 am

no issues here, mostly ceph traffic bonded between the interfaces. counters for only a day as I rebooted for a new kernel last night:

Code: Select all

root@H2:~#  ip -s link show  enp2s0
2: enp2s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 00:1e:06:45:02:54 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    10150452037 24770424 0       1       0       310341  
    TX: bytes  packets  errors  dropped carrier collsns 
    2457583666 5334739  0       0       0       0       
root@H2:~#  ip -s link show  enp3s0
3: enp3s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 00:1e:06:45:02:54 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    6182540793 8514618  0       1       0       87017   
    TX: bytes  packets  errors  dropped carrier collsns 
    15187716916 26932702 0       0       0       0       
root@H2:~# ethtool -i enp2s0
driver: r8169
version: 
firmware-version: rtl8168g-2_0.0.1 02/06/13
expansion-rom-version: 
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
root@H2:~# ethtool  enp2s0
Settings for enp2s0:
	Supported ports: [ TP MII ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Advertised pause frame use: Symmetric Receive-only
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Link partner advertised link modes:  10baseT/Half 10baseT/Full 
	                                     100baseT/Half 100baseT/Full 
	                                     1000baseT/Full 
	Link partner advertised pause frame use: No
	Link partner advertised auto-negotiation: Yes
	Link partner advertised FEC modes: Not reported
	Speed: 1000Mb/s
	Duplex: Full
	Port: MII
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: on
	Supports Wake-on: pumbg
	Wake-on: g
	Current message level: 0x00000033 (51)
			       drv probe ifdown ifup
	Link detected: yes
root@H2:~# uptime
 22:12:17 up 1 day, 11:34,  1 user,  load average: 0.18, 0.29, 0.25
These users thanked the author n2qcn for the post:
powerful owl (Fri Oct 11, 2019 4:29 pm)

jimmycrackernel
Posts: 4
Joined: Thu Oct 10, 2019 7:17 am
languages_spoken: english
ODROIDs: Odroid H2
Has thanked: 0
Been thanked: 2 times
Contact:

Re: Proxmox

Unread post by jimmycrackernel » Fri Oct 11, 2019 8:43 am

Ah ok. If you're running ceph without a problem then it seems unlikely the NICs are the root of my problem.

Actually, since posting ^^ I've started to suspect a RAM issue. Ordering replacements today (this time utilizing the confirmed modules page). If testing supports my theory, it will be the most subtle and elusive RAM problem I've ever seen. Bizarre.

powerful owl
Posts: 100
Joined: Thu Mar 28, 2019 8:57 pm
languages_spoken: english
ODROIDs: 6 x HC1, N2, H2
Has thanked: 19 times
Been thanked: 8 times
Contact:

Re: Proxmox

Unread post by powerful owl » Fri Oct 11, 2019 4:11 pm

I'm curious what specifically you are seeing that seemed to be the NICs? I was getting a lot of dropped packets on mine, turned out to be an upstream switch (i.e. not the one the H2 connected to but one between that switch and the router).

jimmycrackernel
Posts: 4
Joined: Thu Oct 10, 2019 7:17 am
languages_spoken: english
ODROIDs: Odroid H2
Has thanked: 0
Been thanked: 2 times
Contact:

Re: Proxmox

Unread post by jimmycrackernel » Fri Oct 11, 2019 11:48 pm

I don't dare fire it up again to get you some real logs as its already borked at least one SSD, but the main things I remember seeing were kernel panics, constant FS errors (possibly just never being able to gracefully shutdown, but it seemed worse than that) and interfaces cycling into blocking mode, forwarding, then down.
These users thanked the author jimmycrackernel for the post:
powerful owl (Sat Oct 12, 2019 1:20 pm)

jimmycrackernel
Posts: 4
Joined: Thu Oct 10, 2019 7:17 am
languages_spoken: english
ODROIDs: Odroid H2
Has thanked: 0
Been thanked: 2 times
Contact:

Re: Proxmox

Unread post by jimmycrackernel » Thu Oct 17, 2019 11:53 am

I've been able to work out all the issues I was having:

1. The RAM. I purchase another set, this time from the confirmed modules list. Still, this one is just baffling. Makes me suspect a tweak in the BIOS code could allow my previous modules to work.
2. The problem where Proxmox's current OOB config has interference between NTPD and systemd-timesyncd. (just apt purge ntp)
3. IP interference. I didn't realize I had another device already assigned to an IP on one of my storage subnets. (That's a little embarassing.)

All errors cleared right up. At this point I'm actually really impressed with the little guy!
These users thanked the author jimmycrackernel for the post:
odroid (Thu Oct 17, 2019 1:21 pm)

Post Reply

Return to “Other OS”

Who is online

Users browsing this forum: No registered users and 0 guests