Review of the H2 based on FACTS and sparing no one

Post Reply
domih
Posts: 118
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Review of the H2 based on FACTS and sparing no one

Unread post by domih » Fri Jul 12, 2019 4:36 pm

Dear Hard Kernel,

I finally received the H2, cases and accessories from Ameridroid. You said it could run 32GB of DDR4 at 2400 MT/s... Mmm... well it does. You said it would boot into the BIOS the first time... yeah it did. The USB 3 stick with the installer image of Ubuntu 19.04 started and installation on the 512GB NVMe went flawlessly. A few lspci, lsusb, dmidecode, etc later all that should be there is there. Connecting a 1 TB SSD to SATA #1 followed by parted + mfks + blkid + vi /etc/fstab ended up and running with no issue. Wired network works. Plugged in an Odroid WiFi USB and it worked right away. APT update, web browsing and all these first steps worked out without a peep of something going wrong. Everything worked :-(

The whole exercise in 30~45 minutes.

My soldering iron felft unloved and my multimeter is now sulking: they were not involved.

Dear Hard Kernel, why do you make it so easy? We are supposed to be pioneers riding the leading edge of SBC technology. We are people plugging keyboards in places they have never been before. Our mice are supposed to gnaw at strange nibbles. "Normal" people expect us to be able to solve any problem, implement new IoT things or appliances without documentation, without videos on YouTube, without community support. Yet you persist in providing all of that for hardware that anyway works out of the box. It's supposed to be difficult and challenging, but no, with the H2 it's like unpacking an Apple gadget: step #1, step #2, step #3... oops there is no step three.

We are to supposed to labor and suffer in the wee hours before the board even accept to boot. I remember the Radioactive Pi a few months ago, you had to solder or make your own Dupont cables to just have the right to turn it on. I even dug into the basements of Mouser and DigiKey to find the connectors that would allow one to use the Webcam connector as a USB 2. Meanwhile on Reddit they are still debating why this connector has 5 pins instead of 4. Epic. Then you felt like the guy in the Onion's Sony Piece of S..t video for hours and days before you understand that the best way to corrupt a USB disk is to connect it to the Titanic Pi. This thing landed in my "Just Frack It Refuse Box". I'll give it to someone I don't like and delight into schadenfreude.

I also shovelled deep during nights and days for several weeks to find a Linux that works with the PCIe x4 slot (supposedly generic) of the Jewel Pro 64. All of that to find out that it does indeed work with the SATA card they sell but with other cards your mileage will vary and usually be long and torturous. Finally got the thing running with two drives and then one of the 5v shift levels simply died. That's what I call actually testing the functionality of RAID 1 :-) I know it's a hard judgement on the guys who worked on the Linux Kernel and drivers to make it work with the Ruby Pro 64 but as the French say "Qui aime bien, châtie bien". My Pearl Pro 64 did not land in the "Just Frack It Refuse Box" yet. I just put it on the freeze for 6 more months and I'll see if the software finally works with the hardware, like you know if you issue a sudo shutdown [-r,-h] now you expect the board to reboot or shutdown. The H2 does. The Gemstone Pro 64 does... oops, no it doesn't.

The H2 looks good in the case type II. The little LEDs come nicely though the dark acrylic.

Wishes for the case type II mk2:
(a) make it a little bit taller so that a 92mm x25mm can fit inside the box, not on top it. Even a 92mm x14-15mm probably does not fit inside right now. Anyway, if you want a fan with beautiful LEDs, you have to use a 25mm tall. After all we want to impress the kiddies with colors and lights.
(b) if it could a little bit taller then you could have some optional acrylic fastener/pod/cradle for a 2.5" HDD or SSD on each side. With the disks outside, no issue of heat to get out of the box.
(c) optional acrylic fastener/pod/cradle for a 3.5" hard disk on each side (vertical like a USB toaster, but... hey, still using SATA).

CONCLUSION
I know it is a very harsh review, but it's clear that Hard Kernel has recently released two pretty good SBC solutions, the N2 and the H2 rev B. Congratulations to all the engineers, SQA, design teams and their bosses.

PS #1
My ultimate wish now would be to see Hard Kernel produce an SBC with the AMD v1000 series (especially with the v1605b). The latter with 16 PCIe 3 lanes would allow to do all that the H2 does PLUS having a PCIe 3 x16 slot with (x8 electrical) allowing the use of plenty of PCI cards still reserved to PC. There are already "industrial" mini-ITX cards doing this. I got a DFI GH171 (about $480, no memory, nothing), it works with a Mellanox 40/56Gbe. This allowed me to build a low-energy NAS that serves files at max ~19Gbe with IPoIB (and as client to other servers at ~29Gbe). Don't fall for the R1000 series, not enough PCIe lanes. IMHO, 10GBe is already so passé. The big guys in data centers already use 100Gbe, 200Gbe and pilot 400Gbe. So SBCs should be able to catch up a little bit and do 40/56 Gbe (cheap used network cards are plenty on eBay, my favorite is the MCX354A-FCBT and I don't mind if people start buying it like crazy making its used price rise, I already piled up a stash of these cards, enough for replacing the failing ones for the next decade).

PS #2
I saw references to NVMe to PCIe adapters in other threads. I'm definitely going to try it. Not sure that PCI 2 x4 for a network card expecting PCI 3 x8 will deliver though. But if it works, the big difference with the Trinket Pro 64 would be that with the H2 they are still 2 SATA ports for the disks (no NAS with USB disks...!)

PS #3
I'm now going to connect that beautiful LED power button to the GPIO. Wish me luck!
Last edited by domih on Thu Jul 18, 2019 9:54 pm, edited 2 times in total.
These users thanked the author domih for the post (total 4):
powerful owl (Fri Jul 12, 2019 4:54 pm) • rmkimathi (Sat Jul 13, 2019 11:03 am) • odroid (Mon Jul 15, 2019 9:55 am) • BadPritt (Wed Jul 17, 2019 4:07 am)

powerful owl
Posts: 87
Joined: Thu Mar 28, 2019 8:57 pm
languages_spoken: english
ODROIDs: 6 x HC1, N2, H2
Has thanked: 12 times
Been thanked: 8 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by powerful owl » Fri Jul 12, 2019 5:01 pm

Nice review :D My H2 also set up and working / doing a job, so now I have to buy another one to experiment with, this is a big problem... :? :lol:
These users thanked the author powerful owl for the post:
rmkimathi (Sat Jul 13, 2019 11:03 am)

User avatar
mad_ady
Posts: 6558
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 169 times
Been thanked: 128 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by mad_ady » Fri Jul 12, 2019 7:06 pm

You're comparing apples and pears. Of course an intel board will be better supported than an arm board. The intel platform has had decades to improve/iron out bugs. If you want challenges, stick to non-intel.

These users thanked the author mad_ady for the post:
igorpec (Fri Jul 12, 2019 9:32 pm)

brad
Posts: 965
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 N1 N2 H2 (64 bit ftw)
Location: Australia
Has thanked: 16 times
Been thanked: 32 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by brad » Sun Jul 14, 2019 12:19 pm

On the H2 I tested the actual M.2 to pcie 2.0 adaptor lane speed at 5GT/s unidirectional (the max of the spec) against GPU (CPU to GPU memory xfer). 4 lanes at 20GT/s. From basic calculations I assume this would be around 16Gbps 1 way (and assuming no loss bidirectional 32Gps total)

This makes me think that a single 10G SFP could perform up to max spec, 2x 10G SFP's would be limited to around 8Gbps each unidirectional.

Although cool theoretical numbers for a network interface on H2 I think there would be some shortcomings trying to use it it practice but I could be wrong. Some examples

SAN attached storage via a 10G SFP - Example database running on Celeron would struggle to be able to do anything useful with the bandwidth (batch processing for example reading, processing on cpu, writing back to the SAN). Even on a large enterprise grade SAN with 20 core xeon processors this can be a challenge at times in practice.

Public network interface on a 10G SFP running as fileserver with the 2x SATA connected to disk - I think would need a very fast SATA / usb connected SSD's to get remotely close to a 10G SFP cards throughput on PCIe port and the celeron would be a bottleneck in performing any conversion to a file protocol (such as NFS, samba, rsync, etc). Put in software raid on the sata ports and more power again from the cpu's is needed.

Your results on the AMD v1608b are very interesting at ~29Gbe with IPoIB :) I would love to see you connect up a Mellanox 40/56Gbe (or maybe something smaller PCIe 2.0 interfaced device) to the H2 to see how it performs at 4x PCIe 2.0. Most of these cards are defined at 8x PCIe could they run on a 4x bus? ) What types of drives are you running and how many usb connected ones on the AMD v1608b ? How Much does it work out the CPU's? I would be very interested to see.

I'm not sure if hardkernel would try to market for server based market but maybe something they could consider in the future using more expensive processors and hardware.

domih
Posts: 118
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by domih » Sun Jul 14, 2019 1:21 pm

@powerful owl: Damn, I told myself the same thing!

@mad_ady:

<<...You're comparing apples and pears. Of course an intel board will be better supported than an arm board. The intel platform has had decades to improve/iron out bugs. If you want challenges, stick to non-intel...>>

Dear mad_ady,

I believe you are reading my post for what it is not. First, it is indeed tongue in cheek second degree (attempt to) humor, being very positive to Hard Kernel while trying to be funny.

It is NOT about Intel vs ARM, I could have written the same thing about the XU4, HC2 and N2 I own, the latter being mentioned in the conclusion. My NAS and backup system has been running on 3 x HC2 for 6+ months now, they are not Intel, they are ARM. Also note that one of the boards I joyfully joke about uses an Atom, that's Intel, not ARM.

The meaning of my post was rather about expressing real satisfaction about the quality of the products and support from Hard Kernel compared to others SBC OEMs based on my first hand experience. So far, the only competitor to Hard Kernel for SBCs (obviously restricted to the ones I bought products from) which I found producing (a) good product(s) and being supportive is Raxda with the Rock Pi 4. In both cases the product does what it is said it can do, sometimes beyond expectations, and when I reached a problem I could not solve myself I found fixes and workarounds in a timely manner with support from both the OEM and the community.

With the others, I concluded it was rather a borderline joke.

I believe it is an important point, if a newbie buys an ARM board and sees that functionality and support are "kinda optional", the newbie next board will bear an Intel or AMD, not an ARM.

IMHO the "compare Intel boards to ARM boards" being misled or misleading is a moot point as soon as an ARM board can do similar things to x86 boards. The ARM applications occupy a very broad territory. Beyond electronics thingies, IoT and appliances of all manner and flavors, if an ARM enables you to run a full Linux OS with all the services you can think of, you are on x86 territory. There is a clear overlap in usage... and users. So yes, there are plenty of SBC users who are not C++, kernel or driver gurus.

One cannot have it both ways. If an OEM produces an ARM board with a PCIe slot, it better fully implements the PCIe standard correctly and supports it in timely manner. Not doing so is negatively affecting the reputation of the ARM environment as a whole basically demonstrating that ARM CPUs cannot handle what an x86 CPU can. If an OEM were to say you can use it as desktop or use PCIe cards and then takes refuge in the fact that it is an ARM, not an x86 it would then become sweet fodder Intel Sales representatives for joking about ARM as a gadget (which it is not.)

Best,

Domi
These users thanked the author domih for the post:
rmkimathi (Mon Jul 15, 2019 4:19 am)

domih
Posts: 118
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by domih » Sun Jul 14, 2019 1:47 pm

@brad

I ordered the CHIPAL Mini PCI-E (strike that) the M.2 to PCI-E 16X Riser mentioned in an other thread. The one said to be working with a Quadro P400. I expect to get it in a few weeks.

I can test two different cards:
1. Dual VPI Connect-X3 (the MCX354A-FCBT) which supports 40/56Gbe IB or can be configured as Ethernet cards. Mellanox cards support both IB and Ethernet.
2. I also have two Mellanox 10Gbe Ethernet cards (which I'm not currently using).

I'm not expecting miracles. The PCIe 2 x4 is going to downgrade the speed a lot. Several months ago, I tested the MCX354A-FCBT on old PCs with PCIe 2 x8 (plus slow DDR2) and the best I obtained was 12Gbe IPoIB. Example (shorten for clarity):

Code: Select all

[root@issuer ~]# iblinkinfo
CA: r5-2600 HCA-1:
      0x0002c90300a0b8d1      4    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       2    3[  ] "Infiniscale-IV Mellanox Technologies" ( )
CA: simulator HCA-1:
      0xf45214030087c6f1      5    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       2    7[  ] "Infiniscale-IV Mellanox Technologies" ( )
Switch: 0x0002c902004a3e78 Infiniscale-IV Mellanox Technologies:
.../...
           2    3[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       4    1[  ] "r5-2600 HCA-1" ( )
.../...
           2    5[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       3    1[  ] "issuer HCA-1" ( )
.../...
           2    7[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       5    1[  ] "simulator HCA-1" ( )
.../...
CA: issuer HCA-1:
      0x0010e00001675f69      3    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       2    5[  ] "Infiniscale-IV Mellanox Technologies" ( )
(1) iperf3 test over the regular 1Gbe Ethernet NICs (scroll to see the summary):

Code: Select all

[root@issuer ~]# iperf3 -c 192.168.1.58 --bind 192.168.1.57
Connecting to host 192.168.1.58, port 5201
[  4] local 192.168.1.57 port 60268 connected to 192.168.1.58 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   113 MBytes   947 Mbits/sec    0    211 KBytes
[  4]   1.00-2.00   sec   112 MBytes   943 Mbits/sec    0    211 KBytes
[  4]   2.00-3.00   sec   112 MBytes   941 Mbits/sec    0    218 KBytes
[  4]   3.00-4.00   sec   112 MBytes   941 Mbits/sec    0    218 KBytes
[  4]   4.00-5.00   sec   112 MBytes   942 Mbits/sec    0    218 KBytes
[  4]   5.00-6.00   sec   112 MBytes   943 Mbits/sec    0    218 KBytes
[  4]   6.00-7.00   sec   112 MBytes   942 Mbits/sec    0    218 KBytes
[  4]   7.00-8.00   sec   112 MBytes   940 Mbits/sec    0    218 KBytes
[  4]   8.00-9.00   sec   112 MBytes   943 Mbits/sec    0    218 KBytes
[  4]   9.00-10.00  sec   112 MBytes   940 Mbits/sec    0    218 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec    0             sender
[  4]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec                  receiver
(2) iperf3 test over the Infiniband 40Gbe NICs:

Code: Select all

[root@issuer ~]# iperf3 -c 10.10.10.58 --bind 10.10.10.57
Connecting to host 10.10.10.58, port 5201
[  4] local 10.10.10.57 port 37113 connected to 10.10.10.58 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.09 GBytes  9.32 Gbits/sec    0   1.25 MBytes
[  4]   1.00-2.00   sec  1.40 GBytes  12.1 Gbits/sec    0   1.25 MBytes
[  4]   2.00-3.00   sec  1.54 GBytes  13.3 Gbits/sec    0   1.25 MBytes
[  4]   3.00-4.00   sec  1.45 GBytes  12.4 Gbits/sec    0   1.37 MBytes
[  4]   4.00-5.00   sec  1.48 GBytes  12.7 Gbits/sec    0   1.37 MBytes
[  4]   5.00-6.00   sec  1.54 GBytes  13.2 Gbits/sec    0   1.37 MBytes
[  4]   6.00-7.00   sec  1.48 GBytes  12.7 Gbits/sec    0   1.62 MBytes
[  4]   7.00-8.00   sec  1.54 GBytes  13.2 Gbits/sec    0   1.62 MBytes
[  4]   8.00-9.00   sec  1.54 GBytes  13.2 Gbits/sec    0   1.62 MBytes
[  4]   9.00-10.00  sec  1.53 GBytes  13.2 Gbits/sec    0   1.62 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  14.6 GBytes  12.5 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  14.6 GBytes  12.5 Gbits/sec                  receiver
I dare to expect a little better on the H2 because while it's x4 instead of x8, the H2 has a faster CPU and memory compared to the old PCs.

With modern machines (Good average CPU, fast DDR4 3200, PCI 3 x8) the results are much more interesting:

Code: Select all

root@i5-a:~# iperf3 -c 10.10.10.21 --bind 10.10.10.20
Connecting to host 10.10.10.21, port 5201
[  4] local 10.10.10.20 port 46839 connected to 10.10.10.21 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  3.68 GBytes  31.6 Gbits/sec    0   3.18 MBytes      
[  4]   1.00-2.00   sec  3.68 GBytes  31.6 Gbits/sec    0   3.18 MBytes      
[  4]   2.00-3.00   sec  3.68 GBytes  31.6 Gbits/sec    0   3.18 MBytes      
[  4]   3.00-4.00   sec  3.68 GBytes  31.6 Gbits/sec    0   3.18 MBytes      
[  4]   4.00-5.00   sec  3.68 GBytes  31.6 Gbits/sec    0   3.18 MBytes      
[  4]   5.00-6.00   sec  3.68 GBytes  31.6 Gbits/sec    0   3.18 MBytes      
[  4]   6.00-7.00   sec  3.68 GBytes  31.6 Gbits/sec    0   3.18 MBytes      
[  4]   7.00-8.00   sec  3.68 GBytes  31.6 Gbits/sec    0   3.18 MBytes      
[  4]   8.00-9.00   sec  3.68 GBytes  31.6 Gbits/sec    0   3.18 MBytes      
[  4]   9.00-10.00  sec  3.68 GBytes  31.6 Gbits/sec    0   3.18 MBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  36.8 GBytes  31.6 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  36.8 GBytes  31.6 Gbits/sec                  receiver
With the V1605b it falls to ~19 (as server) and ~29 (as client) because the CPU is slower, DDR4 is at 2400MT/s. For pure IB connect it would be better because the CPU is not involved (network does not go through the full IP stack).
Last edited by domih on Thu Jul 18, 2019 9:55 pm, edited 3 times in total.

domih
Posts: 118
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by domih » Sun Jul 14, 2019 1:56 pm

@brad

Since then, I found a full FDR10 (56Gbe) switch on eBay for $150. I have not tried it yet, busy at work and setting up other SBCs.

You can find the specifications of the V1605B board I use attached. See also attached picture of the board installed. On the right two WD Red 10TB in RAID1, on the left the Mellanox card on which I screwed a fan because these cards could cook an egg (less hot should run more years, yeah?)
Attachments
20190713_215253.jpg
20190713_215253.jpg (601.38 KiB) Viewed 1250 times
DFI-GH171-Mini-ITX-DataSheet.pdf
(2.7 MiB) Downloaded 16 times
Last edited by domih on Thu Jul 18, 2019 9:55 pm, edited 1 time in total.

domih
Posts: 118
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by domih » Sun Jul 14, 2019 2:14 pm

@brad

Yes the Mellanox cards will work on PCI 2 x4 (in theory), it just degrade the speed more and more. Anyway, once I receive the PCIe riser it will take a minute to use lspci for checking whether or not the card is up and grab the LnkCap and LnkSta. It would be funny if it works with the H2 because I did spend quite some time trying with the Stone Pro 64: in early versions of the OS, the Mellanox 10Gbe showed up but did not work, in the latter versions that made their SATA card usable, the network card stopped to show up at all with tons of cryptic errors beyond my skills.

If it works: 2 HGST He10 in Raid 1 via SATA should provide enough oomph I believe. Although I'll start with WD Red 10tb for a start.

Hopefully I answered your questions. If not, ping me there.

Attached: a screenshot that made me quite happy, copying a 4GB file from Windows 10 to a Linux Samba server at about 2GB/s, 20 times the Gigabit speed. Once you tasted it, you're hooked :-)
Attachments
Screen Shot 06-02-19 at 08.54 PM.PNG
Screen Shot 06-02-19 at 08.54 PM.PNG (7.76 KiB) Viewed 1247 times
These users thanked the author domih for the post:
rmkimathi (Mon Jul 15, 2019 4:16 am)

brad
Posts: 965
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 N1 N2 H2 (64 bit ftw)
Location: Australia
Has thanked: 16 times
Been thanked: 32 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by brad » Sun Jul 14, 2019 4:48 pm

Very interested to see your results when you get the PCIE riser :)

back2future
Posts: 237
Joined: Sun Jul 23, 2017 3:19 pm
languages_spoken: english
Has thanked: 9 times
Been thanked: 2 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by back2future » Sun Jul 14, 2019 9:50 pm

Being honest, while interested, Your multimeter should be getting a current meter for a while with network highspeed? (Thx)

domih
Posts: 118
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by domih » Mon Jul 15, 2019 9:35 am

back2future wrote:
Sun Jul 14, 2019 9:50 pm
Being honest, while interested, Your multimeter should be getting a current meter for a while with network highspeed? (Thx)
Yep, it's going to hurt on that side. J4105's 10W + Mellanox' 11.5W + Disk + Memory + Riser + ... = ~ 30W?

I'll use a wall power meter.

domih
Posts: 118
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by domih » Mon Jul 15, 2019 1:01 pm

@brad

I reran the iperf3 with the V1605b because my memory was telling me something was wrong in what I wrote.
The ~29Gbe is when it acts as client communicating with a fast machine(i5-9600K). As server it drops to ~19Gbe with the same fast machine as client.
So it sobers even more the expectation with the H2 acting as server. We'll see.
I fixed my original posts.
I apologize for the SNAFU on the numbers!

Details (Gigabit and IPoIB)

v16-server-i5-client

Code: Select all

domih@i5-a:~$ iperf3 -c 192.168.1.22 --bind 192.168.1.20
Connecting to host 192.168.1.22, port 5201
[  4] local 192.168.1.20 port 48997 connected to 192.168.1.22 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   113 MBytes   950 Mbits/sec    0    358 KBytes       
[  4]   1.00-2.00   sec   111 MBytes   934 Mbits/sec    0    358 KBytes       
[  4]   2.00-3.00   sec   111 MBytes   935 Mbits/sec    0    358 KBytes       
[  4]   3.00-4.00   sec   111 MBytes   934 Mbits/sec    0    358 KBytes       
[  4]   4.00-5.00   sec   111 MBytes   934 Mbits/sec    0    358 KBytes       
[  4]   5.00-6.00   sec   111 MBytes   935 Mbits/sec    0    358 KBytes       
[  4]   6.00-7.00   sec   111 MBytes   935 Mbits/sec    0    358 KBytes       
[  4]   7.00-8.00   sec   111 MBytes   934 Mbits/sec    0    358 KBytes       
[  4]   8.00-9.00   sec   111 MBytes   935 Mbits/sec    0    358 KBytes       
[  4]   9.00-10.00  sec   111 MBytes   934 Mbits/sec    0    358 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.09 GBytes   936 Mbits/sec    0             sender
[  4]   0.00-10.00  sec  1.09 GBytes   934 Mbits/sec                  receiver

domih@i5-a:~$ iperf3 -c 10.10.10.22 --bind 10.10.10.20
Connecting to host 10.10.10.22, port 5201
[  4] local 10.10.10.20 port 36215 connected to 10.10.10.22 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  2.16 GBytes  18.5 Gbits/sec    0   3.18 MBytes       
[  4]   1.00-2.00   sec  2.30 GBytes  19.8 Gbits/sec    0   3.18 MBytes       
[  4]   2.00-3.00   sec  2.26 GBytes  19.4 Gbits/sec    0   3.18 MBytes       
[  4]   3.00-4.00   sec  2.19 GBytes  18.8 Gbits/sec    0   3.18 MBytes       
[  4]   4.00-5.00   sec  2.39 GBytes  20.5 Gbits/sec    0   3.18 MBytes       
[  4]   5.00-6.00   sec  2.33 GBytes  20.0 Gbits/sec    0   3.18 MBytes       
[  4]   6.00-7.00   sec  2.28 GBytes  19.6 Gbits/sec    0   3.18 MBytes       
[  4]   7.00-8.00   sec  2.29 GBytes  19.6 Gbits/sec    0   3.18 MBytes       
[  4]   8.00-9.00   sec  2.28 GBytes  19.6 Gbits/sec    0   3.18 MBytes       
[  4]   9.00-10.00  sec  2.29 GBytes  19.6 Gbits/sec    0   3.18 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  22.8 GBytes  19.5 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  22.8 GBytes  19.5 Gbits/sec                  receiver
v16-client-i5-server

Code: Select all

domih@v16:~$ iperf3 -c 192.168.1.20 --bind 192.168.1.22
Connecting to host 192.168.1.20, port 5201
[  4] local 192.168.1.22 port 43071 connected to 192.168.1.20 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   114 MBytes   954 Mbits/sec    0    273 KBytes       
[  4]   1.00-2.00   sec   112 MBytes   942 Mbits/sec    0    290 KBytes       
[  4]   2.00-3.00   sec   112 MBytes   942 Mbits/sec    0    290 KBytes       
[  4]   3.00-4.00   sec   112 MBytes   941 Mbits/sec    0    290 KBytes       
[  4]   4.00-5.00   sec   112 MBytes   941 Mbits/sec    0    290 KBytes       
[  4]   5.00-6.00   sec   112 MBytes   941 Mbits/sec    0    290 KBytes       
[  4]   6.00-7.00   sec   112 MBytes   941 Mbits/sec    0    290 KBytes       
[  4]   7.00-8.00   sec   112 MBytes   941 Mbits/sec    0    290 KBytes       
[  4]   8.00-9.00   sec   112 MBytes   942 Mbits/sec    0    290 KBytes       
[  4]   9.00-10.00  sec   112 MBytes   941 Mbits/sec    0    290 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec    0             sender
[  4]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec                  receiver

iperf Done.
domih@v16:~$ iperf3 -c 10.10.10.20 --bind 10.10.10.22
Connecting to host 10.10.10.20, port 5201
[  4] local 10.10.10.22 port 57889 connected to 10.10.10.20 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  3.42 GBytes  29.4 Gbits/sec    0   3.25 MBytes       
[  4]   1.00-2.00   sec  3.47 GBytes  29.8 Gbits/sec    0   3.25 MBytes       
[  4]   2.00-3.00   sec  3.48 GBytes  29.9 Gbits/sec    0   3.25 MBytes       
[  4]   3.00-4.00   sec  3.48 GBytes  29.9 Gbits/sec    0   3.25 MBytes       
[  4]   4.00-5.00   sec  3.40 GBytes  29.2 Gbits/sec    0   3.25 MBytes       
[  4]   5.00-6.00   sec  3.46 GBytes  29.7 Gbits/sec    0   3.25 MBytes       
[  4]   6.00-7.00   sec  3.46 GBytes  29.7 Gbits/sec    0   3.25 MBytes       
[  4]   7.00-8.00   sec  3.46 GBytes  29.7 Gbits/sec    0   3.25 MBytes       
[  4]   8.00-9.00   sec  3.45 GBytes  29.7 Gbits/sec    0   3.25 MBytes       
[  4]   9.00-10.00  sec  3.44 GBytes  29.6 Gbits/sec    0   3.25 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  34.5 GBytes  29.7 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  34.5 GBytes  29.7 Gbits/sec                  receiver
Last edited by domih on Thu Jul 18, 2019 9:56 pm, edited 1 time in total.

domih
Posts: 118
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by domih » Mon Jul 15, 2019 11:24 pm

@brad

Here are the results using IPoIB 56Gb moving us again above 20Gbe with the v16: 22Gbe as server, 36.5 as client.

I believe we could do better with "pure" Ethernet. The issue is that it is pretty easy to find a used 40/56Gbs InfiniBand switch at very low price on eBay. The used 40Gbe Ethernet switches on the other hand are obscenely expensive. I guess data centers keep their Ethernet switches much longer :-(

v16-server-i5-client

Code: Select all

domih@i5-a:~$ iperf3 -c 10.10.10.22 --bind 10.10.10.20
Connecting to host 10.10.10.22, port 5201
[  4] local 10.10.10.20 port 40509 connected to 10.10.10.22 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  2.61 GBytes  22.4 Gbits/sec    0   3.18 MBytes       
[  4]   1.00-2.00   sec  2.47 GBytes  21.2 Gbits/sec    0   3.18 MBytes       
[  4]   2.00-3.00   sec  2.68 GBytes  23.0 Gbits/sec    0   3.18 MBytes       
[  4]   3.00-4.00   sec  2.52 GBytes  21.7 Gbits/sec    0   3.18 MBytes       
[  4]   4.00-5.00   sec  2.42 GBytes  20.8 Gbits/sec    0   3.18 MBytes       
[  4]   5.00-6.00   sec  2.55 GBytes  21.9 Gbits/sec    0   3.18 MBytes       
[  4]   6.00-7.00   sec  2.39 GBytes  20.5 Gbits/sec    0   3.18 MBytes       
[  4]   7.00-8.00   sec  2.38 GBytes  20.5 Gbits/sec    0   3.18 MBytes       
[  4]   8.00-9.00   sec  2.45 GBytes  21.1 Gbits/sec    0   3.18 MBytes       
[  4]   9.00-10.00  sec  3.11 GBytes  26.7 Gbits/sec    0   3.18 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  25.6 GBytes  22.0 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  25.6 GBytes  22.0 Gbits/sec                  receiver
v16-client-i5-server

Code: Select all

domih@v16:~$ iperf3 -c 10.10.10.20 --bind 10.10.10.22
Connecting to host 10.10.10.20, port 5201
[  4] local 10.10.10.22 port 58265 connected to 10.10.10.20 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  4.16 GBytes  35.7 Gbits/sec    0   3.06 MBytes       
[  4]   1.00-2.00   sec  4.25 GBytes  36.5 Gbits/sec    0   3.06 MBytes       
[  4]   2.00-3.00   sec  4.25 GBytes  36.5 Gbits/sec    0   3.06 MBytes       
[  4]   3.00-4.00   sec  4.29 GBytes  36.8 Gbits/sec    0   3.06 MBytes       
[  4]   4.00-5.00   sec  4.32 GBytes  37.1 Gbits/sec    0   3.06 MBytes       
[  4]   5.00-6.00   sec  4.24 GBytes  36.4 Gbits/sec    0   3.06 MBytes       
[  4]   6.00-7.00   sec  4.33 GBytes  37.2 Gbits/sec    0   3.06 MBytes       
[  4]   7.00-8.00   sec  4.17 GBytes  35.9 Gbits/sec    0   3.06 MBytes       
[  4]   8.00-9.00   sec  4.28 GBytes  36.8 Gbits/sec    0   3.06 MBytes       
[  4]   9.00-10.00  sec  4.15 GBytes  35.7 Gbits/sec    0   3.06 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  42.4 GBytes  36.5 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  42.4 GBytes  36.5 Gbits/sec                  receiver

User avatar
mad_ady
Posts: 6558
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 169 times
Been thanked: 128 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by mad_ady » Tue Jul 16, 2019 2:25 am

The reduced performance may be due to high interrupts on a single cpu core. You may want to look into distribuing interrupts on all cores (cat /proc/interrupts).
I'm not familiar with infiniband (is it mostly found in the datacenter?), but there might be ways to enable interrupt coalescing for ethernet/infiniband, to reduce interrupts
These users thanked the author mad_ady for the post:
domih (Wed Jul 17, 2019 12:17 pm)

back2future
Posts: 237
Joined: Sun Jul 23, 2017 3:19 pm
languages_spoken: english
Has thanked: 9 times
Been thanked: 2 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by back2future » Tue Jul 16, 2019 4:50 am

Yep, it's going to hurt on that side. J4105's 10W + Mellanox' 11.5W + Disk + Memory + Riser + ... = ~ 30W?
I'll use a wall power meter.
10GbE Mellanox Connectx-2 was ~4Watts (specification: ~3.5Watts for 1 port) additional idling power.
11.5W/2GBps gets a 0.72W for 1Gbps, that's within usual power value for a 1Gb phy (Lantiq to Intel ... ~400mW-750mW).

domih
Posts: 118
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by domih » Wed Jul 17, 2019 12:28 pm

mad_ady wrote:
Tue Jul 16, 2019 2:25 am
The reduced performance may be due to high interrupts on a single cpu core. You may want to look into distribuing interrupts on all cores (cat /proc/interrupts).
I'm not familiar with infiniband (is it mostly found in the datacenter?), but there might be ways to enable interrupt coalescing for ethernet/infiniband, to reduce interrupts
TYVM for your post, it made me look at information I hadn't read yet (although I downloaded it months ago...)

See below the cat /proc/interrupts after I ran a few minutes of iperf3 testing. The IB ConnectX-3 card is handled by the mlx4 driver (scroll to the middle). I also attached v16-cat-proc-interrupts.txt for an easier reading. The bandwidth varied from ~18.5 to ~20.1 in multiple tests. I also attached 3 screenshots from different times during the testing showing htop and the Mate System Monitor: the 4 cores/8 threads don't even go over 50% (with just a few peaks at 60%) *&^%$#@!

In the meantime, I'll go through https://community.mellanox.com/s/articl ... x-adapters.

Code: Select all

domih@v16:~$ cat /proc/interrupts
            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
   0:         39          0          0          0          0          0          0          0  IR-IO-APIC    2-edge      timer
   3:          0          0          0          0          1          0          0          0  IR-IO-APIC    3-edge    
   4:          0          0          0          0          1          0          0          0  IR-IO-APIC    4-edge    
   7:          0          0          0          0          0          0          0          0  IR-IO-APIC    7-fasteoi   pinctrl_amd
   8:          0          0          1          0          0          0          0          0  IR-IO-APIC    8-edge      rtc0
   9:          0          0          0          0          0          0          0          0  IR-IO-APIC    9-fasteoi   acpi
  10:          0          0          0          0          2          0          0          0  IR-IO-APIC   10-edge    
  11:          0          0          0          0          2          0          0          0  IR-IO-APIC   11-edge    
  25:          0          0          0          0          0          0          0          0   PCI-MSI 4096-edge      AMD-Vi
  26:         62          0          0          0          0          0          0          0  IR-PCI-MSI 18432-edge      aerdrv, PCIe PME
  27:          0          0          0          0          0          0          0          0  IR-PCI-MSI 20480-edge      aerdrv, PCIe PME
  28:          0          0          0          0          0          0          0          0  IR-PCI-MSI 22528-edge      aerdrv, PCIe PME
  29:          0          0          0          0          0          0          0          0  IR-PCI-MSI 24576-edge      aerdrv, PCIe PME
  30:          0          0          0          0          0          0          0          0  IR-PCI-MSI 133120-edge      PCIe PME
  31:          0          0          0          0          0          0          0          0  IR-PCI-MSI 135168-edge      PCIe PME
  32:          0          0          0          0          0          0          0          0  IR-IO-APIC   31-fasteoi   0000:05:00.7
  33:      18234          0         60          0          0          0          0          0  IR-PCI-MSI 2627584-edge      xhci_hcd
  34:          0          0          0          0          0          0          0          0  IR-PCI-MSI 2627585-edge      xhci_hcd
  35:          0          0          0          0          0          0          0          0  IR-PCI-MSI 2627586-edge      xhci_hcd
  36:          0          0          0          0          0          0          0          0  IR-PCI-MSI 2627587-edge      xhci_hcd
  37:          0          0          0          0          0          0          0          0  IR-PCI-MSI 2627588-edge      xhci_hcd
  38:          0          0          0          0          0          0          0          0  IR-PCI-MSI 2627589-edge      xhci_hcd
  39:          0          0          0          0          0          0          0          0  IR-PCI-MSI 2627590-edge      xhci_hcd
  40:          0          0          0          0          0          0          0          0  IR-PCI-MSI 2627591-edge      xhci_hcd
  42:          0          0         37          0          0          0          0          0  IR-PCI-MSI 2629632-edge      xhci_hcd
  43:          0          0          0          0          0          0          0          0  IR-PCI-MSI 2629633-edge      xhci_hcd
  44:          0          0          0          0          0          0          0          0  IR-PCI-MSI 2629634-edge      xhci_hcd
  45:          0          0          0          0          0          0          0          0  IR-PCI-MSI 2629635-edge      xhci_hcd
  46:          0          0          0          0          0          0          0          0  IR-PCI-MSI 2629636-edge      xhci_hcd
  47:          0          0          0          0          0          0          0          0  IR-PCI-MSI 2629637-edge      xhci_hcd
  48:          0          0          0          0          0          0          0          0  IR-PCI-MSI 2629638-edge      xhci_hcd
  49:          0          0          0          0          0          0          0          0  IR-PCI-MSI 2629639-edge      xhci_hcd
  52:          0          0          0        226          0          0         18          0  IR-PCI-MSI 3145728-edge      ahci0
  53:          0          0          0          0        295          0          0          0  IR-PCI-MSI 3145729-edge      ahci1
  55:         37          0          0          0          0          0          0          0  IR-PCI-MSI 1572864-edge      nvme0q0
  57:          0          0        548          0          0      27635          0        722  IR-PCI-MSI 1048576-edge      enp2s0
  59:          0          0          0          0          0          0          0          0  IR-PCI-MSI 2097152-edge      enp4s0
  60:          0     212887      50175          0          0          0        709          0  IR-PCI-MSI 2621440-edge      amdgpu
  61:      13197          0          0          0          0          0          0          0  IR-PCI-MSI 1572865-edge      nvme0q1
  62:          0       2063          0          0          0          0          0          0  IR-PCI-MSI 1572866-edge      nvme0q2
  63:          0          0       6221          0          0          0          0          0  IR-PCI-MSI 1572867-edge      nvme0q3
  64:          0          0          0       3171          0          0          0          0  IR-PCI-MSI 1572868-edge      nvme0q4
  65:          0          0          0          0       5793          0          0          0  IR-PCI-MSI 1572869-edge      nvme0q5
  66:          0          0          0          0          0       5261          0          0  IR-PCI-MSI 1572870-edge      nvme0q6
  67:          0          0          0          0          0          0      19199          0  IR-PCI-MSI 1572871-edge      nvme0q7
  68:          0          0          0          0          0          0          0       3999  IR-PCI-MSI 1572872-edge      nvme0q8
  69:          0          0          0          0          0          0          0          0  IR-PCI-MSI 1572873-edge      nvme0q9
  70:          0          0          0          0          0          0          0          0  IR-PCI-MSI 1572874-edge      nvme0q10
  71:          0          0          0          0          0          0          0          0  IR-PCI-MSI 1572875-edge      nvme0q11
  72:          0          0          0          0          0          0          0          0  IR-PCI-MSI 1572876-edge      nvme0q12
  73:          0          0          0          0          0          0          0          0  IR-PCI-MSI 1572877-edge      nvme0q13
  74:          0          0          0          0          0          0          0          0  IR-PCI-MSI 1572878-edge      nvme0q14
  75:          0          0          0          0          0          0          0          0  IR-PCI-MSI 1572879-edge      nvme0q15
  76:          0          0          0       3995          0          0          0       1030  IR-PCI-MSI 524288-edge      mlx4-async@pci:0000:01:00.0
  77:          0          0          0          0          0        200          0    1045974  IR-PCI-MSI 524289-edge      mlx4_0-0
  78:          0          0          0          0          0          0     278607          0  IR-PCI-MSI 524290-edge      mlx4_0-1
  79:          0    4142015          0          0          0          0          0          0  IR-PCI-MSI 524291-edge      mlx4_0-2
  80:          0          0          0          0         11          0          0          0  IR-PCI-MSI 524292-edge      mlx4_0-3
  81:          0          0    1011151          0          0          0          0          0  IR-PCI-MSI 524293-edge      mlx4_0-4
  82:          1          0          0          0          0          0          0          0  IR-PCI-MSI 524294-edge      mlx4_0-5
  83:          0          0          0          0          0          3          0          6  IR-PCI-MSI 524295-edge      mlx4_0-6
  84:          0          0          0          0          0    1797813          0          0  IR-PCI-MSI 524296-edge      mlx4_0-7
  85:          0          0          0          0          0          0          0         41  IR-PCI-MSI 524297-edge      mlx4_0-8
  86:          0          0          0          0          0          0          0          0  IR-PCI-MSI 524298-edge      mlx4_0-9
  87:          0          0          0          0          0          0          0          0  IR-PCI-MSI 524299-edge      mlx4_0-10
  88:          0          0          0          0          0          0          0          0  IR-PCI-MSI 524300-edge      mlx4_0-11
  89:          0          0          0          0          0          0          0          0  IR-PCI-MSI 524301-edge      mlx4_0-12
  90:          0          0          0          0          0          0          0          0  IR-PCI-MSI 524302-edge      mlx4_0-13
  91:          0          0          0          0          0          0          0          0  IR-PCI-MSI 524303-edge      mlx4_0-14
  92:          0          0          0          0          0          0          0          0  IR-PCI-MSI 524304-edge      mlx4_0-15
  94:        485          0          0          0          0          0          0          0  IR-PCI-MSI 2623488-edge      snd_hda_intel:card0
  95:          0       1109          0          0          0          0          0          0  IR-PCI-MSI 2633728-edge      snd_hda_intel:card1
 NMI:         10         15          7          4          5          4          5          5   Non-maskable interrupts
 LOC:      84576      35196      81896      37445      75845      31837      94738      46771   Local timer interrupts
 SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
 PMI:         10         15          7          4          5          4          5          5   Performance monitoring interrupts
 IWI:      51332      35838      44445      23148      42192      19702      54409      35380   IRQ work interrupts
 RTR:          0          0          0          0          0          0          0          0   APIC ICR read retries
 RES:     732459      91844     540442     144222     328744     127968     421363     242971   Rescheduling interrupts
 CAL:      23911      24956      27824      15877      22110      16297      22438      14298   Function call interrupts
 TLB:      17204      18907      21877       9135      15652      10116      15791       7888   TLB shootdowns
 TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts
 DFR:          0          0          0          0          0          0          0          0   Deferred Error APIC interrupts
 MCE:          0          0          0          0          0          0          0          0   Machine check exceptions
 MCP:          7          7          7          7          7          7          7          7   Machine check polls
 HYP:          0          0          0          0          0          0          0          0   Hypervisor callback interrupts
 HRE:          0          0          0          0          0          0          0          0   Hyper-V reenlightenment interrupts
 HVS:          0          0          0          0          0          0          0          0   Hyper-V stimer0 interrupts
 ERR:          1
 MIS:          0
 PIN:          0          0          0          0          0          0          0          0   Posted-interrupt notification event
 NPI:          0          0          0          0          0          0          0          0   Nested posted-interrupt event
 PIW:          0          0          0          0          0          0          0          0   Posted-interrupt wakeup event
Attachments
v16-cat-proc-interrupts.txt
(11.86 KiB) Downloaded 20 times
Screenshot at 2019-07-16 19-40-35.png
Screenshot at 2019-07-16 19-40-35.png (423.36 KiB) Viewed 876 times
Screenshot at 2019-07-16 19-40-03.png
Screenshot at 2019-07-16 19-40-03.png (439.29 KiB) Viewed 876 times
Screenshot at 2019-07-16 19-38-44.png
Screenshot at 2019-07-16 19-38-44.png (416.79 KiB) Viewed 876 times

domih
Posts: 118
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by domih » Wed Jul 17, 2019 3:20 pm

mad_ady wrote:
Tue Jul 16, 2019 2:25 am
I'm not familiar with infiniband (is it mostly found in the datacenter?)
InfiniBand is a non-ethernet protocol originally created by Mellanox. There is now an organization IBTA defining the standard (where obviously Mellanox is a big 800-pound Gorilla). Mellanox has been/is enormously successful and was bought for big $$$ by NVidia. Intel could not buy them probably a story of $$$ but also of potential governmental rejection because they already bought QLogic and others in the same industry. NVidia is happy: they now have a fast interconnect (using RDMA, continue reading) for their GPU racks and farms.

Their protocol can run on passive DAC copper or on active optical cables. The latter allows long distances (several hundred meters). Example: universities or large organizations or very huge data centers.

InfiniBand is NOT Ethernet: it does not use the IP stack. It instead uses RDMA (Remote Direct Memory Access). It is based on the notion of "client" app and "server" app. The RDMA concept is very simple: a "client" app issues an RDMA request, the NIC copies the data directly from the app buffers (which the app created for that purpose.) The NIC then takes care of sending it to the "server" NIC. Upon reception the latter copies the data directly to the "server" app memory (which the app created for that purpose.) It is fast, low latency and the CPUs on the "client" and "server" machines are not involved in the network request-response between the two apps.

From there, Mellanox implemented IPoIB (IP over Infiniband) to transport IP-based network communication over its RDMA protocol. Quite successful: data centers and large organizations love it because they can leverage the IB speed using their IP-based apps with no code change. Downside: there is overhead in putting the IP packets on top of RDMA packets and... the CPUs are back in the loop managing the IP stack. Let's say you loose 30% of bandwith, but it is still very fast compared to 1Gbe or 10Gbe Ethernet.

From there, under the pressure of its customers and competitors (i.e. QLogic, Chelsio, etc), Mellanox implemented RoCE (RDMA over Converged Ethernet). Big term just to say that you can do RDMA over Ethernet (basically the reverse of IPoIB). Again the data centers and large organizations love it because if they already have good quality Ethernet cables: they don't have to lay new cables.

Competitors to Mellanox (i.e. Chelsio) will say that InfiniBand is useless since the introduction of RoCE (in their dreams, meanwhile Mellanox makes money).

Intel is impotently furious, they were the top number one for Gigabit Ethernet but since then, they never were able to catch up with Mellanox. Their 100Gbe cards suck and their OmniPath network thingy does not really move the populace. Intel is working hard on catching up the 100Gbe wagon, but Mellanox is already selling 200Gbe solution (NICs are two cards using two PCIe 3 x16). Both Mellanox and Broadcom already have 400Gbe sample products.

Mellanox designs and sells network cards, switches and cables. All the Big Guys like Dell, EMC, HP, etc. distributed/distribute network products made by Mellanox just putting their name on it. Given the torrent of hardware sold in the last ten years and the data centers as well as organizations updating their hardware for even faster speed, the used products end up on eBay at prices you can resist. You can get used dual 40/56Gbps Mellanox card for less $75 (sometimes less than $50), meaning less than the cheapest 10Gbe Ethernet cards on sales (as of this writing). You just need to be patient waiting for a (reputable) guy selling at low price. Risk? Not too much, you just to need check the % feedback of the vendor, read carefully the description and refer to the Mellanox technical archives to find out what the product exactly is. In other words, do your home work.

Note that the MSRP for these cards and cables are in the $300, $400 range. Although these are "enterprise prices", I doubt anybody bought them at these prices. As they say, nothing sells more than a 25, 30, 50% discount, so let's start with very high prices.

Regarding the 'In other words, do your home work.' it failed for me twice: once I received two optical cables that where bent but seemed to work. It's only later I found out that they triggered too many retries (IB is a lossless protocol: it won't theoretically drop a packet.) Cost: $100. Then the 18-port 56Gbps switch: I checked it when I received it, turned it on, connected a PC to it, launched the network manager: all OK! It is only last week-end I found out that this is basically the only things that work: connecting another PC to port 2 to 18 never activated the connection. Cost: $175. So my advice is simple: try everything when receiving the hardware. For me it was too late, also the returning costs (on me) not worth the trouble. I'll buy another one when I'm tired of 40Gbs.

Mellanox cards can be IB or Ethernet cards. With a dual card you can use port 1 for IB, same for port 2 but you can also use Ethernet on port 2 while still using IB on port 1.

As an initial try, you do not need a switch. Just buy two cards and connect them directly, launch the network manager and you have a 40 or 56Gbs connect between your two PCs (depending on the cable). Cost? Let's say $75 per card (if you are impatient) and $50 for a 3-meter optical cable (40Gbs) or $60 (56Gbs). All the software you need is downloadable (free of charge) from the Mellanox web site.

On eBay:

Card (you need two of them)
https://www.ebay.com/sch/i.html?_from=R ... BT&_sop=15 (dual port)
https://www.ebay.com/sch/i.html?_from=R ... BT&_sop=15 (single port)

Note: these used cards come with a low-profile bracket, so buy the full bracket too ($5 on eBay).

Optical cable (you need one)
https://www.ebay.com/sch/i.html?_from=R ... 10&_sop=15 (40Gbs max)
https://www.ebay.com/sch/i.html?_from=R ... 1V&_sop=15 (56Gbs max)

WARNING: as mentioned earlier in this thread, these cards expect a PCIe 3 x8 slot to work at max speed. Less than 8 lanes and/or less than PCIe 3 (i.e. PCIe 2) will significantly degrades the speed (you could end up at 12 GbE with IPoIB on "old" hardware as shown earlier in this thread).

Software you'll need (*):
- Linux: http://www.mellanox.com/page/products_d ... _family=26 (different versions for each distrib, download the one you need)
For linux, you are looking for MLNX_OFED_LINUX-4.6-1.0.1.1-rhel7.6-x86_64.tgz, or MLNX_OFED_LINUX-4.6-1.0.1.1-ubuntu18.04-x86_64.tgz, etc.
- Windows: https://www.mellanox.com/page/products_ ... sw_drivers
For Windows, you are looking for MLNX_VPI_WinOF-5_50_52000_All_win2019_x64.exe or later.
- Linux tools for updating card firmware (package is called MFT, app is called flint): http://www.mellanox.com/page/management_tools
- Most recent firmware for the two cards mentioned above: http://www.mellanox.com/page/firmware_table_ConnectX3IB
- Files are:
MCX354A-FCBT MT_1090120019 fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752 044a3e082f9dc6ec0ac458d3ad0274be
MCX353A-FCBT MT_1100120019 fw-ConnectX3-rel-2_42_5000-MCX353A-FCB_A2-A5-FlexBoot-3.4.752 d18b52f5464dbff50b88271e9a86de66

(*) Sometimes the Mellanox website is broken (missing CSS?). If you see pages that look like mid-90s style, try later until the site looks modern. Each download page has a lot of blah blah blah with at the bottom a clickable, expandable multi-column list. Usually, you also find the links to the corresponding manuals. These manuals do not spend much time into "tutorial mode" so have fun with them.

WARNING: the firmware 2.42.5000 is required by the most recents version of MLNX_OFED so you'll probably have to update the cards firmware. Most of the cards on eBay are OEM cards from Dell, HP, etc not updated with the last firmware. The current Mellanox firmware tool will NOT let you override the OEM firmware. The workaround is to use an older version of MFT that does. After trial and errors going through the older versions... but not too old (!)... I found out that this version of MFT does the job: mft-4.0.0-53.tgz, you'll have to use the flint option: --allow-psid-override (**) It's from memory but I think it is this option.

(**) This means that you KNOW what you're doing: burning the wrong firmware on a card will make it useless.

Because this version of MFT is old, it will be overridden when you then install MLNX OFED. Simple process trick: have an old PC able to recognize the Mellanox PCIe cards and only install mft-4.0.0-53.tgz on it. Burn the firmware on that old PC then move the card to the PC you'll actually use where MLNX OFED is/will be installed. In doing so, you do not spend your time installing MFT, installing MLNX OFED, uninstalling MLNX_OFED, reinstalling MFT for each card you buy. That is if you buy many cards over months waiting for the good deal on eBay.

Note: the card mentioned here are ConnectX-3. Do not buy ConnectX-2, too old. You will quickly see that used ConnectX-4 and ConnectX-5 are horribly expensive (that's because they are pretty recent and allow 100Gbs). ConnectX-3 "only" does 40 and 56Gbs.

If you do not want to update the cards firmware (most the recent features of MLNX_OFED are for ConnectX-4 and ConnectX-5 after all) do not use MLNX OFED. Most of the InfiniBand daemons, drivers and tolls you need are built-in in Linux. The kernel must be RDMA enabled (usually the case on recent distributions) and look for the InfiniBand packages available for your Linux via apt or yum. The MLNX OFED installer for Linux installs very similar files plus niceties (meaning the most recent bug fixes and additional tools), that's it. Mellanox even have manuals telling you which packages you have to install if not using MLNX_OFED (example: Ubuntu_18_04_Inbox_Driver_User_Manual.pdf, Ubuntu_19_04_Inbox_Driver_User_Manual.pdf).

Tip: with Ubuntu 19.04, clone https://github.com/linux-rdma/rdma-core (or use pip3) to get the new Python PyVerbs package. It exposes the InfiniBand entrails for Python. Have fun, documentation is scarce at this point (tip: read the source code and refer to the RDMA C/C++ documentation). Useful only if you want to write a Python that uses RDMA instead of simply communicate with no change over IPoIB.

InfiniBand on ARM? Possible but you'll have to rebuild the kernel with RDMA enabled and then build the github project mentioned in the previous paragraph. That's for the adventurous and you have to find a price-accessible ARM board with a PCIe slot with enough oomph to handle the card.

So after reading this, you precipitously spent about $75 + $75 + $50/$60 on eBay. You got the cards, the cable. You passed flying colors updating the cards firmware, you installed MLNX OFED. What do you do from there? Let's say your 2 PCs run Linux (RedHat, CentOS, Ubuntu, SuSE).

Your two cards will probably have a green light on which means they detected that there is something out there.

Run (as root or sudo) on one machine:

service opensmd start

This will start the network manager (note: you also do that for unmanaged switches, the software comes from one PC via the IB card)

Note: you can start opensmd on several PCs, does not hurt.

Both cards should now have a shining yellow light on. If not rinse and repeat using the documentation (or just continue, maybe the LED is dead).

Tip: use ibv_devinfo which display the current card configuration and status (including port_lid)

On each PC:
- Load the IPoIB driver: modprobe ipoib.
- Using NetworkManager's nmtui, create a new connection (InfiBand will be in the menu), assign a fixed IP address to the card (using a different subnet than your current one is OK and preferable as a first experiment). On dual port cards, this will be ib0. Activate the connection. Get out of nmtui.
- Check the result with:

ip a

At this point you should be able to ping each other PC over IPoIB. If not rinse and repeat using the documentation.

A few "pure" IB commands as first steps:

ibhosts
List hosts present on the local net

ibswitches
List the switches present on the local net (if you finally buy one)

iblinkinfo
List the links present on the local net (each PC should see each other)

ibping -S -d -v
Starts IB ping server on one hosts

ibping 1
Ping first host on the local net, the "1" is the LID of the PC acting as ping server. The LIDs are given in iblinkinfo.

ibstat
Display current card configuration and status


For testing IPoIB, use iperf3 over the IP addresses you assigned to the IB cards. If you have "good" machines you should get 30++GbE transfer over 40Gbs, more over 56Gbs.

Thanks to IPoIB, you can use ssh, samba, whatever IP-based app over the fast connect. For example compare copying files (via Samba) over your regular gigabit network and then over IPoIB to feel the difference (mostly apparent when you copy a giant multi-GB file, with smaller files the overall speed degrades, like it does over 1GbE).

On Windows, you may have to use some obsure incantation in PowerShell to tell the SMB client to use all the NICs, and most importantly the correct one, when accessing the same machine via different names of IPs. I forgot what it was and forgot to type it down so you'll have to look for it.

Microsoft SMB Direct does not work with Samba (because Samba does not implement it yet). So you have to use a recent Windows Server and Windows 10 if you want to evaluate SMB Direct.

If you have 3 PCs, you can still do it without a switch. You can daisy chain using dual cards (speed can suffer between to PCs communicating through the 3rd one in the middle).

Source of information:
- Mellanox Manuals,
- https://community.mellanox.com (registration is free)
- Enterprise web sites forums (i.e. https://forums.servethehome.com/index.php, free registration)

Just be aware that this technology has been going for 10+ years, so there is plenty of obsolete information on the Net. Practice good judgement in your reading.

That's all.

FINAL NOTE

THE INFORMATION IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE INFORMATION OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

I did not write this post, my chimpanzee did.
These users thanked the author domih for the post:
mad_ady (Wed Jul 17, 2019 9:22 pm)

User avatar
odroid
Site Admin
Posts: 32146
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 119 times
Been thanked: 297 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by odroid » Wed Jul 17, 2019 4:44 pm

domih wrote:
Wed Jul 17, 2019 3:20 pm
I did not write this post, my chimpanzee did.
I love it! :D

BTW, can you please write an article about your H2 experiences for our Magazine?

User avatar
mad_ady
Posts: 6558
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 169 times
Been thanked: 128 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by mad_ady » Wed Jul 17, 2019 9:23 pm

Pretty informative, thanks! I hadn't heard of this technology.

domih
Posts: 118
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by domih » Thu Jul 18, 2019 4:28 pm

odroid wrote:
Wed Jul 17, 2019 4:44 pm
domih wrote:
Wed Jul 17, 2019 3:20 pm
I did not write this post, my chimpanzee did.
I love it! :D

BTW, can you please write an article about your H2 experiences for our Magazine?
OK, I'll contact you.

domih
Posts: 118
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Review of the H2 based on FACTS and sparing no one

Unread post by domih » Thu Jul 18, 2019 4:41 pm

mad_ady wrote:
Wed Jul 17, 2019 9:23 pm
Pretty informative, thanks! I hadn't heard of this technology.
You're welcome.

Do not worry, less than one year ago I knew nothing but the name InfiniBand and what it generally does in the blackplane of racks. In addition, less than one year ago I absolutely knew nothing about SBCs except that there was lower power CPUs called ARM powering many appliances that the schmucks like me use.

It all started when I wanted to upgrade to 10GbE. Pricing for new hardware for the populace was high and is still (obscenely) high for what it is. So I decided to dig into eBay to look for used hardware. That's when I discovered there was a whole world of high bandwidth hardware for the taking, the trick being to read a lot and do your home work so that you know what you are buying. In parallel I bought a Pi 3 B+ half for fun and started to dig into SBCs and found Odroid which led quickly to the XU4 then the HC2 which replaced my old QNAP for a fraction of the price (and 100+ MB/s delivery instead of 25+ MB/s with the QNAP). From there it was natural to go N2, Rock Pi4, some other disappointing boards and the H2 plus the V1605B in the middle.

In other words, it is the usual story in the computer industry: you stop swallowing the blue pill (spend and spend for ready to use stuff) and take the red pill (Do It Yourself). Anyway it is fun, isn't it?

Post Reply

Return to “General Topics”

Who is online

Users browsing this forum: No registered users and 1 guest