Add-on to H2 - M2 to pciE slot

RomaT
Posts: 226
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 35 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Sat Aug 17, 2019 8:09 pm

I received my two network cards, LR-LINK 6880BT
with chip AQC107, Vendor 1D6A, Device: 07B1
supports automatic detection MDI/MDIX (card to card or card to switch)
I installed one card in my H2, Ubuntu 18.04.02 LTS amd64 sees this card,
but I can’t check yet, because refinement of my chassis is required
.
hardware.png
hardware.png (86.73 KiB) Viewed 1984 times
.
P8091053s.JPG
P8091053s.JPG (277.99 KiB) Viewed 641 times
.
P8231103s.JPG
P8231103s.JPG (150.08 KiB) Viewed 643 times
.
P8101088s.JPG
P8101088s.JPG (371.31 KiB) Viewed 642 times
.
test2_h2_from_ramdisk_to_pc.png
test2_h2_from_ramdisk_to_pc.png (12.53 KiB) Viewed 637 times
Last edited by RomaT on Sat Aug 31, 2019 4:44 pm, edited 12 times in total.
These users thanked the author RomaT for the post (total 2):
Mullcom (Sat Aug 17, 2019 8:22 pm) • domih (Sat Aug 17, 2019 9:41 pm)

domih
Posts: 119
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Mon Aug 19, 2019 8:13 am

I tested the HX424S14IBK2/32 memory regarding any change in bandwidth running iperf3 over InfiniBand. It shows a 2.3% increase (but really?) see code below. HOWEVER, it's not "reproducible" on repeated testing. So I would conclude it is within statistical lack of precision. BUT, I do get much more consistently 11.1Gbe as client and 14.4Gbe as server in testing with the i5 9660K PC, so I would say the new memory does have some very tiny effect.

Code: Select all

Old Memory (3 minutes testing, loop on dual card)(*)

[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-180.00 sec   540 GBytes  25.8 Gbits/sec    0             sender
[  4]   0.00-180.00 sec   540 GBytes  25.8 Gbits/sec                  receiver

New Memory  (3 minutes testing, loop on dual card)(*)

[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-180.00 sec   553 GBytes  26.4 Gbits/sec    0             sender
[  4]   0.00-180.00 sec   553 GBytes  26.4 Gbits/sec                  receiver
(*) This means: from port #1 to port #2 of the card, no other PC involved. Thus iperf3 records and sums both the client and server throughput, with another PC, the measured speed is obviously halfed.

Funny fact, during the dual port loop testing I did have my own little "wow" effect:

Code: Select all

.../...
[  4]   6.00-7.00   sec  2.92 GBytes  25.1 Gbits/sec    0   1.25 MBytes       
[  4]   7.00-8.00   sec  3.19 GBytes  27.4 Gbits/sec    0   1.25 MBytes       
[  4]   8.00-9.00   sec  4.74 GBytes  40.7 Gbits/sec    0   1.25 MBytes       <== Wow
[  4]   9.00-10.00  sec  3.18 GBytes  27.4 Gbits/sec    0   1.25 MBytes       
[  4]  10.00-11.00  sec  3.16 GBytes  27.2 Gbits/sec    0   1.25 MBytes
.../...
Happened only once obviously. So with just a little push somewhere would the H2 goes up to 20Gbe?

domih
Posts: 119
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Mon Aug 19, 2019 8:41 am

My Odroid H2 / InfiniBand project in its final assembly.
20190818_154854.jpg
The SFX PSU + accessories with the 10TB and SSD assembled on an acrylic bench, the H2 in its own box (case type 2)
20190818_154854.jpg (170.45 KiB) Viewed 1969 times
20190818_154903.jpg
SFX PSU in the background. Right foreground: the ATX breakout card making it easy to grab the 12v. Power led and button on the right (still has to find out a nice top to put on the button. Left foreground: a little fan PWM card. I use it to set the Mellanox card fan to a sufficient but silent speed.
20190818_154903.jpg (136.42 KiB) Viewed 1969 times
20190818_154911.jpg
The ADT Link cable slot is fixed to the side pannel of the H2 case type 2 using screws and white plastic washers from my stash.
20190818_154911.jpg (144.76 KiB) Viewed 1969 times
20190818_155149.jpg
Running Ubuntu Mate 18.04. When in pure NAS mode, it just boots in multi-user target.
20190818_155149.jpg (166.1 KiB) Viewed 1969 times
CONCLUSION: my final assembly is much less sophisticated and nice looking than what RomaT has built!
These users thanked the author domih for the post (total 2):
odroid (Mon Aug 19, 2019 5:59 pm) • rooted (Thu Aug 29, 2019 3:19 pm)

domih
Posts: 119
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Mon Aug 19, 2019 8:53 am

Yet another little board that could qualify for making a not too high TDP SOHO NAS. Given the information I would say it is probably half-way between the Odroid H2 and an AMD v1605b board. It is also much less expensive than the industrial v1605b board I setup months ago.

Biostar A10N-8800E
https://www.biostar.com.tw/app/en/mb/in ... p?S_ID=935
https://www.ebay.com/itm/Biostar-A10N-8 ... SwdtBc9SVc

Review
https://www.anandtech.com/show/14368/bi ... ard-review
https://www.techpowerup.com/review/biostar-a10n-8800e

CPU Comparison
https://www.cpubenchmark.net/cpu.php?cp ... Hz&id=3159
https://www.cpubenchmark.net/cpu.php?cp ... 0P&id=2567
https://www.cpubenchmark.net/cpu.php?cp ... 5B&id=3331

RomaT
Posts: 226
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 35 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Mon Aug 19, 2019 5:46 pm

If I were now considering a miniITX platform,
I would take on the intel B360 or H370 chipset (DMI 3.0 and PCI-E 3.0)
because intel 110/210/310 chipsets with DMI 2.0 and PCI-E 2.0
the processor has already chosen according to the capabilities.
domih wrote:
Mon Aug 19, 2019 8:13 am
Happened only once obviously. So with just a little push somewhere would the H2 goes up to 20Gbe?
This is very doubtful, network cards may communicate at such speeds, but only within themselves, because beyond their PCIe 2.0 with 4 lanes limiting the maximum speed 16 Gbps (2GBytes/s)
I think that even on a RAM-disk from the network it is impossible to write faster 16Gbps.
Try recording from the network to ramdisk H2:

Code: Select all

# mkdir /mnt/ramdisk 
# mount -t tmpfs -o rw,size=16G tmpfs /mnt/ramdisk

User avatar
Mullcom
Posts: 93
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 51 times
Been thanked: 5 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Mon Aug 19, 2019 8:08 pm

PCI Express link performance

GBITS to Gbit
2.0 gigabytes = 16 gigabits

The gigabit has the unit symbol Gbit or Gb.
Using the common byte size of 8 bits, 1 Gbit is equal to 125 megabytes (MB) or approximately 119 mebibytes (MiB).

RomaT
Posts: 226
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 35 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Mon Aug 19, 2019 8:26 pm

I want 126GB/s :D (PCI-E 6.0 16 lanes)
These users thanked the author RomaT for the post:
Mullcom (Mon Aug 19, 2019 8:46 pm)

User avatar
Mullcom
Posts: 93
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 51 times
Been thanked: 5 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Mon Aug 19, 2019 8:47 pm

RomaT wrote:
Mon Aug 19, 2019 8:26 pm
I want 126GB/s :D (PCI-E 6.0 16 lanes)
Can you take a better picher of you H2 build with big fan at top?

I need to have some Ides how to build my.

RomaT
Posts: 226
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 35 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Mon Aug 19, 2019 9:54 pm

Mullcom wrote:
Mon Aug 19, 2019 8:47 pm
Can you take a better picher of you H2 build with big fan at top?
Honestly I didn’t understand what you want to ask.
Can I, at your request, put a large fan on top of the H2 radiator? what for? what will it give ?
a. there are many factors (Environment, flow direction, flow volume, chassis design and location in it, amount of heat generated by all components)
b. this question is beyond the scope of the topic (off topic)
c. look here viewtopic.php?p=266098#p266098
d. More here viewtopic.php?p=261805#p261805

User avatar
mad_ady
Posts: 6558
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 170 times
Been thanked: 129 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by mad_ady » Mon Aug 19, 2019 10:42 pm

I think he meant a better picture.

User avatar
Mullcom
Posts: 93
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 51 times
Been thanked: 5 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Mon Aug 19, 2019 11:31 pm

Yes . More details of the build. Closer shots.

Beem my message up with phone.


User avatar
Mullcom
Posts: 93
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 51 times
Been thanked: 5 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Tue Aug 20, 2019 12:24 am

I did get my networks card today. ;)Image

Beem my message up with phone.

These users thanked the author Mullcom for the post:
domih (Wed Aug 21, 2019 7:49 am)

RomaT
Posts: 226
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 35 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Tue Aug 20, 2019 3:57 pm

That's cool ! Just keep in mind that the actual transmission speed will not exceed the 5Gbit/s,
because the SATA hard drive has a maximum speed of 250 MB/s (2 Gbit/s), SATA SSD maximum 550 MB/s (4.4 Gbit/s)
high speed can only be obtained by creating raid arrays, through expanders, but having only two ports of SATA on 6Gbits/s,
theoretical transmission speed may reach a maximum of 12 Gbits/s ( 1.5 GBytes/s) ~ total speed of seven hard drives.
even if we assume that the work with the ramdisk, then the PCIe bus limit will still not allow to exceed 16 Gbit/s
from here we can conclude that on ODROID-H2 a network card of more than 10 Gbit/s does not make sense, optimally 2.5 - 5 Gbits/s
but 1 Gbps network interfaces are really outdated and their use is just a waste of PCIe bus lanes, which could be used for the SATA controller,
this is a big mistake HARDKERNEL, because 1GbE interfaces can be obtained by cheap USB adapters.
or to satisfy multi-interface lovers - just install a PCIe adapter for at least 6 ports 1GbE.
Last edited by RomaT on Tue Aug 20, 2019 6:25 pm, edited 3 times in total.

User avatar
Mullcom
Posts: 93
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 51 times
Been thanked: 5 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Tue Aug 20, 2019 4:51 pm

RomaT wrote:That's cool ! Just keep in mind that the actual transmission speed will not exceed the 5Gbit/s,
because the SATA hard drive has a maximum speed of 250 MB/s (2 Gbit/s), SATA SSD maximum 550 MB/s (4.4 Gbit/s)
high speed can only be obtained by creating raid arrays, through expanders, but having only two ports of SATA on 6Gbits/s,
theoretical transmission speed may reach a maximum of 12 Gbits/s ( 1.5 GBytes/s) ~ total speed of seven hard drives.
even if we assume that the work with the ramdisk, then the PCIe bus limit will still not allow to exceed 16 Gbit/s
from here we can conclude that on ODROID-H2 a network card of more than 10 Gbit/s does not make sense, optimally 2.5 - 5 Gbits/s
but 1 Gbps network interfaces are really outdated and their use is just a waste of PCIe bus lanes, which could be used for the SATA controller
Thx for the tip.

I am not going to build Nas of this. It's going to be my router out to internet. My Nas going to have 5disk in Raid-Z1 so I going to get good performance. Maybe I put in some m2 disk from Samsung to store virtualization data on.


I know this 40gbit card is to much for H2. But it's going not to be my bottleneck also. Have this setup going to make it easier to upgrade in the future.

Beem my message up with phone.



RomaT
Posts: 226
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 35 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Wed Aug 21, 2019 2:22 am

Finally, the first inclusion and testing of 10 Gb/s network cards,
ODROID-H2 with 10 Gb/s network card, Ramdisk 16GB, through SATA expanders Storage LVM2 HDD+SSD and single HDDs
PC with 10 Gb/s network card, PCIe SSD (NVMe)
according to the SMB Network protocol - Natural tests copying a file of 13 GBytes !!!

from H2 single HDD to PC:
test_h2_to_pc.png
test_h2_to_pc.png (92.49 KiB) Viewed 1730 times
.
from H2 Ramdisk to PC:
test_h2_from_ramdisk_to_pc.png
test_h2_from_ramdisk_to_pc.png (89.41 KiB) Viewed 1730 times
.
from PC to H2 Ramdisk:
test_pc_to_h2_on_ramdisk.png
test_pc_to_h2_on_ramdisk.png (100.25 KiB) Viewed 1730 times
.
from H2 Storage LVM2 HDD+SSD to PC
test_h2_from_LVM2_to_pc.png
test_h2_from_LVM2_to_pc.png (94.86 KiB) Viewed 1730 times
.
from PC to H2 Storage LVM2 HDD+SSD
test_pc_to_h2_on_LVM2.png
test_pc_to_h2_on_LVM2.png (102.89 KiB) Viewed 1730 times
These users thanked the author RomaT for the post:
odroid (Wed Aug 21, 2019 11:16 am)

User avatar
Mullcom
Posts: 93
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 51 times
Been thanked: 5 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Wed Aug 21, 2019 2:32 am

RomaT wrote:Finally, the first inclusion and testing of 10 Gb/s network cards,
ODROID-H2 with 10 Gb/s network card, Ramdisk 16GB, through SATA expanders Storage LVM2 HDD+SSD and single HDDs
PC with 10 Gb/s network card, PCIe SSD (NVMe)
according to the SMB Network protocol - Natural tests copying a file of 13 GBytes !!!

from H2 single HDD to PC:
test_h2_to_pc.png
.
from H2 Ramdisk to PC:
test_h2_from_ramdisk_to_pc.png
.
from PC to H2 Ramdisk:
test_pc_to_h2_on_ramdisk.png
.
from H2 Storage LVM2 HDD+SSD to PC
test_h2_from_LVM2_to_pc.png
.
from PC to H2 Storage LVM2 HDD+SSD
test_pc_to_h2_on_LVM2.png
Is only me? Quality of IMG is so low that I can't see text or numbers.



Beem my message up with phone.


RomaT
Posts: 226
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 35 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Wed Aug 21, 2019 2:44 am

your application doesn’t work well with PNG pictures
I squeezed screenshots into JPG files:
net.zip
(595.12 KiB) Downloaded 13 times
These users thanked the author RomaT for the post:
Mullcom (Wed Aug 21, 2019 2:54 am)

User avatar
Mullcom
Posts: 93
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 51 times
Been thanked: 5 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Wed Aug 21, 2019 2:55 am

RomaT wrote:your application doesn’t work well with PNG pictures
I squeezed screenshots into JPG files:
net.zip
So you get full BW then. Nice.

I see that CPU load is 23% that is not to much. I like it.

Beem my message up with phone.


RomaT
Posts: 226
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 35 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Wed Aug 21, 2019 3:03 am

All screenshots are made on the PC side, Which has an Intel i5-9600K processor :D
.
pc.jpg
pc.jpg (97.87 KiB) Viewed 1715 times

User avatar
Mullcom
Posts: 93
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 51 times
Been thanked: 5 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Wed Aug 21, 2019 4:40 am

RomaT wrote:All screenshots are made on the PC side, Which has an Intel i5-9600K processor :D
.
pc.jpg
Okej. Did you see how CPU load on H2?

It's a bit interesting how much it need to work with this setup.

Beem my message up with phone.


domih
Posts: 119
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Wed Aug 21, 2019 7:56 am

RomaT wrote:
Tue Aug 20, 2019 3:57 pm
...from here we can conclude that on ODROID-H2 a network card of more than 10 Gbit/s does not make sense...
I agree to disagree on that particular point :D

RomaT
Posts: 226
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 35 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Wed Aug 21, 2019 12:31 pm

Mullcom wrote:
Wed Aug 21, 2019 4:40 am
Did you see how CPU load on H2?
when copying from 5% to 15%, at rest 2-3%
domih wrote:
Wed Aug 21, 2019 7:56 am
I agree to disagree on that particular point :D
it's just IMHO, my natural tests showed that even when working with ramdisk, the speed does not exceed 8Gbps, when copying files over a network.
dancing with tambourines around "strict sync" and size "Jombo Packet" - Only have reduced speed.
I liked caching LVM2, when reading from it, the speed is close to the speed of ramdisk.
What surprised me, because SATA port is limited to 6Gbps, LVM array (HDD+SSD)on one port SATA H2, through the expander.
backups of the system began to pass twice as fast:
.
backup.png
backup.png (29.79 KiB) Viewed 1611 times
.

only i noticed one detail, When creating metadata size, we specified only 48 MB,
while 1000 times less cache size is recommended, what's for a 480GB drive = 480MB
which is 10 times more than the specified.
can this be somehow fixed on the fly, without data loss?

domih
Posts: 119
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Wed Aug 21, 2019 3:25 pm

To RomaT and Mullcom (mostly RomaT)

it's just IMHO, my natural tests showed that even when working with ramdisk, the speed does not exceed 8Gbps, when copying files over a network.

Yep, I'm not into dogma either. The thing is that if I were to spend 4x or 5.6x the $$$ to get 40Gb/s or 56Gb/s on a hardware that would never reach it, yes it would not make any sense. But that's not the case, the second hand InfiniBand/Ethernet Mellanox hardware on eBay is something like 3 times less expensive compared to brand new cheap 10Gbe cards (from $25 to $35 for a connectX-2, I'm not even talking about the so-called "Intel" ones that reach $400 or even $500, where the ratio becomes 20 times less expensive and still providing more performance) and 10 times less expensive for the switches ($150 for a 40Gb/s or 56Gb/s 18-port InfiniBand switch while a brand new 10Gbe Ethernet switch is around $1,500 on this side of the ocean). I guess it is also a question of philosophy or use case anyway. I do a lot of distributed stuff so I want the fastest network without spending the $$$ they do in the data centers otherwise I would have gone with 100 or 200 Gb/s :o, I guess I will in 20 years when the second hand hardware lends on eBay in 2039 :D With 56Gb/s in FDR mode (card to card or via switch), I reach about 45Gbe between an i5 9660K and a R9 3900X. The Odroid H2 will never reach these speeds, but I find it nice it can talk to them on the same network with a more than decent speed given the level of the hardware, its costs and its power consumption at around 35-40W when receiving or sending files.

...the speed does not exceed 8Gbps...
I do not know what kind of encoding 10Gbe uses, if it is like "regular" InfiniBand, it's 8b10b so right of the bat 20% of the bandwidth goes in smoke in terms of payload. If you do reach closed to 8Gb/s in iperf3 you should be very happy (again this is assuming that the 10Gbe encoding is 8b10b which I do not know if it is the case).

Dancing with tambourines around "strict sync" and size "Jombo Packet" - Only have reduced speed.
I liked caching LVM2, when reading from it, the speed is close to the speed of ramdisk.
What surprised me, because SATA port is limited to 6Gbps, LVM array (HDD+SSD)on one port SATA H2, through the expander.


You do not get consistent timings both because Windows and Linux cache files one way or another. It's a little better on Linux where 'sync' vs 'async' actually means 'sync' and 'async' when testing with NFS (consistent and repeatable). When you get more than 6Gb/s that probably because the file you read is already in memory. The best is to simply reboot between tests. I will check the "strict sync" with Samba, I did not have time yet. For NFS over RDMA (no IP stack involved, no CPU involved), it was an obvious dramatic speed increase from sync to async. See one of my previous posts.

Can this be somehow fixed on the fly, without data loss?
You know as much as I do. I just use the sizes recommended in the man pages. If you want to change the sizes, I'm pretty sure you have to reformat. On my side I'm planning to tinker with I/O buffers for several months before putting the H2 NAS "in production". I have 32GB of memory (at least half of it) available for the RDMA and TCP buffers. I already know that for RDMA in async, it's pretty much like having RAM disk. I prefer this line of inquiry because then one does not have to handle the sync between the RAM disk and the physical disks. Plenty of docs reading in perspective.

Mullcom: Is only me? Quality of IMG is so low that I can't see text or numbers.
I think you meant "I can't read" instead of "I can't see". On my side I can recognize a few of the Cyrillic characters (but I don't read Russian!) can guess the words based on the context. On the other hand, it almost look like the font is not anti-aliased and yes the digits in the charts look deformed. I downloaded the ZIP and the images are of the same quality. Overall they look pretty much as good as they can get on both the forum and from the ZIP.

CONCLUSION: you are now the proud owner of an H2 that leaves all the Gigabit-challenged PCs in the dust. Welcome to the club!

I'm going to bed now. Hi to the Kama River people :-)
These users thanked the author domih for the post:
Mullcom (Wed Aug 21, 2019 7:31 pm)

RomaT
Posts: 226
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 35 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Wed Aug 21, 2019 3:38 pm

domih wrote:
Wed Aug 21, 2019 3:25 pm
The thing is that if I were to spend 4x or 5.6x the $$$ to get 40Gb/s or 56Gb/s on a hardware that would never reach it, yes it would not make any sense.
When viewed from the cost and other computers, then yes, it’s more attractive to build like you.
I considered from the point of view of the possibility of ODROID-H2.
domih wrote:
Wed Aug 21, 2019 3:25 pm
The best is to simply reboot between tests.
just so the above tests were performed (before each test, the PC was rebooted)
only two tests were performed with ramdisk on H2 - recording file on ramdisk from the network and reading file from ramdisk from the network
domih wrote:
Wed Aug 21, 2019 3:25 pm
you are now the proud owner of an H2 that leaves all the Gigabit-challenged PCs in the dust. Welcome to the club!
Hi to the Kama River people :-)
;)
domih wrote:
Wed Aug 21, 2019 3:25 pm
it almost look like the font is not anti-aliased and yes the digits in the charts look deformed.
at work I looked at my own screenshots in another browser and they really look blurry :lol:
but at home I have a clear image on my computer, I don’t know why so ...
These users thanked the author RomaT for the post:
Mullcom (Wed Aug 21, 2019 7:32 pm)

User avatar
Mullcom
Posts: 93
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 51 times
Been thanked: 5 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Wed Aug 21, 2019 7:35 pm

RomaT wrote:
domih wrote:
Wed Aug 21, 2019 3:25 pm
The thing is that if I were to spend 4x or 5.6x the $$$ to get 40Gb/s or 56Gb/s on a hardware that would never reach it, yes it would not make any sense.
When viewed from the cost and other computers, then yes, it’s more attractive to build like you.
I considered from the point of view of the possibility of ODROID-H2.
domih wrote:
Wed Aug 21, 2019 3:25 pm
The best is to simply reboot between tests.
just so the above tests were performed (before each test, the PC was rebooted)
only two tests were performed with ramdisk on H2 - recording file on ramdisk from the network and reading file from ramdisk from the network
domih wrote:
Wed Aug 21, 2019 3:25 pm
you are now the proud owner of an H2 that leaves all the Gigabit-challenged PCs in the dust. Welcome to the club!
Hi to the Kama River people :-)
;)
domih wrote:
Wed Aug 21, 2019 3:25 pm
it almost look like the font is not anti-aliased and yes the digits in the charts look deformed.
at work I looked at my own screenshots in another browser and they really look blurry Image
but at home I have a clear image on my computer, I don’t know why so ...
This is how it looks like at my phone. I think somehow application decrypting the quality. I have look for settings in app but didn't find any settings to shut that off . Problebly depend what you used on client side then. ;)Image

Beem my message up with phone.


RomaT
Posts: 226
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 35 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Wed Aug 21, 2019 7:54 pm

I didn’t even know that the picture could be distorted like that - it's generally some kind of mosaic.
but this picture shows you well? She is also a PNG:
viewtopic.php?p=261999#p261999
my phone shows like this (This is a photo of a phone, not a screenshot) :
.
2019-08-21_16-02-35.jpg
2019-08-21_16-02-35.jpg (176.35 KiB) Viewed 1532 times
These users thanked the author RomaT for the post:
Mullcom (Wed Aug 21, 2019 9:34 pm)

User avatar
Mullcom
Posts: 93
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 51 times
Been thanked: 5 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Wed Aug 21, 2019 10:05 pm

RomaT wrote:I didn’t even know that the picture could be distorted like that - it's generally some kind of mosaic.
but this picture shows you well? She is also a PNG:
viewtopic.php?p=261999#p261999
my phone shows like this (This is a photo of a phone, not a screenshot) :
.
2019-08-21_16-02-35.jpg
It's problebly my app. Taptalk that do this. Anyway no big deal. ;) You upload zip file with IMG.


And I like to comment out one thing alså.

To gett 100% BW is nearly impossible. To many thing that Affect. Fist of all is one mbit not one mb it's less then one mb. I think you can find the calculation on that in wiki page I post before.

Coppar and fiber. How long is the cable and there are different protocols to Handel transfer that done to so well like samba for instance. Tcp and handshake and udp can you tweek abit to locate the top of when you get package drop and set up a QnQ right under so you never get packed drops that make stable transfer and you get a overall faster transfer. This is nice to check in the wan side because this can optimize your wan go over what ISP deliver.


Different Hardware are critical to. How good nic card can handel the traffic and so on. How good is the CPU for network and do nic card take care of some load intend of sending it to CPU and back again. You problebly know all this and get my point by now. If you can read and understand what I am writing of course ;)

These are not one Solution for everyone becous we are dynamic ;) and The chain is not stronger than the weakest link. Nice to read all of this info that comes from you.



Beem my message up with phone.


domih
Posts: 119
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Thu Aug 22, 2019 9:35 am

@RomaT, @Mullcom

On the other hand, RomaT's images look fine on my computers and on my iPad, no pixelating, no blur. Q for RomaT: what do you use to make screenshots? Do you post-process the screenshots with another app?

domih
Posts: 119
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Thu Aug 22, 2019 9:37 am

Yeah, my 10+ Gbe Odroid H2 has left the workbench and joined my "amateur living room data center" :D
20190821_171254.jpg
20190821_171254.jpg (175.01 KiB) Viewed 1476 times
These users thanked the author domih for the post:
Mullcom (Thu Aug 22, 2019 2:11 pm)

domih
Posts: 119
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Thu Aug 22, 2019 9:56 am

RomaT wrote:
Wed Aug 21, 2019 3:38 pm
domih wrote:
Wed Aug 21, 2019 3:25 pm
The thing is that if I were to spend 4x or 5.6x the $$$ to get 40Gb/s or 56Gb/s on a hardware that would never reach it, yes it would not make any sense.
When viewed from the cost and other computers, then yes, it’s more attractive to build like you.
I considered from the point of view of the possibility of ODROID-H2.
Yes but in addition what I am telling you is that the Odroid H2 can go further than the 8Gbe you are witnessing. I believe you are indeed measuring the limit of the 10Gbe cards and cables, not the limit of the Odroid H2. To be sure, you have to test the following:

Code: Select all

# On the i5
iperf3 -s --bind <your_i5_10gbe_ip_address>
# On the H2
iperf3 -c --bind <your_h2_10gbe_ip_address> -t 60

# Then

# On the H2
iperf3 -s --bind <your_h2_10gbe_ip_address>
# On the i5
iperf3 -c<your_h2_10gbe_ip_address> --bind <your_i5_10gbe_ip_address>  -t 60
Note: for fun (and schadenfreude about people still stuck in 1Gbe) you can also do this with your regular 1Gbe IP addresses and see the difference.

On my side I respectively get 11.1 Gbe and 14.4 Gbe (see earlier posts) so the H2 does pass the 8Gbe and 10Gbe limits. As I wrote in my last reply: you should be very happy if you get 8Gbe (assuming the 8b10b encoding is used by your 10Gbe cards - which again I don't know whether or not this is the case).

One additional way, you can certify and confirm your results is to plug your two 10Gbe cards into the i5, loop them with the cable and run iperf3 from one card to the other. All logic says that you should anyway see a 8Gbe limit (note: in fact iperf3 should report 16Gbe because it sums the output bandwidth and the input one when looping on the same host, so you'll have to divide by two,)

On my H2, doing so with the two ports of the ConnectX-2, I get about 25.5 Gbe, which not surprisingly is equal to 11.1Gbe + 14.4 Gbe.

If you do NOT get approximately these results, then it becomes interesting and more inquiry will be required.

Note: iperf3 does not give a good idea of what Samba or NFS speed will actually be, but it gives you a very top limit. It's big advantage on the other hand is to be consistent and repeatable (no disk accesses, not caching involved), pure networking. After using it for months now, I ended up using a "divided by two" for projected max speed in Samba or NFS and "divided by two and a half" for a more average expected speed in Samba or NFS (in best case scenario, meaning copying a multi-gig file from SSD to SSD). Then only comes further optimizations you can do with LVM2 caching, I/O buffers or even RAM disk as you did.

Finally, all 10Gbe are not equal (fiber vs copper, lanes, encoding), see https://en.wikipedia.org/wiki/10_Gigabit_Ethernet and http://units.folder101.com/cisco/sem1/N ... coding.htm as a starter.
Last edited by domih on Thu Aug 22, 2019 11:00 am, edited 1 time in total.

domih
Posts: 119
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Thu Aug 22, 2019 10:24 am

@Mullcom,

For the cables, I'm finally settling on the following rules:
- from 1 to 2 or 3 meters: use passive DAC copper (less expensive)
- from 3 to 100+ meters: use active optical (passive copper has bandwidth and quality of signal issues on long length)

Reasons:
- My 2-meter active optical were getting too entangled in my "amateur living room data center" and it is recommended to not bend optical cables under a radius of 17cm.
- In comparison the passive DAC copper are rustic and won't easily bend to small radius.
- Second hand passive DAC copper from 1 to 3 meters are cheap and ubiquitous on eBay. Got several 2-m for $16 a piece (shipping included).
- Above 5 meters, use active optical to maintain bandwidth and quality of signal. Curiously, short length optical are rare on eBay, but long length are plenty.

So these rules are mainly based on practicality and availability on eBay and NOT on the merits of one or the other :-)
Last edited by domih on Thu Aug 22, 2019 10:56 am, edited 2 times in total.

domih
Posts: 119
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Thu Aug 22, 2019 10:34 am

Mullcom wrote:
Tue Aug 20, 2019 12:24 am
I did get my networks card today. ;)
You're definitely a lucky guy, you got both the small and tall brackets. I always had to buy the tall ones separately (+$5 bucks)!

domih
Posts: 119
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Thu Aug 22, 2019 10:41 am

RomaT wrote:
Mon Aug 19, 2019 8:26 pm
I want 126GB/s :D (PCI-E 6.0 16 lanes)
Plus this (see below), all fitting on a board the size of a watch and priced at about $50. We won't get that but our kids or grand-kids will(*) :D

(*) projection based on what we had in the late 70s and what we have today, so imagine in 40 years from now...
.
Screenshot at 2019-08-21 18-31-09.png
Screenshot at 2019-08-21 18-31-09.png (71.53 KiB) Viewed 1459 times

RomaT
Posts: 226
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 35 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Thu Aug 22, 2019 12:21 pm

domih wrote:
Thu Aug 22, 2019 9:35 am
what do you use to make screenshots? Do you post-process the screenshots with another app?
desktop on i5 under Windows 10 OS
ALT+PrintScreen or CTRL+PrintScreen (copy screenshot to system buffer) Then paste (CTRL+V) to Paint there are scaled and saved in the desired file format
domih wrote:
Thu Aug 22, 2019 9:56 am
is to plug your two 10Gbe cards into the i5, loop them with the cable
This is not possible for two reasons (more precisely possible, but it will take a lot of time, therefore there is no desire).
the first on i5 there are no free PCI-E slots (motherboard: ASUS Z390-I) moreover, in the chassis does not fit a second network card
second at the moment, removing a network card with H2 is very time-consuming (disassemble assemble will take all day)
domih wrote:
Thu Aug 22, 2019 9:56 am
you have to test the following:

Code: Select all

# On the i5
iperf3 -s --bind <your_i5_10gbe_ip_address>
# On the H2
iperf3 -c --bind <your_h2_10gbe_ip_address> -t 60

# Then

# On the H2
iperf3 -s --bind <your_h2_10gbe_ip_address>
# On the i5
iperf3 -c<your_h2_10gbe_ip_address> --bind <your_i5_10gbe_ip_address>  -t 60
i5 console on the left (192.168.101.2), H2 console on the right (192.168.101.1)
.
test_iperf3_h2_to_pc.jpg
test_iperf3_h2_to_pc.jpg (349.26 KiB) Viewed 1431 times
.
test_iperf3_pc_to_h2.jpg
test_iperf3_pc_to_h2.jpg (405.24 KiB) Viewed 1431 times
.
test_iperf3_pc_to_h2_2.jpg
test_iperf3_pc_to_h2_2.jpg (350.47 KiB) Viewed 1431 times
These users thanked the author RomaT for the post:
domih (Fri Aug 23, 2019 11:58 am)

User avatar
Mullcom
Posts: 93
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 51 times
Been thanked: 5 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Thu Aug 22, 2019 2:58 pm

seams that max BW with this cards is right below 8 gbit. When i am looking at the test i see some times it drops performance. This can be that the card get the maximum speed and then you get package drops. then network card try a lower speed of package- if you want a stable BW and minimize this drops you can sett up a limit BW so it never get to the top and get this typs of drops. overall it should be in the end faster transfer.

RomaT
Posts: 226
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 35 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Thu Aug 22, 2019 3:25 pm

Maximum value when transferring, which I received when copying files using the SMB protocol 8 Gbits/s
.
test2_h2_from_ramdisk_to_pc.png
test2_h2_from_ramdisk_to_pc.png (12.53 KiB) Viewed 1281 times
.
maximum stable transferring from ramdisk H2 to the network and back 920MB/s ~ 7.2 Gbits/s
.
test_h2_from_ramdisk_to_pc.jpg
test_h2_from_ramdisk_to_pc.jpg (21.68 KiB) Viewed 1279 times
.
array LVM2 unstable transferring, "sinusoidally", from 200 MB/s to 960MB/s (from 1.5 to 7.5 Gbits/s)
.
test5_h2_from_LVM2_to_pc.jpg
test5_h2_from_LVM2_to_pc.jpg (20.92 KiB) Viewed 1281 times
.
it remains to wait for cheap SATA SSD large volumes :D (For example, in 3.5" format by 8...10TB at least 550 MB/s rw speed at the price as for HDD)
These users thanked the author RomaT for the post:
domih (Fri Aug 23, 2019 11:57 am)

domih
Posts: 119
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Fri Aug 23, 2019 11:54 am

RomaT wrote:
Thu Aug 22, 2019 12:21 pm
This is not possible for two reasons (more precisely possible, but it will take a lot of time, therefore there is no desire).
the first on i5 there are no free PCI-E slots (motherboard: ASUS Z390-I) moreover, in the chassis does not fit a second network card
second at the moment, removing a network card with H2 is very time-consuming (disassemble assemble will take all day)
After looking at the pictures, yes don't do it, I concur 100%. In fact, I'm impressed by how you were able to pack so much stuff in such as small box.

Anyway, the tests between the i5 and the H2 are pretty consistent and make sense.

In one direction you top at almost 6Gb (more precisely 5.85Gb) and in the other direction you top at almost 8Gb (more precisely 7.44Gb). All in all this is pretty good.

This also seems to indicate that the encoding is indeed 8b10b. You might want to explore that. Both http://www.lr-link.com/products/LREC6880BT.html and https://www.aquantia.com/products/contr ... on-aqc107/ state that they implement 10GBase-T. In other words they implement IEEE 802.3an-2006 (see https://en.wikipedia.org/wiki/10_Gigabi ... #10GBASE-T). IEEE 802.3an-2006 specifies that 10GBase-T can be encoded 8b10b or 64b66b. See discussion about these encodings there https://www.fmad.io/blog-10g-ethernet-l ... b-66b.html and there https://www.rfwireless-world.com/Termin ... oding.html.

With 8b/10b, your 10Gbs link tops at 10 x 8 / 10 = 8 Gbe.
With 64b/66b, it would top at 10 * 64 / 66 = 9.69 Gbe.

In your shoes, I would contact the tech support of both LR Link and Aquantia asking them:
1) Do they use 8b/10b. As a confirmation.
2) Do they also implement 64b/10b? On Windows? On Linux?
3) If so, what are the settings for switching to 64b/66b on each platform.

Then you should get better than 8Gbe max.

Finally, regarding I/O buffers and others vexing details like these, have a read of https://www.kernel.org/doc/ols/2009/ols ... 69-184.pdf. It's probable that spreading the smp_affinity through the IRQs used by your card should bring some more (tiny?) optimizations. Bigger buffers should help too, Then you do not need to pump up the numbers with a RAM disk.
These users thanked the author domih for the post:
RomaT (Fri Aug 23, 2019 12:16 pm)

domih
Posts: 119
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Fri Aug 23, 2019 12:06 pm

RomaT wrote:
Thu Aug 22, 2019 12:21 pm
domih wrote:
Thu Aug 22, 2019 9:35 am
what do you use to make screenshots? Do you post-process the screenshots with another app?
desktop on i5 under Windows 10 OS
ALT+PrintScreen or CTRL+PrintScreen (copy screenshot to system buffer) Then paste (CTRL+V) to Paint there are scaled and saved in the desired file format
For taking screenshots, you may want to try https://www.gadwin.com/printscreen/. More convenient and never had any issue of resolution, pixelating or blur with anybody with it. Instead of Paint, you may want to try https://www.getpaint.net/ (almost as good as Gimp).

Both are free.

domih
Posts: 119
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 38 times
Been thanked: 27 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Fri Aug 23, 2019 12:29 pm

@RomaT: you might also want to check and read about the cable. Cat 7 or Cat 6a should carry a good 10Gbe signal over 100 meters. However, there are cables and there are cables. The devil is in the details: shielded or not shielded, crosstalk, frequency, and so on. The card probably auto-senses these to decide what to do at the physical layer. Issue: it's kind of difficult to go up the information chain from the product ID to the real manufacturer and find the actual testing documents which they are supposed to have to justify the claim their cable is really Cat 6a or 7. Have fun!

Signing off. Going to bed. Have a good day.

RomaT
Posts: 226
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 35 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Fri Aug 23, 2019 12:43 pm

Category 7 Shielded Cable 1.5m
.
cable.jpg
cable.jpg (61.15 KiB) Viewed 1244 times

User avatar
Mullcom
Posts: 93
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 51 times
Been thanked: 5 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Sat Aug 24, 2019 12:50 am

Did get my pci extension today. Feels like it's really God quality.ImageImageImageImageImage

Beem my message up with phone.


User avatar
Mullcom
Posts: 93
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 51 times
Been thanked: 5 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Sat Aug 24, 2019 12:57 am

Also I got this. So soon I can start to build my H2 and fixed in my 19"rack. Image

Beem my message up with phone.


User avatar
mad_ady
Posts: 6558
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 170 times
Been thanked: 129 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by mad_ady » Sat Aug 24, 2019 2:07 am

Nice assorted collection of risers. Care to post a link to its source?

User avatar
Mullcom
Posts: 93
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 51 times
Been thanked: 5 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Sat Aug 24, 2019 2:14 am

mad_ady wrote:Nice assorted collection of risers. Care to post a link to its source?
It is this one on eBay

I still waiting for this two
Link 1
Link 2

Beem my message up with phone.

These users thanked the author Mullcom for the post (total 2):
mad_ady (Sat Aug 24, 2019 3:54 am) • rooted (Thu Aug 29, 2019 3:31 pm)

RomaT
Posts: 226
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 35 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Sun Aug 25, 2019 12:38 am

Mullcom wrote:
Sat Aug 24, 2019 12:50 am
I did get my networks card today.
Did get my pci extension today.
Now it's your turn to tell and show the network ;)
These users thanked the author RomaT for the post:
Mullcom (Sun Aug 25, 2019 12:46 am)

User avatar
Mullcom
Posts: 93
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 51 times
Been thanked: 5 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Sun Aug 25, 2019 12:50 am

RomaT wrote:
Mullcom wrote:
Sat Aug 24, 2019 12:50 am
I did get my networks card today.
Did get my pci extension today.
Now it's your turn to tell and show the network ;)
I am sorry. But it going to take a while. My NAS server is the next projekt after my server room.

Right now is NAS picked up yo be upgraded with new hardware and water-cooling.

Beem my message up with phone.


brad
Posts: 965
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 N1 N2 H2 (64 bit ftw)
Location: Australia
Has thanked: 16 times
Been thanked: 32 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by brad » Sun Aug 25, 2019 8:54 am

@RomaT

as @domih mentioned I think you are seeing the upper limits of the card and its 8b/10b encoding but there may be some things you can try to even out both directions and stabilise the max throughput.

- Significant changes to the linux kernel supplied Aquantia drivers over the past 12 months. Which kernel version are you using on the H2? Some months ago in mainline default ring buffer and frame sizes were increased and there have been many other bug fixes for flow control. See https://github.com/torvalds/linux/commi ... a/atlantic for history.

- Aquantia also provide a slightly different driver on their website but I think most of this is already incorporated into mainline but you could try it. Same goes for the windows side I would assume.

- Check the interrupt usage on the H2 side "cat /proc/interrupts | grep eth" to see if they are spread across the cores (I think they should be) as both hardware IRQ's and the cores the drivers runs on should both have cpu affinity for all cores.

- The Aquantia drivers have some configuration options you can try in later driver releases. An interesting runtime configuration change you can try is to disable Interrupt coalescing. The idea is that by default it will try to bundle ethernet frames to take some load off the interrupt subsystem and save power but this may impact latency and possibly throughput if the card is struggling. If the cpu's handling the interrupts are struggling to keep up it might have the opposite effect. You may see benifits here over samba / NFS where the round trip latency the network can have a big impact when transferring small files and we must do lots of smaller round trips for metadata, file locking, and synchronous mode file access or writes.

Code: Select all

ethtool -C <ethX> tx-usecs 0 rx-usecs 0 tx-max-frames 1 tx-max-frames 1
- Aquantia drivers have some other more advanced changes but for these you would need to recompile the driver after modifying them (the defaults are probably the best but you never know). These options are for the latest release of Linux kernel

See: https://github.com/torvalds/linux/blob/ ... lantic.txt
AQ_CFG_RX_PAGEORDER
----------------------------------------
Default value: 0
RX page order override. Thats a power of 2 number of RX pages allocated for
each descriptor. Received descriptor size is still limited by AQ_CFG_RX_FRAME_MAX.
Increasing pageorder makes page reuse better (actual on iommu enabled systems).

AQ_CFG_RX_REFILL_THRES
----------------------------------------
Default value: 32
RX refill threshold. RX path will not refill freed descriptors until the
specified number of free descriptors is observed. Larger values may help
better page reuse but may lead to packet drops as well.

AQ_CFG_VECS_DEF
------------------------------------------------------------
Number of queues
Valid Range: 0 - 8 (up to AQ_CFG_VECS_MAX)
Default value: 8
Notice this value will be capped by the number of cores available on the system.

AQ_CFG_IS_RSS_DEF
------------------------------------------------------------
Enable/disable Receive Side Scaling

This feature allows the adapter to distribute receive processing
across multiple CPU-cores and to prevent from overloading a single CPU core.

Valid values
0 - disabled
1 - enabled

Default value: 1

AQ_CFG_NUM_RSS_QUEUES_DEF
------------------------------------------------------------
Number of queues for Receive Side Scaling
Valid Range: 0 - 8 (up to AQ_CFG_VECS_DEF)

Default value: AQ_CFG_VECS_DEF

AQ_CFG_IS_LRO_DEF
------------------------------------------------------------
Enable/disable Large Receive Offload

This offload enables the adapter to coalesce multiple TCP segments and indicate
them as a single coalesced unit to the OS networking subsystem.
The system consumes less energy but it also introduces more latency in packets processing.

Valid values
0 - disabled
1 - enabled

Default value: 1

AQ_CFG_TX_CLEAN_BUDGET
----------------------------------------
Maximum descriptors to cleanup on TX at once.
Default value: 256

After the aq_cfg.h file changed the driver must be rebuilt to take effect.
These users thanked the author brad for the post:
RomaT (Sun Aug 25, 2019 10:32 am)

brad
Posts: 965
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 N1 N2 H2 (64 bit ftw)
Location: Australia
Has thanked: 16 times
Been thanked: 32 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by brad » Sun Aug 25, 2019 9:28 am

Additionally can I ask how you created you MBR and partitioned your LVM physical volumes on the H2?

Was it done over USB or direct SATA on the H2 or another system and which OS?

Ensuring you have the partitions aligned correctly with the physical disk (especially for spinning disk) can be very important. Sometimes USB connected drives don't report the correct sizing like direct SATA connected drives can. If parted is used on sata connected disk you should always get the correct alignments for each disk when they are created.

brad
Posts: 965
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 N1 N2 H2 (64 bit ftw)
Location: Australia
Has thanked: 16 times
Been thanked: 32 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by brad » Sun Aug 25, 2019 9:36 am

brad wrote:
Sun Aug 25, 2019 9:28 am
Additionally can I ask how you created you MBR and partitioned your LVM physical volumes on the H2?

Was it done over USB or direct SATA on the H2 or another system and which OS?

Ensuring you have the partitions aligned correctly with the physical disk (especially for spinning disk) can be very important. Sometimes USB connected drives don't report the correct sizing like direct SATA connected drives can. If parted is used on sata connected disk you should always get the correct alignments for each disk when they are created.
PS: When you are transferring "files" from a single disk to network interface over a protocol you can technically break the speed limit of the SATA port bandwidth using disk compression and or block level deduplication. Compression means less data is xfered to the disk but we need CPU power to compress/decompress on the fly. Block level deduplication means we only need to xfer unique blocks over the sata port but we usually need lots of RAM to hold our metadata and hash tables to get good performance using deduplication.

RomaT
Posts: 226
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 35 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Sun Aug 25, 2019 10:51 am

brad wrote:
Sun Aug 25, 2019 9:28 am
Additionally can I ask how you created you MBR and partitioned your LVM physical volumes on the H2?

Was it done over USB or direct SATA on the H2 or another system and which OS?
a) All discs have been converted from MBR to the GPT
b) All drives worked only on SATA, never connected to USB-SATA adapters
c) LVM2 was created on H2, All drives were installed in the H2 clean, without partitions, connected to SATA ports - All actions were performed on H2
d) Ubuntu 18.04.02 LTS amd64 operating system, kernel 4.18.0-25-generic x86_64
e) RAM 32GB

Post Reply

Return to “Hardware and peripherals”

Who is online

Users browsing this forum: No registered users and 0 guests