Add-on to H2 - M2 to pciE slot

domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Sun Jul 28, 2019 1:40 pm

RomaT wrote:
Fri Jul 26, 2019 1:54 pm
I think this is normal, it’s high time for everyone to move to 10GBe and more ...
IMHO, H2 needed to be done with two PCIe slots (m.2), one for 4 lanes, the second for two lanes,
or two PCIe lanes for integrated controller 4-port SATA, instead of two 1GBE,
it seems to me a mistake to use 2 lanes PCIe for the two 1GbE,
if anyone needs 1GBE, they can use USB adapters 1GBE ...
In addition, the processor has 8 ports USB, they could bring at least a GPIO Slot.
Keep cool :D The J4105 has only 6 PCIe lanes so that translates into compromises when you do implement a board. I guess Hard Kernel wanted to cover as much as territory as possible and having 2 x 1Gbe NICs enables users who want to implement all the router solutions they want. I would also venture that Hard Kernel with its first venture in x86 was prudent in providing as many solutions as possible. Hard Kernel is probably discovering that the x86 crowd is not exactly the same as the ARM crowd. The thing is that once the product is out the fearless kiddies (me for instance) start using it in ways that were not foreseen? But Hark Kernel started it itself with the multiple articles they published in the Wiki or the magazine where the NVMe was adapted into PCIe x4. So they are the guilty ones here :D

That's the reason I'm pseudo-evangelizing about the AMD v1605b, it is more powerful than the J4015 and brings 4C/8T, 16 PCIe lanes, PCIe 3 instead of PCIe 2, with a 10 -> 25W TDP, which is still reasonable for home server application. And I know for a fact (because I built one) that it enables you building a NAS with 20+ Gbe networking. On the Intel side, browsing https://ark.intel.com/content/www/us/en ... Processors, it is difficult to best the J4105 without jumping to higher TDP. As I wrote earlier, the AMD v1605b would allow Hard Kernel to design an "H3" with everything the H2 has plus 1Gbe or 10Gbe plus a PCIe x16 slot (x8 signal) opening the board to many PCIe 3 cards.

In another thread, Hard Kernel responded that 10Gbe infrastructure is still non-existing at home or in small offices. True, but IMHO short-sighted. I have been looking for 10Gbe solutions for quite some time. In 2018 and 2019, we finally see motherboards or appliances bringing 10Gbe NICs. Clearly the floodgates of high pricing have been leaking for 2 years. I expect 10Gbe to explode on the prosumer market in 2020-21. So the question for OEM is simple: will we have products to sell to answer the new need of the savage populace? The same question exists for the switch from Wireless AC to Wireless AX.

Now, I do NOT know how much the AMD v1605b costs to an OEM. Further more, taking Hard Kernel shoes, if once the dust has settled, the H2 x86 sales is only a fraction of the N2, XU4, HC1, HC2, etc sales the debate of making or not another x86 board is indeed a very valid one. H2 seems to be recurrently out of stock, so that sounds good. The question is can Hard Kernel developed x86 and ARM series in parallel? So far, apparently so.
Last edited by domih on Sun Jul 28, 2019 9:37 pm, edited 1 time in total.

domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Sun Jul 28, 2019 2:01 pm

More pictures from the last tests.
Attachments
005.jpg
Let's rotate the H2 from upside down to on the side so that the CPU fan can blow off steam.
005.jpg (249.13 KiB) Viewed 1711 times
004.jpg
When in the mood, the network passes the 14Gbe mark.
004.jpg (241.47 KiB) Viewed 1711 times
003.jpg
Then the switching slows down and down. During the 30 minute test, the "active" I/O core stayed the same for more than a minute.
003.jpg (331.41 KiB) Viewed 1711 times
002.jpg
Then core switching slows down.
002.jpg (326.46 KiB) Viewed 1711 times
001.jpg
At the beginning of test, the switch from core to core happens a lot.
001.jpg (249.53 KiB) Viewed 1711 times

domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Sun Jul 28, 2019 2:24 pm

More pictures from the last tests.

The switch is a Mellanox IS5023 that provides 40Gbps on 18 ports. You can get one for around $100 on eBay. Built like a truck. Unmanaged so no configuration headaches. Just plug 'n' play. The network management (/etc/init.d/opensmd start) comes from at least one PC connected to it.

Its big brother providing up to 56Gbps on 18 ports is, among others, the SX6015 which can be found for around $120 on eBay. Also built like a truck. Unmanaged too, blah blah blah.

In both cases, if you do not have a garage, a basement or an attic, know that these are very noisy beasts. If to be used inside the home, you will want to customize and replace the fans. Easy on the IS5023, more challenging on the SX6015.

Cards are from $50 to $90 (or more) depending on the vendor. Optical cables are $50 or higher depending on length and vendor. Cables for 40Gbps and 56Gbps are not the same.

It pays to follow eBay over weeks, good deals appear and disappear on a recurrent basis.
Attachments
007.jpg
I replaced the 4 15K RPM 40x25mm fans with 80x25mm slow speed silent Noctua ones. The top cover is NOT put back. The red light on the left is the switch saying there is something wrong with the fans (duh!) Works fine. In addition, the switch while unmanaged therefore just plug and play is not entirely stupid: it has thermal control. With this setting, I never witnessed it anyway. I guess that after running full throttle on the 18 port in a data center rack, the switch is happy in its retirement role.
007.jpg (240.96 KiB) Viewed 1707 times
006.jpg
The InfiniBand switch in my living room. Some customizing and creative furniture building makes it a very silent one.
006.jpg (251.3 KiB) Viewed 1707 times
These users thanked the author domih for the post:
Mullcom (Mon Aug 05, 2019 2:10 am)

User avatar
Mullcom
Posts: 87
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 47 times
Been thanked: 3 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Sun Jul 28, 2019 6:55 pm

Have now update some info in main post. Thx for all info.

Skickat från min SM-G955F via Tapatalk


User avatar
Mullcom
Posts: 87
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 47 times
Been thanked: 3 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Sun Jul 28, 2019 7:47 pm

I am getting my "9u 19rack soon". And then i can start to build my router with H2 and implement that into the rack. Going fore a clean design.

I am going to find a proper fiber networks cad so it can deliver 10gb to switch.

For storage i am going to build a 4u case and install freenas into a bigger rack in the server room. Problebly put in a fiber card to that one also. I have not so much knowledge of fiber right now so need to dig in to that so everything is supported.

I am also looking in to openstack to see if this can be a good ground for all server power. Combining servers so i can remove one or have redundancy.

For power delivery i have some ideas
1. Need to have a power meeter before all my equipment so i cam see how much power everything drain.
2. Find a good setup to deliver power to IoT equipment like raspberry pi4.

IMG down below is my server room i am building now.
Image

Skickat från min SM-G955F via Tapatalk


domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Sun Jul 28, 2019 10:11 pm

@Mullcom

After research, the Mellanox MT25408 I'm currently using is not even a ConnectX-2 but a ConnectX first generation. Phew. I could not find the MT25408 on eBay anymore, on the other hand the ConnectX-2 in its various versions is plenty on eBay. It is also a PCIe 2 card so it should be OK on the H2. I ordered the 20Gbps ($10) and a 40Gbps ($20) ConnectX-2 card. For more information about ConnectX-2, see https://www.mellanox.com/related-docs/p ... I_Card.pdf. I'll try them on the H2 when I receive them.

Note: ConnectX3 or ConnectX3-pro are preferable and still supported by Mellanox. For your PCs, use a ConnectX3. With the H2, it is OK to use ConnectX or ConnectX-2.

domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Mon Jul 29, 2019 10:44 am

Samba results are there.

<<...Obviously, we won't reach these speeds during real testing. I'll consider myself lucky if I get half or above of it...>>

Two disks are mounted on the H2:

Code: Select all

domih@h2a:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
.../...
/dev/sda1       938G   77M  891G   1% /mnt/hda  <-- 1TB 860 Pro
/dev/sdb1       9.1T  4.3G  8.6T   1% /mnt/hdb  <-- 10TB WD Red
Client: Windows 10 running on a "old" i7-3770K PC.

See table below.
---
Screen Shot 07-28-19 at 06.08 PM.PNG
Screen Shot 07-28-19 at 06.08 PM.PNG (44.8 KiB) Viewed 1634 times
Same thing with the 10TB hard disk.
---
Screen Shot 07-28-19 at 06.16 PM 001.PNG
Screen Shot 07-28-19 at 06.16 PM 001.PNG (33.13 KiB) Viewed 1634 times
Writing and Reading a 22.1GB folder with small and big files
---
Screen Shot 07-28-19 at 06.21 PM.PNG
Screen Shot 07-28-19 at 06.21 PM.PNG (34.25 KiB) Viewed 1634 times
CONCLUSIONS
- With Samba you never get the same result!
- As expected (read feared) disk I/O and network I/O compete for the CPU.
- With Gigabit, the network (read or write) tops at 113MB/s and therefore does not saturate the disk I/O. Anything I tried in gigabit was sustaining 111-113 MB/s.
- With IP over IB, the network bandwidth saturates the CPU and disk I/O, speed varied from 150 to 700+ MB/s between tests or even during a test.
- From 5-- to 7++ times the speed of 1 Gbe is good enough to me given the involved hardware.
- It will be very interesting to compare with "regular" 10Gbe cards which are entirely different beasts compared to the network card I'm using. I do not even exclude better results from the 10Gbe cards.
These users thanked the author domih for the post:
Mullcom (Mon Jul 29, 2019 5:16 pm)

User avatar
Mullcom
Posts: 87
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 47 times
Been thanked: 3 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Mon Jul 29, 2019 5:03 pm

Found a good blog post that describes a bit copy and analysis file share.

https://blogs.technet.microsoft.com/jos ... o-instead/

Hope this is a good read for you all.


This is a good tool for testing disk speed in Windows. Easy and simpel and do what it should do.
https://hdd.userbenchmark.com/Software

Skickat från min SM-G955F via Tapatalk


User avatar
Mullcom
Posts: 87
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 47 times
Been thanked: 3 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Mon Jul 29, 2019 5:23 pm

domih wrote:Samba results are there.

<<...Obviously, we won't reach these speeds during real testing. I'll consider myself lucky if I get half or above of it...>>

Two disks are mounted on the H2:

Code: Select all

domih@h2a:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
.../...
/dev/sda1       938G   77M  891G   1% /mnt/hda  <-- 1TB 860 Pro
/dev/sdb1       9.1T  4.3G  8.6T   1% /mnt/hdb  <-- 10TB WD Red
Client: Windows 10 running on a "old" i7-3770K PC.

See table below.
---
Screen Shot 07-28-19 at 06.08 PM.PNG
Same thing with the 10TB hard disk.
---
Screen Shot 07-28-19 at 06.16 PM 001.PNG
Writing and Reading a 22.1GB folder with small and big files
---
Screen Shot 07-28-19 at 06.21 PM.PNG
CONCLUSIONS
- With Samba you never get the same result!
- As expected (read feared) disk I/O and network I/O compete for the CPU.
- With Gigabit, the network (read or write) tops at 113MB/s and therefore does not saturate the disk I/O. Anything I tried in gigabit was sustaining 111-113 MB/s.
- With IP over IB, the network bandwidth saturates the CPU and disk I/O, speed varied from 150 to 700+ MB/s between tests or even during a test.
- From 5-- to 7++ times the speed of 1 Gbe is good enough to me given the involved hardware.
- It will be very interesting to compare with "regular" 10Gbe cards which are entirely different beasts compared to the network card I'm using. I do not even exclude better results from the 10Gbe cards.
As expected (read feared) disk I/O and network I/O compete for the CPU.

1. What do you meant with compete?
Do you mean there less BW in disk so speed of network can't get full potential or the other way?


What i can see is H2 perform really good. To get disk working at higher BW i think raid need to be done or using SSDs.

Thx for all info here.



Skickat från min SM-G955F via Tapatalk


domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Tue Jul 30, 2019 3:47 am

Yes, don't get me wrong, the performance of the H2 in these tests is definitely top-notch, especially from a system that sucks in less than 50W when loaded and close to 25W idle!

The only RAID types I still use is 1 or 10. The former with two disks, the latter with at least eight. I won't use RAID 0 on SSD, sounds to me like a Final Destination Highway to Hell story.

Yes, due to local "limited" resources, you can't reach the full network speed. In some of my "modern" PCs the network card is inserted in front of the video card (Mellanox in slot #1, GC in slot #3)... That's one reason why server hardware is more expensive compared to PC for the populace: more oomph along the connect chain and storage.

I think I still have a few venues to top the current results:
- The ConnectX-2 will probably be more efficient than the current ConnectX (grabbing a few percent?)
- Stop using IPoIB and switch from Samba(*) to NFS over RDMA (thus getting the CPU totally out of the networking operations) (expecting double-digit percentage difference here)
- Run Ubuntu Server instead of Ubuntu Mate (mostly to get rid of non-necessary services) (grabbing a few percent?)

(*) My use of Samba is going to drastically drop anyway. With the upcoming death of Windows 7, I have been phasing out Windows at home, don't feel much about Windows 10. Most of my machines now run CentOS or Ubuntu.

domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Tue Jul 30, 2019 8:47 am

All components together. See picture below.

PCIe bus, network card, its fan, 2 x SATA disks, a small fan board, an additional "chassis" fan and finally the H2 itself all powered 12V by the 3rd party PSU. Works AOK.

The CZH Labs ATX breakout card will be replaced by this one: https://www.aliexpress.com/item/32962553060.html.

I'll go with an open case solution. I ordered two of this: https://www.aliexpress.com/item/33032050394.html to make a 2-level open case with some customization. Planning to have the PSU, breakout board and fan board on the lower level, the H2 and the network card on the upper level. The hard disk and SSD will probably land in the lower level with a mezzanine made out of the extra acrylic plate. We'll see. I'll get the pieces in about three weeks.

Meanwhile:
- Switch to Ubuntu Server.
- Try the ConnectX-2 next week.
- Investigate and test NFS-RDMA.
- Find out the correct incantations to put in the LVM conf file so that I don't lose the LVM2 tiering config at power off :-)
Attachments
20190729_161856.jpg
20190729_161856.jpg (519.28 KiB) Viewed 1557 times

User avatar
odroid
Site Admin
Posts: 31852
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 89 times
Been thanked: 255 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by odroid » Tue Jul 30, 2019 9:49 am

@domih,
Nice picture.
It's time to write an article about 10000Mbps Ethernet in the embedded world. ;)
Please consider writing an article for our Magazine. If it is okay, our chief editor Rob will contact you soon.

domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Wed Jul 31, 2019 12:32 pm

odroid wrote:
Tue Jul 30, 2019 9:49 am
@domih,
Nice picture.
It's time to write an article about 10000Mbps Ethernet in the embedded world. ;)
Please consider writing an article for our Magazine. If it is okay, our chief editor Rob will contact you soon.
Yes, Rob can contact me. However:

1) I'm not done. I have extra things I want to test.

2) I think it is very wise to see what the other members (i.e. RomaT) get with a normal end-user 10 Gbe.

3) I really do not think that talking about 10000Mbps Ethernet is the right choice of words. We don't get 10000Mbps at normal usage level. We get something like less than 5 times up to plus 7 times faster than Gigabit. (as of this writing). The > 10GBe is only with iperf3 which definitely does not represent normal use and is like a pipe-dream.

4) I want to try the ConnectX-2.

5) I finally solved my problem with the 56Gbps switch. It happens I landed with a good vendor on eBay. He replaced the switch and I bought a 2nd one for plan B in the future. Results:

- InfiniBand comes with 40Gbps physical and uses 8b10b encoding, so the max pure InfiniBand speed is 32Gbps. It called QDR. With moderns PC it translates to 28.5 to 31 GBe with IP over IB with iPerf3 with translates into 2GB/s throughput while copying large files over Samba (20 times Gigabit speed). Going down 500MB to 750MB with the H2.

- InfiniBand also comes with 56Gbps physical using 8b10b encoding too, so the max pure InfiniBand speed is 40Gbps. It's called FDR10. With moderns PC it translates to 29.9 to 38 GBe with IP over IB with iPerf3, just tested it yesterday. I have not testing Samba yet. I have not tested the H2 yet (coz' the card I'm using right now does not support FDR10.) I'm waiting to see if a ConnectX-2 supports FDR10. I don't think so, but hey...

- Mellanox also had a proprietary 56Gbps physical with a 64b/66b encoding, ramping up to max pure InfiniBand speed is 54Gbps. It's called FDR. I don't have card supporting FDR (they only support FDR10). I ordered a ConnectX-3 Pro ($90) to try. If it's works, I'll buy more.

This is why I'm really interested in finding out why the ConnectX-3 do not load with the H2. Tools like lspci, dmesg, etc do not tell me anything. So, please look at the thread "Ignorant Bastard in Need of Pointers" so that I can try to at least find out why these cards do not load. They are said to be able to work on PCI 2 and I indeed successfully tried them on old PCs PCI 2 x8 and x4. With a ConnectX-3 on the H2 I can push to FDR10. If the ConnectX-3 Pro supports FDR and works on the H2 I can push to pure 54Gbps, meaning probably ~48Gbe with IP over IB.

6) Finally, there is an interesting review there: https://www.servethehome.com/freenas-mi ... ays-10gbe/. It looks like my numbers beat their numbers (on page 2). They used the obsolete Intel NASPT to get them. Obsolete or not, I intend to try it next week-end. In addition it looks like to get these numbers they used an 8-disk ZFS RAID plus SSD cache. I'm getting my numbers with one SSD or one WD Red 10TB :-) Stay tuned.

User avatar
mad_ady
Posts: 6401
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 150 times
Been thanked: 109 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by mad_ady » Wed Jul 31, 2019 1:12 pm

If you have enough ram you can create a ramdisk - to get away from disk limitations.
Anyway - even pure network speed has its advantages (e.g. as traffic generator/stress tester).

RomaT
Posts: 218
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 34 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Wed Jul 31, 2019 4:23 pm

According to PCI-E v2 specification, bandwidth on four lanes 16 Gbps (2GBytes/s)
Therefore, I think the 10Gbit/s (1.25 GBytes/s) network card is optimal.
I think I'll take LR-LINK 6880BT on Aquantia AQC107
small size, 4 lanes, PCI-E v2 and UEFI support, rj45 - no transceivers required or DAC

P.s.: PCI-E v3 specification, bandwidth on four lanes 31.5 Gbps (3.94 GBytes/s), on eight lanes 63 Gbps (7.88 GBytes/s)
Last edited by RomaT on Wed Jul 31, 2019 5:59 pm, edited 1 time in total.

User avatar
Mullcom
Posts: 87
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 47 times
Been thanked: 3 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Wed Jul 31, 2019 5:57 pm

For the article it's should be nice to read about the Support and performance for FreeBSD, Linux and Windows. And finding out how well it's working with popular open source application like Pfsense and Freenas.

I think also there are advancement home users that want more 10g network kommunikations.
Hard to finding bords like that setup.

A bord like this is nearly perfect.
But i have some suggestions ;)

2* slot ddr
2*m2 slots. One for fast SSD and One for extensions for pciX.
1*AMD low power CPU with many cors and more pciX lains.
Less USB ports.
And some way add sata3 ports as many CPU can handel


With a bord like this you going to have the power to either make a small NAS or a firewall or make a cluster with binding many bords together and use them for virtualisation.


Skickat från min SM-G955F via Tapatalk


RomaT
Posts: 218
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 34 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Wed Jul 31, 2019 6:07 pm

Mullcom wrote:
Wed Jul 31, 2019 5:57 pm
home users that want more 10g network kommunikations.
more than 10 Gbit/s is unlikely to happen, for this you need a different processor, with DMI3.0 or PCI-E 3.0
These users thanked the author RomaT for the post:
Mullcom (Wed Jul 31, 2019 6:21 pm)

User avatar
Mullcom
Posts: 87
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 47 times
Been thanked: 3 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Wed Jul 31, 2019 6:24 pm

RomaT wrote:
Mullcom wrote:
Wed Jul 31, 2019 5:57 pm
home users that want more 10g network kommunikations.
more than 10 Gbit/s is unlikely to happen, for this you need a different processor, with DMI3.0 or PCI-E 3.0,
as a close variant - miniITX motherboard Asus H310T with processor G5500 ...
I didn't meant above 10gbit. I meant more of 10gbit ports. Sorry my English is a bit bad.

Skickat från min SM-G955F via Tapatalk


RomaT
Posts: 218
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 34 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Wed Jul 31, 2019 7:09 pm

With h310t I was mistaken there DMI2.0 and PCI-E v 2.0 chipset, from their post I deleted...
on the other hand, it is not clear from where the PCI-E bus goes to M2, from the chipset or directly from the processor
But if you're interested, I can find out, because I have this board with this processor :D
I don’t know other motherboards with power from 12 to 19V and 170/270/370 chipset (with DMI3.0 and PCI-E 3.0)
These users thanked the author RomaT for the post:
Mullcom (Wed Jul 31, 2019 7:42 pm)

User avatar
Mullcom
Posts: 87
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 47 times
Been thanked: 3 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Wed Jul 31, 2019 7:47 pm

RomaT wrote:With h310t I was mistaken there DMI2.0 and PCI-E v 2.0 chipset, from their post I deleted...
on the other hand, it is not clear from where the PCI-E bus goes to M2, from the chipset or directly from the processor
But if you're interested, I can find out, because I have this board with this processor :D
I don’t know other motherboards with power from 12 to 19V and 170/270/370 chipset (with DMI3.0 and PCI-E 3.0)
It's okej. You do not need to. I going for H2 at the moment and see how far it can take me.

I have install Pfsense on it and and later find a pciX Network card to add some more ports.

Thx again.

Skickat från min SM-G955F via Tapatalk


brad
Posts: 963
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 N1 N2 H2 (64 bit ftw)
Location: Australia
Has thanked: 15 times
Been thanked: 31 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by brad » Fri Aug 02, 2019 4:49 pm

Mullcom wrote:
Wed Jul 31, 2019 7:47 pm
RomaT wrote:With h310t I was mistaken there DMI2.0 and PCI-E v 2.0 chipset, from their post I deleted...
on the other hand, it is not clear from where the PCI-E bus goes to M2, from the chipset or directly from the processor
But if you're interested, I can find out, because I have this board with this processor :D
I don’t know other motherboards with power from 12 to 19V and 170/270/370 chipset (with DMI3.0 and PCI-E 3.0)
It's okej. You do not need to. I going for H2 at the moment and see how far it can take me.

I have install Pfsense on it and and later find a pciX Network card to add some more ports.

Thx again.

Skickat från min SM-G955F via Tapatalk
Iv'e seen some 4 or 6 port 1G pci cards. I want to try this myself and see how well works as a router with VPN / crypto. Should be able to use emmc for the OS storage.

I've now been using Windows 10 on the H2 with a GPU card for a month or so now as my netflix, yotube & gaming machine. Network performance on the realtek PHY's is good and stable since I upgraded realtek drivers from their site rather than the windows default.

RomaT
Posts: 218
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 34 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Fri Aug 02, 2019 10:46 pm

domih wrote:
Tue Jul 30, 2019 8:47 am
- Find out the correct incantations to put in the LVM conf file so that I don't lose the LVM2 tiering config at power off :-)
Could you write a step-by-step instruction?
As I understand it, you made SSD as a cache for HDD ?
I've already ordered two network cards (10Gbe on AQC107 chip), one for H2, the second for the other PC
Upon their arrival, I would like to be ready to set everything up...

User avatar
Mullcom
Posts: 87
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 47 times
Been thanked: 3 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Fri Aug 02, 2019 11:41 pm

RomaT wrote:
domih wrote:
Tue Jul 30, 2019 8:47 am
- Find out the correct incantations to put in the LVM conf file so that I don't lose the LVM2 tiering config at power off :-)
Could you write a step-by-step instruction?
As I understand it, you made SSD as a cache for HDD ?
I've already ordered two network cards (10Gbe on AQC107 chip), one for H2, the second for the other PC
Upon their arrival, I would like to be ready to set everything up...
How 2 should be helpful. I am going to buy one to soon.

Skickat från min SM-G955F via Tapatalk


User avatar
Mullcom
Posts: 87
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 47 times
Been thanked: 3 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Fri Aug 02, 2019 11:44 pm

brad wrote:
Mullcom wrote:
Wed Jul 31, 2019 7:47 pm
RomaT wrote:With h310t I was mistaken there DMI2.0 and PCI-E v 2.0 chipset, from their post I deleted...
on the other hand, it is not clear from where the PCI-E bus goes to M2, from the chipset or directly from the processor
But if you're interested, I can find out, because I have this board with this processor :D
I don’t know other motherboards with power from 12 to 19V and 170/270/370 chipset (with DMI3.0 and PCI-E 3.0)
It's okej. You do not need to. I going for H2 at the moment and see how far it can take me.

I have install Pfsense on it and and later find a pciX Network card to add some more ports.

Thx again.

Skickat från min SM-G955F via Tapatalk
Iv'e seen some 4 or 6 port 1G pci cards. I want to try this myself and see how well works as a router with VPN / crypto. Should be able to use emmc for the OS storage.

I've now been using Windows 10 on the H2 with a GPU card for a month or so now as my netflix, yotube & gaming machine. Network performance on the realtek PHY's is good and stable since I upgraded realtek drivers from their site rather than the windows default.
I can confirm that installing Pfsense on emmc working. Didn't have any problem.

The hard part is the extension and separate nic added later how well this is supported on Pfsense.



Skickat från min SM-G955F via Tapatalk


User avatar
Mullcom
Posts: 87
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 47 times
Been thanked: 3 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Sat Aug 03, 2019 11:06 pm

https://www.onestopsystems.com/expansion-kits

Interesting stuff that i found out. Problebly not cheap

Skickat från min SM-G955F via Tapatalk


domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Sun Aug 04, 2019 1:31 pm

For those interested: 2.5Gbe via USB 3. See discussion at: https://forum.radxa.com/t/2-5g-ethernet ... erver/1590. Product at: https://cpc.farnell.com/pro-signal/psg9 ... dp/CS32425 or https://www.ebay.com/itm/USB-3-0-to-2-5 ... SwA3lc~410. If it works for an RK3399, it should go easy for a Celeron.

domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Sun Aug 04, 2019 1:40 pm

Received and tested a Mellanox ConnectX-2: works AOK on the H2. In fact the card is physically smaller, will fit better in my open case, so it is now my default card. Same performance at the ConnectX first generation. For those interested, product brief is there: http://www.mellanox.com/related-docs/pr ... I_Card.pdf. Links to manuals, firmware, etc are there: http://www.mellanox.com/page/products_d ... ectx_2_vpi. These cards are VPI: it means they can be used as InfiniBand NICs or Ethernet NICs or both at the same time for dual ports cards. These cards are no longer on sale but you can easily find them on eBay at prices starting around $20. For PCs having PCIe 3, use ConnectX-3 instead.

domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Sun Aug 04, 2019 1:50 pm

The last Ubuntu 18.04 update bringing kernel 5.0.0-23 breaks the Mellanox OFED stack 4.6-1.0.1.1. The solution is to switch to what Mellanox calls the "inbox" stack and drivers, in other words, the packages coming from the APT repositories. Process:

Code: Select all

# Become root in his home folder
sudo su -
# Go to the Mellanox OFED installation kit
cd MLNX_OFED_LINUX-4.6-1.0.1.1-ubuntu18.04-x86_64/
# Uninstall it
./uninstall.sh
# Get rid of it 
cd ..
rm -fr MLNX_OFED_LINUX-4.6-1.0.1.1-ubuntu19.04-x86_64
rm -f MLNX_OFED_LINUX-4.6-1.0.1.1-ubuntu18.04-x86_64 MLNX_OFED_LINUX-4.6-1.0.1.1-ubuntu18.04-x86_64.tgz
exit
# Install the apt repositories versions
sudo apt install rdma-core opensm ibutils ibverbs-utils infiniband-diags perftest qperf mstflint
The same maneuver applies to Ubuntu 19.04. Complete info there: https://www.google.com/search?q=infinib ... ox+drivers

domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Sun Aug 04, 2019 2:13 pm

RomaT wrote:
Fri Aug 02, 2019 10:46 pm
domih wrote:
Tue Jul 30, 2019 8:47 am
- Find out the correct incantations to put in the LVM conf file so that I don't lose the LVM2 tiering config at power off :-)
Could you write a step-by-step instruction?
As I understand it, you made SSD as a cache for HDD ?
I've already ordered two network cards (10Gbe on AQC107 chip), one for H2, the second for the other PC
Upon their arrival, I would like to be ready to set everything up...
Here it is:

Code: Select all

# References
# https://www.digitalocean.com/community/tutorials/how-to-use-lvm-to-manage-storage-devices-on-ubuntu-18-04
# http://man7.org/linux/man-pages/man7/lvmcache.7.html
# http://strugglers.net/~andy/blog/2017/07/19/bcache-and-lvmcache/
# https://manpages.ubuntu.com/manpages/precise/man8/update-initramfs.8.html
#
# /dev/sda is the big slow hard disk
# /dev/sdb is the fast small SSD

# Let's use the dm-cache based version (write through caching)
sudo lvcreate -n storage-origin-lv -l 100%PVS storage-vg /dev/sda
sudo lvcreate -n storage-cache-meta-lv -L 48M storage-vg /dev/sdb
sudo lvcreate -n storage-cache-lv -l 99%FREE storage-vg /dev/sdb
sudo lvconvert --type cache-pool --poolmetadata storage-vg/storage-cache-meta-lv storage-vg/storage-cache-lv
sudo lvconvert --type cache --cachepool storage-vg/storage-cache-lv storage-vg/storage-origin-lv
sudo mkfs -t ext4 /dev/storage-vg/storage-origin-lv

# Check it
sudo mount /dev/storage-vg/storage-origin-lv /mnt/storage
df -h
sudo umount /mnt/storage
sudo vi /etc/fstab
# use /dev/mapper/storage--vg-storage--origin--lv as device name
sudo mount -a
df -h

# Check all one last time
sudo pvscan
sudo vgscan
sudo lvscan

# Update the initial ram fs so that the LVM2 volume shows up
sudo update-initramfs -u
WARNING: the last line supposedly fixes my problem of the LVM2 volume not mounting at boot. I found this solution in CentOS forums. Apparently, LVM2 usage in Ubuntu is rare compared to CentOS. I have NOT tried it yet, so you're on your own.

The cache is a write through cache. Does not help much when writing, helps a lot when reading.

HTH.
These users thanked the author domih for the post:
RomaT (Sun Aug 04, 2019 2:48 pm)

RomaT
Posts: 218
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 34 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Sun Aug 04, 2019 3:12 pm

what packages should be installed ?
.
lvm2.png
lvm2.png (22.5 KiB) Viewed 1229 times

domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Sun Aug 04, 2019 3:16 pm

Do you sync or do you async? That is the question.

Since version 4.7 Samba sets "strict sync" to yes and "sync always" to no by default(*). This means (on Windows) that Samba writes according to the Windows file caching settings. For more information see discussion there: https://lists.samba.org/archive/samba/2 ... 43609.html. So with a Windows client, the file copy progress bar windows closes when all the files have been sent but it does not mean the files have all been written to the server disk. So if in the meantime there was a power failure or a crash on the server, then the Windows client lied to you: it told you your files were on the server while some of them are not. Don't panic because all these years you used SMB windows or Samba, this has always been the case anyway. There is always caching. The best way to see it is to copy a bunch of big files then shutdown the server. You'll see the server disk running extra time before the server shuts down: in other words the server is flushing the file cache. Same thing on Linux where the shutdown process syncs the disks. The same happens when you copy a big bunch of files between local disks and shutdown right away: the disks will run longer before the shutdown happens. In the era of UPSes who cares anyway.

(*) To see all the default values use 'testparm -v'.

Most of the times you do not see the effect of this caching because modern hard disks are much faster than their ancestors from the 90s. In other words, the file writing proceeds at pace with the 1 Gbe network. Once you start to use much faster networks it is another story. The transfer over the network completes much earlier than the asynchronous file writing. With NFS in sync mode, copying files within the desktop results in the progress bar jumping almost to completion and then stuck there until the files have been written. The "progress bar jumping almost to completion" corresponds to the network transfer and the "then stuck" corresponds to the file writing.

With networking > 10Gbe, the effect is striking.

To eliminate the Windows file caching I proceeded with Samba tests in Linux only, I also used the opportunity to test NFS both in sync and async modes.

NFS Setup

Code: Select all

# Reference: https://community.mellanox.com/s/article/howto-configure-nfs-over-rdma--roce-x
sudo apt update
sudo apt upgrade

Server
======
sudo mkdir /mnt/hda/nfs
# Don't care about security for the time being
sudo chown nobody:nogroup /mnt/hda/nfs
sudo chmod 0777 /mnt/hda/nfs

sudo apt install nfs-common nfs-kernel-server
sudo vi /etc/exports
.../...
/mnt/hda/nfs	*(rw,sync,no_subtree_check)
.../...
sudo modprobe svcrdma
sudo systemctl restart nfs-kernel-server
# Always execute this line after rebooting the nfs server
echo rdma 20049 | sudo tee /proc/fs/nfsd/portlist
sudo cat /proc/fs/nfsd/portlist

Client
======
sudo apt install nfs-common
sudo modprobe xprtrdma

# For testing 1Gbe
sudo mount i5-a:/mnt/hda/nfs /nfs/i5-a
time cp -v Desktop/CentOS-7-x86_64-DVD-1804.iso /nfs/i5-a
sudo umount /nfs/i5-a

# For testing RMDA IB
sudo mount -o rdma,port=20049 i5-a.ib:/mnt/hda/nfs /nfs/i5-a.ib
time cp -v Desktop/CentOS-7-x86_64-DVD-1804.iso /nfs/i5-a.ib
sudo umount /nfs/i5-a.ib
Note the "sync" in the exports file. For asynchronous testing, we change it to "async".

Results

Note: these tests are between 2 PCs, I did not have time to test the H2 yet, but I expect the same behavior. I tested the copy of a file 4.5+ GB in size. Best case scenario.

Code: Select all

Samba over 1 Gbe
================

domih@ripper:~$ time smbclient //i5-a/i5-a -U domih%xxxxxxxx -c 'put Desktop/CentOS-7-x86_64-DVD-1804.iso CentOS-7-x86_64-DVD-1804.iso'
putting file Desktop/CentOS-7-x86_64-DVD-1804.iso as \CentOS-7-x86_64-DVD-1804.iso (106082.9 kb/s) (average 106082.9 kb/s)
real	0m41.198s
user	0m0.045s
sys	0m1.360s
domih@ripper:~$ time smbclient //i5-a/i5-a -U domih%xxxxxxxx -c 'put Desktop/CentOS-7-x86_64-DVD-1804.iso CentOS-7-x86_64-DVD-1804.iso'
putting file Desktop/CentOS-7-x86_64-DVD-1804.iso as \CentOS-7-x86_64-DVD-1804.iso (90807.8 kb/s) (average 90807.8 kb/s)
real	0m48.119s
user	0m0.062s
sys	0m1.394s
domih@ripper:~$ time smbclient //i5-a/i5-a -U domih%xxxxxxxx -c 'put Desktop/CentOS-7-x86_64-DVD-1804.iso CentOS-7-x86_64-DVD-1804.iso'
putting file Desktop/CentOS-7-x86_64-DVD-1804.iso as \CentOS-7-x86_64-DVD-1804.iso (101912.3 kb/s) (average 101912.3 kb/s)
real	0m42.878s
user	0m0.060s
sys	0m1.275s

NFS over 1 Gbe (sync)
=====================

domih@ripper:~$ time cp -v Desktop/CentOS-7-x86_64-DVD-1804.iso /nfs/i5-a
'Desktop/CentOS-7-x86_64-DVD-1804.iso' -> '/nfs/i5-a/CentOS-7-x86_64-DVD-1804.iso'
real	0m40.558s
user	0m0.000s
sys	0m2.034s
domih@ripper:~$ time cp -v Desktop/CentOS-7-x86_64-DVD-1804.iso /nfs/i5-a
'Desktop/CentOS-7-x86_64-DVD-1804.iso' -> '/nfs/i5-a/CentOS-7-x86_64-DVD-1804.iso'
real	0m49.695s
user	0m0.004s
sys	0m2.178s
domih@ripper:~$ time cp -v Desktop/CentOS-7-x86_64-DVD-1804.iso /nfs/i5-a
'Desktop/CentOS-7-x86_64-DVD-1804.iso' -> '/nfs/i5-a/CentOS-7-x86_64-DVD-1804.iso'
real	0m49.786s
user	0m0.000s
sys	0m2.236s

NFS over 1 Gbe (async)
=====================

domih@ripper:~$ time cp -v Desktop/CentOS-7-x86_64-DVD-1804.iso /nfs/i5-a
'Desktop/CentOS-7-x86_64-DVD-1804.iso' -> '/nfs/i5-a/CentOS-7-x86_64-DVD-1804.iso'
real	0m39.877s
user	0m0.000s
sys	0m2.142s
domih@ripper:~$ time cp -v Desktop/CentOS-7-x86_64-DVD-1804.iso /nfs/i5-a
'Desktop/CentOS-7-x86_64-DVD-1804.iso' -> '/nfs/i5-a/CentOS-7-x86_64-DVD-1804.iso'
real	0m40.174s
user	0m0.000s
sys	0m2.234s
domih@ripper:~$ time cp -v Desktop/CentOS-7-x86_64-DVD-1804.iso /nfs/i5-a
'Desktop/CentOS-7-x86_64-DVD-1804.iso' -> '/nfs/i5-a/CentOS-7-x86_64-DVD-1804.iso'
real	0m40.060s
user	0m0.000s
sys	0m2.206s

Samba over IPoIB
================

domih@ripper:~$ time smbclient //i5-a.ib/i5-a -U domih%xxxxxxxx -c 'put Desktop/CentOS-7-x86_64-DVD-1804.iso CentOS-7-x86_64-DVD-1804.iso'
putting file /home/domih/Desktop/CentOS-7-x86_64-DVD-1804.iso as \CentOS-7-x86_64-DVD-1804.iso (215156.6 kb/s) (average 215156.6 kb/s)
real	0m20.337s
user	0m0.040s
sys	0m1.672s
domih@ripper:~$ time smbclient //i5-a.ib/i5-a -U domih%xxxxxxxx -c 'put Desktop/CentOS-7-x86_64-DVD-1804.iso CentOS-7-x86_64-DVD-1804.iso'
putting file /home/domih/Desktop/CentOS-7-x86_64-DVD-1804.iso as \CentOS-7-x86_64-DVD-1804.iso (208746.7 kb/s) (average 208746.7 kb/s)
real	0m20.958s
user	0m0.050s
sys	0m1.632s
domih@ripper:~$ time smbclient //i5-a.ib/i5-a -U domih%xxxxxxxx -c 'put Desktop/CentOS-7-x86_64-DVD-1804.iso CentOS-7-x86_64-DVD-1804.iso'
putting file /home/domih/Desktop/CentOS-7-x86_64-DVD-1804.iso as \CentOS-7-x86_64-DVD-1804.iso (216791.4 kb/s) (average 216791.4 kb/s)
real	0m20.181s
user	0m0.038s
sys	0m1.675s

NFS over RDMA IB (sync)
=====================

domih@ripper:~$ time cp -v Desktop/CentOS-7-x86_64-DVD-1804.iso /nfs/i5-a.ib
'Desktop/CentOS-7-x86_64-DVD-1804.iso' -> '/nfs/i5-a.ib/CentOS-7-x86_64-DVD-1804.iso'
real	0m22.266s
user	0m0.000s
sys	0m2.084s
domih@ripper:~$ time cp -v Desktop/CentOS-7-x86_64-DVD-1804.iso /nfs/i5-a.ib
'Desktop/CentOS-7-x86_64-DVD-1804.iso' -> '/nfs/i5-a.ib/CentOS-7-x86_64-DVD-1804.iso'
real	0m22.202s
user	0m0.000s
sys	0m2.077s
domih@ripper:~$ time cp -v Desktop/CentOS-7-x86_64-DVD-1804.iso /nfs/i5-a.ib
'Desktop/CentOS-7-x86_64-DVD-1804.iso' -> '/nfs/i5-a.ib/CentOS-7-x86_64-DVD-1804.iso'
real	0m21.694s
user	0m0.000s
sys	0m2.141s


NFS over RDMA IB (async)
======================

domih@ripper:~$ time cp -v Desktop/CentOS-7-x86_64-DVD-1804.iso /nfs/i5-a.ib
'Desktop/CentOS-7-x86_64-DVD-1804.iso' -> '/nfs/i5-a.ib/CentOS-7-x86_64-DVD-1804.iso'
real	0m3.447s
user	0m0.000s
sys	0m1.979s
domih@ripper:~$ time cp -v Desktop/CentOS-7-x86_64-DVD-1804.iso /nfs/i5-a.ib
'Desktop/CentOS-7-x86_64-DVD-1804.iso' -> '/nfs/i5-a.ib/CentOS-7-x86_64-DVD-1804.iso'
real	0m3.819s
user	0m0.000s
sys	0m2.081s
domih@ripper:~$ time cp -v Desktop/CentOS-7-x86_64-DVD-1804.iso /nfs/i5-a.ib
'Desktop/CentOS-7-x86_64-DVD-1804.iso' -> '/nfs/i5-a.ib/CentOS-7-x86_64-DVD-1804.iso'
real	0m3.949s
user	0m0.000s
sys	0m2.064s
Note: I have not tested Samba in async mode, but I expect the result to be similar. On a Windows client, it is.

CONCLUSION
- Samba or NFS: about the same speed.
- NFS sync over RDMA IB as slow as NFS sync over 1 Gbe: asynchronous file writing is at pace with the network speed with 1Gbe. With RMDA IB, we measure the file writing speed, not the network speed.
- NFS async over RDMA IB leaves NFS async over 1 Gbe in the dust. So the network transfer actually took less than 4 seconds with RDMA IB.

For your entertainment, I made a video of a NFS async over RDMA IB file copy within the desktop, see attachment. You have to download and uncompress it. Attachments do not support video files.

FINAL CONCLUSION
In a SOHO environment, do you need sync or async? You probably can get away with async making your network files copy going full speed when you use a 10+ Gbe network.
Attachments
simplescreenrecorder-2019-08-02_20.26.24.mkv.gz
(628.64 KiB) Downloaded 17 times
Last edited by domih on Sun Aug 04, 2019 3:29 pm, edited 2 times in total.

domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Sun Aug 04, 2019 3:17 pm

RomaT wrote:
Sun Aug 04, 2019 3:12 pm
what packages should be installed ?
.
lvm2.png
LVM2 with its dependencies should do it. I do not think you need the thin things.

domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Sun Aug 04, 2019 3:21 pm

mad_ady wrote:
Wed Jul 31, 2019 1:12 pm
If you have enough ram you can create a ramdisk - to get away from disk limitations.
Anyway - even pure network speed has its advantages (e.g. as traffic generator/stress tester).
See post "Do you sync or do you async? That is the question.". Solves the issue of speed when writing files. Using a RAM disk would be in essence "simulating" the same thing.
These users thanked the author domih for the post:
mad_ady (Sun Aug 04, 2019 6:21 pm)

domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Sun Aug 04, 2019 3:37 pm

@RomaT

See discussion of a few pioneers who tried 10GBe cards identical to the ones your ordered on the RockPro64. https://forum.pine64.org/showthread.php?tid=6964

I tried it on the RockPro64 but like the Ryan member, I could not make the card (Mellanox 10Gbe) work. Same experience as he described. I guess the other ones were Linux Kernel savvy enough to make it work.

Anyway, you should get very close even better results with the H2. Have fun!

RomaT
Posts: 218
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 34 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Sun Aug 04, 2019 3:48 pm

It would be cool if instead of integrated NICs there was a second M2 connector with two lanes PCIe for the SATA controller...
and in accessories they sold M2 dual NICs and M2 SATA controller or even M.2 Wi-Fi with bluetooth
everyone could choose what is more important to him...

User avatar
Mullcom
Posts: 87
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 47 times
Been thanked: 3 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Mon Aug 05, 2019 2:02 am

Hey!! Look at the price!

https://rover.ebay.com/rover/0/0/0?mpre ... 3664807690

Domih

What is the name for this cables type ?

My knowledge is pretty low on this type of ports.
It's not regular ssf that i have in my switch.

To bad i am not living in US.... :(



Skickat från min SM-G955F via Tapatalk





User avatar
Mullcom
Posts: 87
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 47 times
Been thanked: 3 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Mon Aug 05, 2019 3:27 am

I yast buy 2 of

Mellanox MHQH29C-XTR Network Adapter

Om eBay.

Going to put one in my freenas box and other in H2. Hope this setup going to work.

Skickat från min SM-G955F via Tapatalk


domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Mon Aug 05, 2019 3:47 am

Mullcom wrote:
Mon Aug 05, 2019 2:02 am
Hey!! Look at the price!

https://rover.ebay.com/rover/0/0/0?mpre ... 3664807690

Domih

What is the name for this cables type ?

My knowledge is pretty low on this type of ports.
It's not regular ssf that i have in my switch.

To bad i am not living in US.... :(



Skickat från min SM-G955F via Tapatalk
Yes, this switch is the model running in my living room. It's a 40Gbps model. The cable are QSFP (Quad Small Form-Factor Pluggable) which means they are SFP and carry 4 channels of 10Gbps, hence the 40Gbps top speed. 40 Gbps cables from Mellanox that work with this switch are either DAC Copper (passive) or Optical (active). For home usage and small lengths you can use either ones and passive coppers are less expensive.

For optical active, search for "MC2206310" on ebay. For passive copper, search for "MC2206130 copper" on eBay.

https://en.wikipedia.org/wiki/Small_for ... ransceiver
https://store.mellanox.com/categories/i ... 0gb-s.html
https://store.mellanox.com/categories/i ... b-s-1.html
These users thanked the author domih for the post:
Mullcom (Mon Aug 05, 2019 3:49 am)

domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Mon Aug 05, 2019 3:49 am

Mullcom wrote:
Mon Aug 05, 2019 3:27 am
I yast buy 2 of

Mellanox MHQH29C-XTR Network Adapter

Om eBay.

Going to put one in my freenas box and other in H2. Hope this setup going to work.

Skickat från min SM-G955F via Tapatalk
Note that you do not need a switch right away. You can connect the two cards directly.

User avatar
Mullcom
Posts: 87
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 47 times
Been thanked: 3 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Mon Aug 05, 2019 4:03 am

domih wrote:
Mullcom wrote:
Mon Aug 05, 2019 3:27 am
I yast buy 2 of

Mellanox MHQH29C-XTR Network Adapter

Om eBay.

Going to put one in my freenas box and other in H2. Hope this setup going to work.

Skickat från min SM-G955F via Tapatalk
Note that you do not need a switch right away. You can connect the two cards directly.
I did understand that. When i have read all Good infoation you have deliver in here. :)

This is what i havet in mind. I did check my switch i have and it only supported SFP so it time to do some changes.

Skickat från min SM-G955F via Tapatalk


RomaT
Posts: 218
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 34 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Mon Aug 05, 2019 8:25 pm

domih wrote:
Sun Aug 04, 2019 2:13 pm
WARNING: the last line supposedly fixes my problem of the LVM2 volume not mounting at boot.
I found this solution in CentOS forums. Apparently, LVM2 usage in Ubuntu is rare compared to CentOS.
I have NOT tried it yet, so you're on your own.
Today I bought the SSD Samsung 860Pro 512GB

I seem to have succeeded, but I have not yet rebooted,
devices without created partitions cannot be seen on the lvmdiskscan,
I created on both disks partition ext4, after that they appeared in the lvmdiskscan,
Only in your instructions, in the beginning, there was no:

Code: Select all

sudo lvmdiskscan
sudo pvcreate /dev/sda1 /dev/sdb1
sudo vgcreate storage-vg /dev/sda1
sudo vgextend storage-vg /dev/sdb1
further fine, but you need to specify the device name partitions (sda1 or sdb1, not sda or sdb):

Code: Select all

sudo lvcreate -n storage-origin-lv -l 100%PVS storage-vg /dev/sda1
sudo lvcreate -n storage-cache-meta-lv -L 48M storage-vg /dev/sdb1
sudo lvcreate -n storage-cache-lv -l 99%FREE storage-vg /dev/sdb1
sudo lvconvert --type cache-pool --poolmetadata storage-vg/storage-cache-meta-lv storage-vg/storage-cache-lv
sudo lvconvert --type cache --cachepool storage-vg/storage-cache-lv storage-vg/storage-origin-lv
sudo mkfs -t ext4 /dev/storage-vg/storage-origin-lv
further I mounted already on UUID and edit fstab:

Code: Select all

UUID=1234-5678-91011 /mnt/storage auto nosuid,nodev,nofail 0 0
.
lvm.png
lvm.png (9.08 KiB) Viewed 970 times
Last edited by RomaT on Wed Aug 07, 2019 10:27 pm, edited 3 times in total.
These users thanked the author RomaT for the post:
domih (Tue Aug 06, 2019 3:21 pm)

domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Tue Aug 06, 2019 3:15 pm

Oops, it looks like I indeed forgot a few lines in my local HOWTO file. Sorry :oops:

I will double-check my history files. It could be that my drives were already initialized during previous unrelated tests.

P.S. When you receive the NICs, if you end up with 2.5Gbe or 5Gbe max, it could be due to the RJ-45 cable you use. CAT-5e is not supposed to be able to carry 10Gbe. Remedy: use a CAT-6a instead. I do not know about CAT-6, but CAT-6a will carry 10Gbe. There are numerous discussions on the web about that. Just google it.

RomaT
Posts: 218
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 34 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Tue Aug 06, 2019 3:32 pm

I will use a shielded 7th category ...
I do not have a 10Gbps switch yet, I will connect the card to the card, cross method.

User avatar
Mullcom
Posts: 87
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 47 times
Been thanked: 3 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Tue Aug 06, 2019 9:55 pm

Following post of my projects.

Start here. viewtopic.php?p=264892#p264892

Skickat från min SM-G955F via Tapatalk


Shelled is only use when you have no other choice then need to have the cable lying between big power cables.
Last edited by Mullcom on Tue Aug 06, 2019 9:55 pm, edited 2 times in total.

User avatar
Mullcom
Posts: 87
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 47 times
Been thanked: 3 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Tue Aug 06, 2019 11:20 pm

domih wrote:Oops, it looks like I indeed forgot a few lines in my local HOWTO file. Sorry :oops:

I will double-check my history files. It could be that my drives were already initialized during previous unrelated tests.

P.S. When you receive the NICs, if you end up with 2.5Gbe or 5Gbe max, it could be due to the RJ-45 cable you use. CAT-5e is not supposed to be able to carry 10Gbe. Remedy: use a CAT-6a instead. I do not know about CAT-6, but CAT-6a will carry 10Gbe. There are numerous discussions on the web about that. Just google it.
I can confirm that cat 6e is Good enough to carry 10gb. Different between cat 6 and 6e is the Hz frequency. That i have not so much knowledge about. More then i think it's how fare you can transfer in that speed.

Wiki
Whereas Category 6 cable has a reduced maximum length of 55 meters when used for 10GBASE-T, Category 6A cable (or Augmented Category 6) is characterized to 500 MHz and has improved alien crosstalk characteristics, allowing 10GBASE-T to be run for the same 100 meter maximum distance as previous Ethernet variants.

Link to all types.
https://en.m.wikipedia.org/wiki/Categor ... net_cables



Skickat från min SM-G955F via Tapatalk



RomaT
Posts: 218
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 34 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Wed Aug 07, 2019 1:58 pm

after reboot is not mounted :?
there is no directory /dev/storage-vg/
and there are no array-related devices in directory /dev/*
but the array itself seems to be alive:
.
lvm2_no mount.png
lvm2_no mount.png (14.52 KiB) Viewed 832 times
Last edited by RomaT on Wed Aug 07, 2019 5:41 pm, edited 1 time in total.
These users thanked the author RomaT for the post:
Mullcom (Wed Aug 07, 2019 6:20 pm)

RomaT
Posts: 218
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 34 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Wed Aug 07, 2019 5:33 pm

the problem is solved by the team:

Code: Select all

vgchange -a y storage-vg
initially there was an error on

Code: Select all

/usr/sbin/cache_check: execvp failed: No such file or directory
found a blog about this https://bugs.debian.org/cgi-bin/bugrepo ... bug=773731
this file is contained in the package thin-provisioning-tools, about which I was just asking you if it was needed or not
apt install thin-provisioning-tools
and it worked ;)
after rebooting everything is mounted, everything is fine
i.e. installing this package solves the mount problem on reboot !
Summarizing your instructions, the following:

Code: Select all

# References
# https://www.digitalocean.com/community/tutorials/how-to-use-lvm-to-manage-storage-devices-on-ubuntu-18-04
# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=773731
#
# /dev/sda is the big slow hard disk
# /dev/sdb is the fast small SSD
sudo apt update
sudo apt install lvm2 thin-provisioning-tools
sudo lvmdiskscan
sudo pvcreate /dev/sda1 /dev/sdb1
sudo vgcreate storage-vg /dev/sda1
sudo vgextend storage-vg /dev/sdb1
# Let's use the dm-cache based version (write through caching)
sudo lvcreate -n storage-origin-lv -l 100%PVS storage-vg /dev/sda1
sudo lvcreate -n storage-cache-meta-lv -L 48M storage-vg /dev/sdb1
sudo lvcreate -n storage-cache-lv -l 99%FREE storage-vg /dev/sdb1
sudo lvconvert --type cache-pool --poolmetadata storage-vg/storage-cache-meta-lv storage-vg/storage-cache-lv
sudo lvconvert --type cache --cachepool storage-vg/storage-cache-lv storage-vg/storage-origin-lv
sudo mkfs -t ext4 /dev/storage-vg/storage-origin-lv

# Check it
sudo mount /dev/storage-vg/storage-origin-lv /mnt/storage
df -h
sudo umount /mnt/storage
sudo vi /etc/fstab
# use /dev/mapper/storage--vg-storage--origin--lv as device name
# UUID=1234-5678-91011 /mnt/storage auto nosuid,nodev,nofail 0 0
sudo mount -a
df -h

# Check all one last time
sudo pvscan
sudo vgscan
sudo lvscan

# Update the initial ram fs so that the LVM2 volume shows up
sudo update-initramfs -u
sudo reboot
# if not /dev/storage-vg/* then 
sudo vgchange -a y storage-vg
Last edited by RomaT on Wed Aug 07, 2019 6:27 pm, edited 1 time in total.
These users thanked the author RomaT for the post (total 2):
Mullcom (Wed Aug 07, 2019 6:20 pm) • domih (Fri Aug 09, 2019 1:11 pm)

User avatar
Mullcom
Posts: 87
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 47 times
Been thanked: 3 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by Mullcom » Wed Aug 07, 2019 6:27 pm

RomaT wrote:the problem is solved by the team:

Code: Select all

vgchange -a y storage-vg
initially there was an error on

Code: Select all

/usr/sbin/cache_check: execvp failed: No such file or directory
found a blog about this https://bugs.debian.org/cgi-bin/bugrepo ... bug=773731
this file is contained in the package thin-provisioning-tools, about which I was just asking you if it was needed or not
apt install thin-provisioning-tools
and it worked ;)
after rebooting everything is mounted, everything is fine
i.e. installing this package solves the mount problem on reboot !
Summarizing your instructions, the following:

Code: Select all

# References
# https://www.digitalocean.com/community/tutorials/how-to-use-lvm-to-manage-storage-devices-on-ubuntu-18-04
# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=773731
#
# /dev/sda is the big slow hard disk
# /dev/sdb is the fast small SSD
sudo apt update
sudo apt install lvm2 thin-provisioning-tools
sudo lvmdiskscan
sudo pvcreate /dev/sda1 /dev/sdb1
sudo vgcreate storage-vg /dev/sda1
sudo vgextend storage-vg /dev/sdb1
# Let's use the dm-cache based version (write through caching)
sudo lvcreate -n storage-origin-lv -l 100%PVS storage-vg /dev/sda1
sudo lvcreate -n storage-cache-meta-lv -L 48M storage-vg /dev/sdb1
sudo lvcreate -n storage-cache-lv -l 99%FREE storage-vg /dev/sdb1
sudo lvconvert --type cache-pool --poolmetadata storage-vg/storage-cache-meta-lv storage-vg/storage-cache-lv
sudo lvconvert --type cache --cachepool storage-vg/storage-cache-lv storage-vg/storage-origin-lv
sudo mkfs -t ext4 /dev/storage-vg/storage-origin-lv

# Check it
sudo mount /dev/storage-vg/storage-origin-lv /mnt/storage
df -h
sudo umount /mnt/storage
sudo vi /etc/fstab
# use /dev/mapper/storage--vg-storage--origin--lv as device name
# UUID=1234-5678-91011 /mnt/storage auto nosuid,nodev,nofail 0 0
sudo mount -a
df -h

# Check all one last time
sudo pvscan
sudo vgscan
sudo lvscan

# Update the initial ram fs so that the LVM2 volume shows up
sudo update-initramfs -u
sudo reboot
# if not /dev/* then sudo vgchange -a y storage-vg
Nice good work i add it in main post and link to this post.

Skickat från min SM-G955F via Tapatalk


domih
Posts: 108
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 34 times
Been thanked: 23 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by domih » Fri Aug 09, 2019 1:19 pm

RomaT wrote:
Wed Aug 07, 2019 5:33 pm
the problem is solved by the team:

Code: Select all

vgchange -a y storage-vg
initially there was an error on

Code: Select all

/usr/sbin/cache_check: execvp failed: No such file or directory
found a blog about this https://bugs.debian.org/cgi-bin/bugrepo ... bug=773731
this file is contained in the package thin-provisioning-tools, about which I was just asking you if it was needed or not
apt install thin-provisioning-tools
and it worked ;)
after rebooting everything is mounted, everything is fine
i.e. installing this package solves the mount problem on reboot !
Summarizing your instructions, the following:

Code: Select all

# References
# https://www.digitalocean.com/community/tutorials/how-to-use-lvm-to-manage-storage-devices-on-ubuntu-18-04
# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=773731
#
# /dev/sda is the big slow hard disk
# /dev/sdb is the fast small SSD
sudo apt update
sudo apt install lvm2 thin-provisioning-tools
sudo lvmdiskscan
sudo pvcreate /dev/sda1 /dev/sdb1
sudo vgcreate storage-vg /dev/sda1
sudo vgextend storage-vg /dev/sdb1
# Let's use the dm-cache based version (write through caching)
sudo lvcreate -n storage-origin-lv -l 100%PVS storage-vg /dev/sda1
sudo lvcreate -n storage-cache-meta-lv -L 48M storage-vg /dev/sdb1
sudo lvcreate -n storage-cache-lv -l 99%FREE storage-vg /dev/sdb1
sudo lvconvert --type cache-pool --poolmetadata storage-vg/storage-cache-meta-lv storage-vg/storage-cache-lv
sudo lvconvert --type cache --cachepool storage-vg/storage-cache-lv storage-vg/storage-origin-lv
sudo mkfs -t ext4 /dev/storage-vg/storage-origin-lv

# Check it
sudo mount /dev/storage-vg/storage-origin-lv /mnt/storage
df -h
sudo umount /mnt/storage
sudo vi /etc/fstab
# use /dev/mapper/storage--vg-storage--origin--lv as device name
# UUID=1234-5678-91011 /mnt/storage auto nosuid,nodev,nofail 0 0
sudo mount -a
df -h

# Check all one last time
sudo pvscan
sudo vgscan
sudo lvscan

# Update the initial ram fs so that the LVM2 volume shows up
sudo update-initramfs -u
sudo reboot
# if not /dev/storage-vg/* then 
sudo vgchange -a y storage-vg
Thank you very much for solving the issue. One can wonder why the hell they don't make this package as dependency since the issue has been identified and solved 4 years ago *&^%$#@!

PS: pvcreate, disks or partitions? Both are possible but with a caveat, see http://tldp.org/HOWTO/LVM-HOWTO/initdisks.html. So I guess I'll follow your tracks here and use partitions from now on.

RomaT
Posts: 218
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 34 times
Contact:

Re: Add-on to H2 - M2 to pciE slot

Unread post by RomaT » Sun Aug 11, 2019 9:35 pm

at least in ubuntu does not allow to create a physical volume to disks,

Code: Select all

# pvcreate /dev/sda
Reports an error. Disk did not contain partitions - empty.

but specifying disks partitions is easy to create, one partition was previously created.

Code: Select all

# pvcreate /dev/sda1
complete.

Post Reply

Return to “Hardware and peripherals”

Who is online

Users browsing this forum: No registered users and 0 guests