10GB ethernet instead of 1GB...

Post Reply
earlye
Posts: 1
Joined: Tue Jan 15, 2019 5:34 am
languages_spoken: english
Contact:

10GB ethernet instead of 1GB...

Unread post by earlye » Tue Jan 15, 2019 5:47 am

Have you considered, or _would_ you consider building a variant of the H2 with 10GB rj-45 ports rather than 1GB? It seems to me that such a beast, coupled w/ 1TB nvme support, would make a fantastic low-cost node for large kubernetes or cassandra clusters.

crashoverride
Posts: 4314
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: 10GB ethernet instead of 1GB...

Unread post by crashoverride » Tue Jan 15, 2019 7:47 am

The board would need to be based on an Intel Core processor (instead of Celeron) to make use (PCIe 3+) of 10GB + NVMe.

tkaiser
Posts: 671
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Contact:

Re: 10GB ethernet instead of 1GB...

Unread post by tkaiser » Tue Jan 15, 2019 4:46 pm

earlye wrote:
Tue Jan 15, 2019 5:47 am
Have you considered, or _would_ you consider building a variant of the H2 with 10GB rj-45 ports rather than 1GB? It seems to me that such a beast, coupled w/ 1TB nvme support, would make a fantastic low-cost node for large kubernetes or cassandra clusters.
  • Those Gemini Lake CPUs (regardless whether they're called Pentium or Celeron) only have 6 Gen2 PCIe lanes.
  • A fast NVMe SSD wants 4 Gen3 lanes (so already bottlenecked since Gemini Lake provides only half the data rates with Gen2 PCIe)
  • Each 10GbE PCIe controller wants 4 Gen2 lanes.
For what you want you would need at least a Celeron G4900 featuring sufficient count of PCIe Gen3 lanes...

With H2 in mind still hoping for the intermediate NBase-T/2.5GbE step instead of unrealistic 10GbE: https://github.com/ThomasKaiser/Knowled ... networking

PigLover
Posts: 21
Joined: Wed Nov 15, 2017 3:55 am
languages_spoken: english
ODROIDs: C2, XU4, HC1, MC1
Contact:

Re: 10GB ethernet instead of 1GB...

Unread post by PigLover » Wed Jan 16, 2019 3:07 am

2x Nbase-T 2.5 is the right approach for this board, as noted by @tkaiser. For things like Cassandra, Gluster or Ceph nodes it would be MUCH better than the 2x 1gbe we have today (though I don't think it would make much difference for router/firewall apps due to small packet performance bottlenecking in the CPU).

Its an almost trivial design change for Odroid to replace the PHYs in another product spin.

User avatar
mad_ady
Posts: 5403
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Contact:

Re: 10GB ethernet instead of 1GB...

Unread post by mad_ady » Wed Jan 16, 2019 4:07 am

How affordable/available is 2.5Gbps networking gear? Are 10Gbps switches usually compatible with 2.5Gbps ports?
The networking gear I'm familiar with (ISP level) makes no mention of anything between 1 and 10Gbps.

crashoverride
Posts: 4314
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: 10GB ethernet instead of 1GB...

Unread post by crashoverride » Wed Jan 16, 2019 4:35 am

There are two USB 3.0 ports that can be used to add multiple 2.5/5GB Ethernet ports (and cost) at a customers discretion.
https://www.aquantia.com/products/contr ... u-aqc112u/

tkaiser
Posts: 671
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Contact:

Re: 10GB ethernet instead of 1GB...

Unread post by tkaiser » Wed Jan 16, 2019 4:54 am

mad_ady wrote:
Wed Jan 16, 2019 4:07 am
How affordable/available is 2.5Gbps networking gear?
https://translate.google.de/translate?s ... E3328_2017

But more important... consumption decreases compared to 10GbE and you don't need new cables for 2.5GbE: https://en.wikipedia.org/wiki/2.5GBASE-T_and_5GBASE-T
mad_ady wrote:
Wed Jan 16, 2019 4:07 am
Are 10Gbps switches usually compatible with 2.5Gbps ports?
Not with older 10GbE switches. More recent ones adopt NBase-T for the simple reason that we'll see more and more 2.5GbE capable clients and other network gear (access points for example that implement 802.11ac Wave 2 or Wi-Fi 6 are already bottlenecked by Gigabit Ethernet). Same with NICs. Intel for example added 2.5GbE and 5GbE compatibility already 2.5 years ago: https://www.tomshardware.com/news/multi ... 32019.html

Mainboard vendors like ASRock, ASUS and Gigabyte more and more integrate AQC-108 (5GbE) and RTL8125 (2.5GbE) controllers... mostly on their 'gaming' mainboards for now.

crashoverride
Posts: 4314
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: 10GB ethernet instead of 1GB...

Unread post by crashoverride » Wed Jan 16, 2019 9:52 am

I was revisiting this topic today for something non-related and discovered the math is wrong.

PCIe Speed by Version (per lane):
1.0 = 2.5GT/s (8b/10b) = 250MB/s
2.x = 5.0GT/s (8b/10b) = 500MB/s
3.0 = 8.0GT/s (128b/130b) = 984.6MB/s

This means it would be (theoretically) possible to remove the two NICs and replace them with a single 10Gbit/s Ethernet controller (2 x 5.0GT/s) while still providing twice that (4 x 5.0GT/s) to a NVMe. This implication is that it would be (theoretically) possible to easily saturate a 10Gbit/s link from a PCIe2.0 x 4 NVMe source.

The current prices for 10Gb/s Ethernet PCIe cards I saw today indicate this would likely add $60 to $100 to the parts cost.

tkaiser
Posts: 671
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Contact:

Re: 10GB ethernet instead of 1GB...

Unread post by tkaiser » Wed Jan 16, 2019 4:46 pm

crashoverride wrote:
Wed Jan 16, 2019 9:52 am
it would be (theoretically) possible to remove the two NICs and replace them with a single 10Gbit/s Ethernet controller (2 x 5.0GT/s)
This is also practically possible but the remaining question is how bottlenecked such a controller then will be and how maximum transfer rates via Ethernet will look like. Doing 'math' this way ('2 x 5.0GT/s' == 10... and '10 Gbps' is also 10!) is possible but seems not to be useful given that each and every PCIe attached 10 Gigabit network adapter out there wants PCIe Gen2 x4 (single port) or Gen2 x8 (dual port).

A good indication of why this sort of math could be flawed give NVMe bandwidth tests. SSDs that are known to exceed 2000 MB/s on Gen3 x4 ports do not go beyond 1700 MB/s when attached via Gen2 x4 while 'math' tells 2000 MB/s should be the '4 x 5.0GT/s' limit.

Next question: Does the host allow to saturate the PCIe bus fully? Some practical insights (though from server platforms where PCIe lanes often originate at the PCH interconnected via DMI to the CPU): https://www.cnx-software.com/2019/01/09 ... ent-559851

crashoverride
Posts: 4314
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Contact:

Re: 10GB ethernet instead of 1GB...

Unread post by crashoverride » Wed Jan 16, 2019 11:58 pm

A better way to put it is that 2x5GT/s for the network and 4x5GT/s for the NVMe is the best the processor can achieve using all of its 6 PCIe 2.0 lanes. Even if you attribute 20% bandwidth loss to protocol overhead, etc. that would still leave 8Gb/s on a 10Gb/s interface. Since H2 is not a datacenter/enterprise product, this level of cost to performance is more than adequate.

It may be possible to test half of this theory. Using a NVMe to PCIe adapter board, a 10Gbs network interface card could be used. Two of the PCIe lanes could then be disabled to achieve a 2x5GT/s link to the card. The bandwidth could then be measured using iperf.

back2future
Posts: 197
Joined: Sun Jul 23, 2017 3:19 pm
languages_spoken: english
Contact:

Re: 10GB ethernet instead of 1GB...

Unread post by back2future » Sat Feb 09, 2019 2:57 am

some experiences with rk3399 Soc on Rockpro64 and 10GbE on 4x PCIe 2.1
https://forum.pine64.org/showthread.php?tid=6964

User avatar
rooted
Posts: 6435
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Contact:

Re: 10GB ethernet instead of 1GB...

Unread post by rooted » Sat Feb 09, 2019 3:52 am

back2future wrote:some experiences with rk3399 Soc on Rockpro64 and 10GbE on 4x PCIe 2.1
https://forum.pine64.org/showthread.php?tid=6964
330 MB/s is pretty good, not sure who needs this but if you did at least it works...sort of.

User avatar
mad_ady
Posts: 5403
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Contact:

Re: 10GB ethernet instead of 1GB...

Unread post by mad_ady » Sat Feb 09, 2019 4:28 am

Without interrupt coalescing supported by the card/driver, interrupts will be the bottleneck

back2future
Posts: 197
Joined: Sun Jul 23, 2017 3:19 pm
languages_spoken: english
Contact:

Re: 10GB ethernet instead of 1GB...

Unread post by back2future » Sat Feb 09, 2019 7:16 am

Means wider controller busses to/from pcie transfering serialized multiple pipe width data packets for less interrupting (one) cpu (per each pcie transfer cycle).
Never heard pcie 6.0, but Thunderbolt instead? https://www.synopsys.com/designware-ip/ ... signs.html
But that's for common mainstream (lower cost) sbc probably at least ~1 decade away?

If higher network bandwidth reduces cpu time and power consumption it could be worth the efforts pretty fast?
Last edited by back2future on Sun Feb 10, 2019 8:29 pm, edited 2 times in total.

PigLover
Posts: 21
Joined: Wed Nov 15, 2017 3:55 am
languages_spoken: english
ODROIDs: C2, XU4, HC1, MC1
Contact:

Re: 10GB ethernet instead of 1GB...

Unread post by PigLover » Sat Feb 09, 2019 2:30 pm

Still seems to me that 2x Nbase-T 2.5Gbe would be the most appropriate upgrade. Swapping the RTL8111G Phy for the RTL8125 is pretty much just a part-pick change for manufacturing and then test. Seems to me that HardKernel could spin an H2+ pretty quick. And the SoC should be able to handle 2.5Gbe with little problem.

As I said previously - its not 10Gbe, but it would be a MASSIVE improvement in performance for quite a few workloads.

back2future
Posts: 197
Joined: Sun Jul 23, 2017 3:19 pm
languages_spoken: english
Contact:

Re: 10GB ethernet instead of 1GB...

Unread post by back2future » Sat Feb 09, 2019 7:05 pm

back2future wrote:
Sat Feb 09, 2019 2:57 am
some experiences with rk3399 Soc on Rockpro64 and 10GbE on 4x PCIe 2.1
https://forum.pine64.org/showthread.php?tid=6964
Thy tell, that 10GbE v3 x4 is 3.47-4.84W, while rtl8111e ("At 1Gbps with heavy network traffic") is 0.2-0.3W.
While getting a new rtl8111 2-port pcie card for ~25$ (1-port ~5$, 4-port ~27$), used mlx v3 are ~45$ and additional cabling cost.
Rtl8125 (per ? pcs lot) ~$, drivers for various platforms from Realtek support, but not much additional information for this 2018 ic.

[ overview Aquantia 5-10GbE 2016/2017: https://www.anandtech.com/Show/Index/10 ... rs-for-pcs ]

Post Reply

Return to “General Topics”

Who is online

Users browsing this forum: No registered users and 4 guests