Testing options SATA expansion via M.2

[brad]

Happy new year all,

Im currently looking to setup SAN/NAS and considering physical disks.

Not 100% sure if this will work but planning to have 6 x ssd sata discs connected.

M.2 to mini SAS adaptor - https://www.ebay.com.au/itm/StarTech-U- ... 3884696603
mini SAS to 4x SATA - https://www.ebay.com.au/itm/Mini-SAS-SF ... 3522114198

This should hopefully give 6 high speed SATA ports that I can use in total on the H2+. Ive ordered another H2+ and products listed above for a test before I go out and buy all the drives.

RAID 10: 3 disk stripped together and then mirrored
RAID 50: 2 x RAIDZ mirrored

RAID 50 should allow fastest read speeds and most available disk space whereas RAID10 will have better write performance. Plan is to use 6x 500Gb SSD's which in RAID10 is 1.5Tb usable.

I looked at connecting M.2 to 5x sata adapter but they all use M.2 connectors with only 2x pcie lanes so going to test if the path of M.2 to mini SAS to 4x SATA is possible.

Plan was to install FreeNAS as use it as block / file storage (which is now TrueNAS CORE) but the new TrueNAS SCALE which is about to be released looks better. It allows horizontal scaling of the cluster by adding more machines and some other nice features like running containers or vm's.

[TurdFurguson]

One concern / question. The "mini SAS" adapter is actually a U.2 adapter. Now, I could be wrong (and often am), but while the connectors are the same is the SAS protocol the same as U.2?

[brad]

One concern / question. The "mini SAS" adapter is actually a U.2 adapter. Now, I could be wrong (and often am), but while the connectors are the same is the SAS protocol the same as U.2?

This is the question I need to answer with my test I think, from all my research I'm unsure at this stage it might some down to firmware even.

[brad]

The mini SAS to sata test did not work, essentially the adapter requires the SATA lanes to exist on the connector.

I do believe this will work https://www.amazon.com.au/Internal-Non- ... B07T3RMFFT as it has a proper pcie to sata controller (JMB585).

The only issue I don't like here is only 2xPCIE lanes at pcie 2.0 speeds. Will likely be a bottleneck I think for 4 x SSD but I think it would work well for up to 2 SSD's (or 4+ HDD). Maybe I'm wrong?

[odroid]

Sad news.
I think that HDDs will not have a problem while SSDs will have some band width bottleneck certainly.

[brad]

I had a good look around at M.2 cards but could find nothing with pciex4 interface they are all max pciex2 (even if they don't make that obvious). I also looked at standard full pcie to sata adaptors and all (most) of these are pciex2 only.

Final option I think is a ServeRAID H1110 SAS/SATA Controller https://lenovopress.com/tips0831-serveraid-h1110

A relatively small pcie card with 4 x 6G sata, can fit in the case (type 5 case) along with 6x 2.5 SSD. Has a PCIEx4 version 2.0 connection, can be flashed with non raid firmware and is quiet cheap. Drivers exist for windows, linux & BSD. Think ill order one and try my luck.

[defcon]

I do believe this will work https://www.amazon.com.au/Internal-Non- ... B07T3RMFFT as it has a proper pcie to sata controller (JMB585).

Yep, works nice for me on H2+ with 3x 12TB disks, more disks coming. JMB585 is quite fast, some tests https://forums.unraid.net/topic/41340-s ... ity-check/
But you will need custom heatsink on the chip, or you'll get I/O errors after a while. Like this. There is blue silicone thermal pad 1.5mm thick between chip and heatsink and its zip tied. And the black wire stuffed in heatsink is temp sensor from case fan control board.

IMG_0159.jpg (248.8 KiB) Viewed 171 times

[brad]

I do believe this will work https://www.amazon.com.au/Internal-Non- ... B07T3RMFFT as it has a proper pcie to sata controller (JMB585).

Yep, works nice for me on H2+ with 3x 12TB disks, more disks coming. JMB585 is quite fast, some tests https://forums.unraid.net/topic/41340-s ... ity-check/
But you will need custom heatsink on the chip, or you'll get I/O errors after a while. Like this. There is blue silicone thermal pad 1.5mm thick between chip and heatsink and its zip tied. And the black wire stuffed in heatsink is temp sensor from case fan control board.

Thanks for the details on JMB585 M.2 board. Interesting that the heatsink was needed to prevent IO errors (I guess the small M.2 board had difficulties dissipating the heat).

Also the page with all the tests is very interesting and helpful as well. Just need to remember that the odroid H2 / H2+ is limited to PCIE version 2.0 but the page shows the JMB585 in PCIE 3.0 with 2 channels so results will be much different on Odroid. The tests look good for the H1110 though with PCIE 2.0 x4

[defcon]

Nice find, I did not know about PCIe 2.0. Anyway, 2x lanes give 1000MB/s, 200 per disk. It does hit performance of spinning hdds, but very little. SSDs are out of luck

[brad]

I have ordered a ServeRaid M1015 and ServeRaid H1110 card for testing, hopefully they don't take too long to arrive.

The H1110 is 4x pcie 2.0 4 port SATA/SAS and the M1015 is 8x pcie 2.0 8 port SATA/SAS. They can be flashed with either RAID or non RAID firmware. I grabbed the 8 port sata version as well so I can try to connect up some HDD's along side the SSD's for an additional storage pool.

I also did some initial testing with TrueNAS SCALE (It is the linux based alpha version of TrueNAS CORE). Unfortunately it would not detect an emmc based device so I needed to install the OS to a SSD meaning it could not be used for the main storage pool (although OS could be installed to USB). TrueNAS scale alpha looks like it will be very nice once complete but at the moment the nighty builds are missing features.

Next to test if TrueNAS CORE (BSD based) will detect the emmc for installation of the OS.

[defcon]

test if TrueNAS CORE (BSD based) will detect the emmc for installation of the OS

It will. I tested FreeNAS initially, different SBC though.Then OMV and Unraid and next going back to TrueNAS (bsd).

M1015 is 8x pcie 2.0 8 port SATA/SAS

Its not for Odroid H2+, right?

[brad]

test if TrueNAS CORE (BSD based) will detect the emmc for installation of the OS

It will. I tested FreeNAS initially, different SBC though.Then OMV and Unraid and next going back to TrueNAS (bsd).

M1015 is 8x pcie 2.0 8 port SATA/SAS

Its not for Odroid H2+, right?

It is actually, the card can run with PCIE x4 bus. Although it will mainly serve the 4 x SSD's i'm also thinking about 2x HDD's as well for another storage pool and maybe a hot SSD spare drive.

I was also going to consider testing out the raid 10 function of the card, as RAID 10 has mirrored disks only half the bandwidth is needed on the PCIE interface.