Topic: Library Expansion and Protection of large libraries (Read 4563 times)

Hilton · « **on:** January 13, 2019, 02:48:59 am »

Hi All

I've almost filled up my 24TB Hardware RAID5 pool.

After adding lots of 4K content and having concurrent read/write buffering issues on the network, I've decided it's time to make some changes for;
1. better performance,
2. better protection, and
3. setting up for easier growth.

So...

I've already played around with FlexRaid RAID-F and RAID-T and after having reliability issues and poor performance in testing (was even slow with a RAID-T SSD Landing Disk) and the overly complex management of those solutions, I've decided to go down the Drive Bender path.

Fixing Performance
I also installed and played with Stablebit DrivePool but didnt like the interface at all. DriveBender is just so much nicer and easier and performance is fantastic, achieving native disk performance speeds.

I've fixed my immediate performance issues with buffering when concurrent read/write are going on, by converting my existing 24TB Hardware RAID5 to a Drivebender Pool (with data inplace) and adding a 500GB SSD landing disk. (write cache)
This has significantly improved concurrent read/write access to the media library.

Now I can happily RIP content to the library and it goes straight to landing disk, which doesn't affect any other streaming going on in the house. After midnight it flushes from SSD to disk.
Problem 1 solved.... for now

Data Protection
The current Library is protected by Hardware RAID 5 and has been added to a DriveBender pool with a SSD landing disk in front of it. That's pretty good protection but I'll improve on that in the next couple rounds of changes.

Backups
The next immediate problem is backups, so I've now put my old 8 bay Hardware RAID box into Spanning mode. It works a little like DriveBender in that it fills up one disk and then moves onto the next. It's hardware based but it's better than RAID because I can pull any individual disk out of the RAID Box and access it on any PC.
It has 24TB made up of 3TB disks and I'm using Rich Copy 4 with a saved profile to create a backup job that copies the media library to the backup drive with flag set for "if timestamp is newer" .
It overwrites older copies but, otherwise it skips files.
Data Protection and Backups solved.... for now... and it's fast.

Gearing up for simple growth
So now comes the new harder problem to solve and I think I have a plan that I'm happy with that I'd thought I'd bounce of you guys before I commit to it as it's rather expensive to put the new solution in, but it's cheaper to expand and easier to maintain, while still keeping all the benefits of pooling HDD of different sizes. (something my hardware RAID solutions dont currently do)

Proposed Hardware
Silverstone RM420 Server/Disk Case. (~$800)

rm420-34right-top by Hilton, on Flickr

Adaptec 24i SAS HBA (~$1000 with cables)

Adaptec-24i-SAS-HBA by Hilton, on Flickr

That will give me 20 hotswap drive slots to play with on the main media library.

The Transplant
Existing Workstation/Media server transplanted into RM420 Case.
Plug in the Adaptec HBA and backplane cables.

Ensure I have the latest backup of media library.
Format all the RAID5 drives and put them in the new RM420 case.
Configure DriveBender with 4x8TB drives and the 500GB of SSD for landing disk.

Restore data from backup to the new Pool.

For expansion with this new solution I can just buy new 8 or 10TB drives as needed in pairs, one for library and one for backup pool.
I need to also look at a better backup pool solution so will likely have to do the same build again for another server to put in another part of the house.
I know I can get the Norco case for about half the price, but i've read about a few issues with sloppy backplane construction on those cases.

Thoughts? Is there another way to achieve a similar outcome without all the cost?

After looking at all these costs, I wonder if I might be better off spending the money on better AV gear instead and just live with streaming and plastic discs!
It's getting pretty expensive to store raw 80-100GB 4K movie RIPS!
It's a tough one! Convenience of instant playback from your own library vs streaming and disc swapping.... I'm very torn!

Cheers

Write performance to library while doing a backup at the same time. 400MB/s Writes and upto 150MB/s Reads

current performance by Hilton, on Flickr

Hilton · « **Reply #1 on:** January 13, 2019, 04:57:06 am »

I found a couple other more cost effective solutions. They are not as space efficient, but they would still work in a Rack.

Orico 8 Bay USB3 JBOD Enclosure - about $400 - looks very nice.

Orico-8bay by Hilton, on Flickr

ICYBox 10 Bay USB3 JBOD Enclosure - about $500 - better rack fit.

10bay-ICY by Hilton, on Flickr

and... a JBOD version of my current 8 Bay RAID Box for about $350. (I have a 4 bay and 8 Bay RAID version of this that is over 5 years old and hasnt skipped a beat)

H82-SU3S2-f1 by Hilton, on Flickr

As much as I love the idea of redoing my workstation/media server case with 20 bays and a 24 port SAS HBA these look like much better propositions...

I dont need anywhere near 20 bays yet but I could buy 2 x 10 bay ICY's and attach one as the primary library and the other as backup, and they'd both fit in the rack and I could store one offsite at work. hmmm

Manfred · « **Reply #2 on:** January 13, 2019, 11:55:24 am »

Quote

adding a 500GB SSD landing disk. (write cache)

I have the same solution.

Data Protection
I have a Raid (Win 10 Storage Space Mirroring) solution only for Audio, Download from Mediathek etc. but not for movies & TV series. I fill my 10 TB disks and then copy movies & TV series to old disks and put them in a different location. I also have the disk as DR solution.

Quote

After looking at all these costs, I wonder if I might be better off spending the money on better AV gear instead and just live with streaming and plastic discs!

I asked this question myself several times. Since one year I did not bought any movie disk anymore, instead I used a streaming service. I bought some TV series and music concerts and ripped them. I have no UHD disk. If I would have not a long time history in buying BD's and DVD's I think my solution would look like different compared to my current solution. One 4 TB ssd would be sufficient for my media content and the rest I would stream (movies & TV series). You asked a good question and I am asking me always the same does it make sense to rip movies & TV series and store them. I personally have not found a final answer.

RoderickGI · « **Reply #3 on:** January 13, 2019, 08:32:59 pm »

Are there any USB 3.1 Gen2 USB-C solutions out there yet? That is what I would be looking for, even though hard drives aren't going to max out the USB 3.0 5 Gb/s throughput theoretically, we all know that practice doesn't follow theory.

It seems there are Thunderbolt 3 40 Gb/s solutions out there already. Why not USB 3.1 Gen2 10 Gb/s? Or "don't hold your breath" USB 3.2 20 Gb/s.

Aside from that, I think you have the options pretty much covered. Cloud backup could play a part though. Plus I would be going for the largest additional or replacement drives I could. 12TB gives a real 10TiB drive, and I like the idea of that!

I like the DriveBender solution over the alternatives you mentioned. Although it doesn't provide any data protection, does it?... Oh wait, it has folder level Redundancy. Doubles space requirements but would protect against a single drive failure. They don't mention a rebuild process to duplicate lost files though. Hopefully that happens when a disk is replaced.

If only money didn't matter, we could go all out and all run datacentres at home!

Hilton · « **Reply #4 on:** January 14, 2019, 02:57:57 am »

@RoderickGI

Interesting that you made that point - after I logged off from the forums last night I did look into USB-C and thunderbolt 3 enclosures and the short answer is there's not much out there that's USB 3.1 Gen 2 USB-C and thunderbolt 3 is a lot more expensive than even the Adaptec HBA and new case solution.

I did find an 8 bay JBOD USB-c gen 2 enclosure that is made by the same brand as my current RAID boxes for just a little more than the USB3.0 JBOD I posted above. ($369USD plus shipping)

So that's now my preferred route with a new $90 dual port USB-C card. That will give theoretical throughput of over 1GByte/s transfer rate if you had fast enough disks.
(ASM2142 chipset on this card is good for 16Gbit/s on PCIE3 x 2 slot to give full 10Gbit performance across the two USB-C ports)

That's less than a quarter of the price of the 20 bay SAS enclosure solution above. (and less than half the price for 16 bays) Problem solved! Back to USB-C JBOD boxes.

Mediasonic ProBox USB-C Gen 2 8 x bay JBOD (same brand as my current 4 bay and 8 bay RAID boxes) $369USD

ProRaid-USB-C 3.1 Gen2_ by Hilton, on Flickr

Startech 2 Port USB-C Gen 2 Card $90

2 port USB-C Gen2 controller by Hilton, on Flickr

Hilton · « **Reply #5 on:** January 14, 2019, 03:27:00 am »

Looks like the MediaSonics are good for 800MB/s possibly more on USB-C 3.1 Gen 2. They also use a ASMedia chipset in the box.
That's good enough for me!

Order's placed! $687.34 AU$ shipped from Amazon store and $89 for USB-C card from a local AU supplier.

HUR6-SU31 Raid 0 SSD by Hilton, on Flickr

RoderickGI · « **Reply #6 on:** January 14, 2019, 05:05:05 am »

That all looked too good to be true... until I realised those were US$ !

I couldn't find a User Manual for that Mediasonic H82-SU31C. Did you find one? Not much in the way of specs either. But I guess you have experience with the brand so you are confident with the purchase. There was a time I would touch anything from Mediasonic, but I guess they have improved.

So how many 12TB disks are you going to put in it? Plus an SSD. That is when the cost really starts to add up!

PS: Amazon is going to kill local tech retailers for this sort of high value, low volume device, even with the exchange rates and those shipping costs.

Although, take a look at this on PCCaseGear, and compared to this on Amazon AU. It's not the USB 3.1 Gen 2 version, it is rebranded, but the pricing is close, without the shipping cost.

Maybe it still pays to search Australian sites. But PCCaseGear doesn't have the USB 3.1 Gen 2 version, and I couldn't find it locally anywhere else either. So, Amazon it had to be.

PPS: I also noted that the USB 3.0 version has been around with improving capabilities (i.e. supported drive capacities) since 2011. They say technology moves fast. Not with USB take-up. I wonder when we will see USB 3.2 devices and enclosures. It has been defined for a while now.

Hilton · « **Reply #7 on:** January 14, 2019, 05:21:20 am »

Yeah I was hoping PCCaseGear, Scorptec, MWave or others might pick this new USB-C 3.1 version up.
I was nearly going to go for the USB3.0 JBOD Mediasonic but I also want a little more performance out of it. (my 4 bay maxed out @ 250MB/s with 4 x WDBlack 1TB disks in RAID 0)

Just to be clear, that Mediasonic disk performance chart was for a new generation 2 bay USB-C but they generally use the same Chipset's across the range just with different drive slot configurations.

Yeah thinking about it, my current boxes are nearly 10 years old for the 4bay and 8 years old for the 8 bay. I've never had a problem. Not even a single disk failure, though I have replaced the fan on the 4 bay.
The 4 bay is so old it even has firewire800 on it from memory.

PS> I even have one of their 4bay docking stations that's a few years old and it hasnt missed a beat either. It support 3.5 and 2.5 drives including SSD. (no manual anywhere online) I'll be putting 4 x 8TB and see if I can get an existing SSD or 2 in there with a 3.5 to 2.5 adaptor of some sort.

RoderickGI · « **Reply #8 on:** January 14, 2019, 06:15:38 am »

Quote from: Hilton on January 14, 2019, 05:21:20 am

I was nearly going to go for the USB3.0 JBOD Mediasonic but I also want a little more performance out of it.

It looks like the USB 3.0 version only support 8 or 10 TB drives as well, depending which web site you read, and not 12 TB drives that the USB 3.1 Gen 2 supports.

I did think that performance chart was for a USB 3.1 Gen 1 enclosure, based on your comment. But on re-reading that makes more sense, because Gen 1 at 5Gb/s should max out at 625 MB/s, or more likely 500 MB/s, while Gen 2 would have a theoretical max throughput of 1250 MB/s, with a more likely max of 1000 MB/s. So a little upside if there are any faster SSDs to test with!

This is all good. You are giving me ideas for future upgrades! Well, maybe that is bad, because upgrades cost!

Hilton · « **Reply #9 on:** January 14, 2019, 07:12:24 am »

Do it... you know you want the speed.. and capacity

I've ordered 4 x icybox 3.5 to 2.5 backplane adapters to mount some SSD's to test performance.
With my existing 4x 120GB Sandisk SSD's can hit just over 900MB/s on Intel RAID 0 Sata II ports on the motherboard. (no where near the 2500MB/s of my Intel NVME drive.

)

So theoretically the USB-C 3.1 JBOD in a software RAID 0 configuration should be about the same, as long as they are using SATA III chipset in the box and the Sata III to USB-C bridge chip can keep up.

Anything over 200MB/s with spinning disks will be good enough for a media server with a write cache SSD in front of them.

Hilton · « **Reply #10 on:** January 18, 2019, 07:33:03 am »

Hi there

We'll the good news is the new usb-c enclosure and SSD adaptors arrived. The performance is outrageous... infact too good to be true so I ran two different performance tests and got the same performance. wait for it..... 1.6GBytes a sec with Crystal diskmark and 1.7 to 2.1GBytes seqential reads a sec with Atto using drivebender with SSD cache. (sequential writes a peak of 846MBytes /s )
(that's using USB-C gen 1 on my surface book - the USB-C Gen 2 card for PC hasnt arrived)

Something cant be right. I used 2GB file with ATTO and 4GB file with Crystal tests.

My spinny WD Reds managed 177 to 180Mbytes sec reads and write.

drive bender probox usb-c perf by Hilton, on Flickr

Usb-c gen2 probox by Hilton, on Flickr

RoderickGI · « **Reply #11 on:** January 18, 2019, 08:11:32 am »

The tests must have been reading from multiple SSDs to get those results.

It looks like you have four installed. The tests probably pick the fastest drives in the pool. It looks like only two of your SSDs have data on them, each with about 1GB of data, and all other drives are effectively empty.

The write figures for the F: drive test would also indicate one or two SSDs were being used.

Besides, if you have the SSDs set up as the landing zone, then that's where the test data would be written. Then as the data in landing zone hadn't been moved to the hard drives yet, the data would be read back from the landing zone SSDs.

The result is still outrageous, because the USB 3.1 Gen 2 @ 10 Gb/s can only support 1.25GB/s tops. Anything over that makes no sense, that I can think of right now anyway... unless the test data was being cached on a local SSD and never sent to the enclosure. But still, and SSD doesn't support 2GB/s... but wait, don't you have a fancy high-speed memory SSD in your PC?

Hilton · « **Reply #12 on:** January 18, 2019, 08:35:06 am »

I agree the results dont make sense. I think Drivebender must be distorting the figures, maybe the reads are coming from the drivebender driver memory cache. Anyway when I get the USB-C card in my PC I'll do more proper testing.

A quick test from my NVME drive to the new USB-C plugged into USB3.0 on my PC gives me 170MB/s without a landing disk configured. I've started restoring my data and will look for a local USB-C card tomorrow.

Hilton · « **Reply #13 on:** January 21, 2019, 04:08:35 am »

Hi - Well I have some more realistic results now with the USB 3.1 GEN 2 USB-C card.
USB-C 3.1 Gen 2 is the real deal.. nice and fast.

Below are all Write transfers from NVME to the Enclosure with different numbers of threads copying to individual disks.
Reads are about the same.

8 x transfers from NVME drive to 4 x SSD and 4 x 8TB WD REDs maxed @ 720MB/s
4 x transfers from NVME drive to 4 x 8TB WD REDs maxed @ 600MB/s
2 x transfers from NVME drive to 2 x 4TB WD REDs maxes @ 360MB/s

DriveBender Flushed SSD Cache Disk to HDD @ 180MB/s to a single HDD (about 60MB/s faster than when using a USB3.0 interface for the enclosure)

Overall disk throughput increased about 50-60MB/s per HDD from 120MB/s to 180MB/s per spinning disk.

My Intel NVME is capable of sequential reads 2.5GB/s

SANDISK SSD are faster on the Intel Chipset than in the enclosure because I dont think the enclosure supports TRIM.
The SSDs were only writing at about 130MB/s when each one is capable of 400MB/s Writes.

Probox-8x transfers usb-C Gen2 720MBs by Hilton, on Flickr

Probox-4x transfers usb-C Gen2 WD RED 8TB 600MBs by Hilton, on Flickr

INTEL NVME

NVME by Hilton, on Flickr

RoderickGI · « **Reply #14 on:** January 21, 2019, 05:26:54 am »

Quote from: Hilton on January 21, 2019, 04:08:35 am

I dont think the enclosure supports TRIM.

That is a bit of a worry, from a reliability and longevity point of view.

Quote from: Hilton on January 21, 2019, 04:08:35 am

The SSDs were only writing at about 130MB/s when each one is capable of 400MB/s Writes.

And that is a big drop for just not having TRIM. Unless your SSDs were already chokers with data.

Hilton · « **Reply #15 on:** January 21, 2019, 05:49:06 am »

Quote from: RoderickGI on January 21, 2019, 05:26:54 am

That is a bit of a worry, from a reliability and longevity point of view.

And that is a big drop for just not having TRIM. Unless your SSDs were already chokers with data.

It was @ 90% when testing. I'll do some more testing another time and run the trim check tool on it.

PS. researched and another of their USB-C Gen 2 enclosure definitely doesn't support TRIM so unlikely this one does. Never mind - I wasn't going to use SSD's in it anyway.

Hilton · « **Reply #16 on:** February 16, 2019, 08:42:18 pm »

Hi All

I was having random drives dropping with Drivebender. I don't know if it's the USB-C enclosure, USB-Controller, the drives or Drive bender, but I've migrated the pool over to stablebit drive pool to see if behaves any better. (There are no drive defects detected with Hard Disk Sentinel Pro)

Migration was simple, just moved the contents from each drives drivebender GUID to the new drive pool GUID. Will keep you posted if drives behave better.

Hendrik · « **Reply #17 on:** February 17, 2019, 02:30:58 am »

I use StableBit Drivepool myself and haven't had any complaints or problems of drives disappearing.

jmone · « **Reply #18 on:** February 17, 2019, 02:15:33 pm »

I use Drivepool as well and never had an issue with drive disappearing (or when I was on Drive Bender before that) but I'm all Sata. There was a post by the Drivepool mod awhile ago about issues with USB drives dropping out in general (I did try one of those Icey docks at one point).

Hilton · « **Reply #19 on:** February 18, 2019, 04:48:21 am »

Fingers crossed - no issues in over 24 hours with a fair bit of work going on in the pool.. Stablebit hasnt thrown a single error.

Drivebender never went more than 24 hours without a drive dropping and file enumeration cache going offline.

Looks like Drivebender's driver or an incompatibility with windows 10 or my configuration is at fault.

I read up recently before I made the switch that Stablebit Drivepool (SDP) uses it's own in house built kernel mode driver where as DriveBender (DB) uses a user mode driver used by others that could cause issues that they would be at the mercy of the original developer to remedy. Where as Drivepool can make changes and adapt their driver. (and they do)

Snippet from a reddit thread where someone from codecove (stablebit developer) was talking about performance differences between DriveBender and Stablebit DrivePool

"We use a kernel mode driver, while Drive Bender uses a user mode driver. Additionally, our driver is build entirely in house (so we have complete control over it, for better and worse), while Drive Bender uses a prebuilt solution (ElDos, IIRC, which is super common)."

Bottom line - Drivepool handles my USB-C enclosure just fine and hasnt error-ed with more than 1TB being moved around at sustained throughput while normal file balancing and file playback from the library was occuring.

I did notice DrivePool doesnt push disk utilisation to 100% like Drivebender did though, which means it's a little slower.
~~Drivepool sits around 85% to 93% disk utilisation and at about 100MB/s to 115MB/s where DriveBender just thrashed everything at 100% disk utilisation hitting upto 180MB/s per disk.~~ I take it back - the 100 to 115MB/s was to one of the 4TB green drives, the 8TB Reds are hitting 160MB/s. Performance of DrivePool is the same or better.

USB based controllers probably don't like being bashed at 100% for extended periods without some allowances in the virtual disk driver for a USB error or timeouts which was probably causing issues on DriveBender. (non of the internal disks ever dropped, it was just USB-C disks)

Anyway Stablebit looks to have been the better solution for my setup. (its also lighter on CPU utilisation by A LOT)

Hilton · « **Reply #20 on:** February 20, 2019, 03:52:28 am »

Stablebit DrivePool has been... well.. stable..

no issues at all.

So that confirms the USB-C enclosure and USB-C card are a great solution for fast access to loads of media.
Also no buffering issues on the streaming to any devices anymore no matter what's going on in the library
Ripping UHD disks, moving data, balancing, and just generally dumping data have not caused a single frame drop or stutter.

Previously, even 1 sustained write job to the Hardware RAID 5 array would bring streaming to it's knees on just a single 4K stream.
Now multiple 4K streams and general library maintenance and ripping are not a problem.

Very Happy!

Absinthe · « **Reply #21 on:** November 20, 2019, 01:13:11 pm »

Quote from: Hilton on January 13, 2019, 02:48:59 am

I've fixed my immediate performance issues with buffering when concurrent read/write are going on, by converting my existing 24TB Hardware RAID5 to a Drivebender Pool (with data inplace) and adding a 500GB SSD landing disk. (write cache)
This has significantly improved concurrent read/write access to the media library.

Now I can happily RIP content to the library and it goes straight to landing disk, which doesn't affect any other streaming going on in the house. After midnight it flushes from SSD to disk.
Problem 1 solved.... for now

Hilton,
Similar to you I have several utilities running on my media server that insist on writing to local SSD drives and do not behave well when writing to a NAS location (despite the fact that the transfer rate is several orders of magnitude higher than the download rate from the streaming server). Therefore, I would like to implement a "landing drive" and have the utility move them to the NAS during off-peak hours. However, the drive pooling capabilities of drive bender would go totally unused. Are there other utilities you would recommend that would use a landing drive and move the files to the NAS at a later time? $49 isn't that bitter a pill to swallow but I hate buying software when I'm only using less than half of its capabilities. I didn't see this capability on StableBit. Is it capable of managing a landing drive?

INTERACT FORUM

Author Topic: Library Expansion and Protection of large libraries (Read 4563 times)

Absinthe