Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[jmone]

Mmmm - looks like dev behind DriveBender is winding down development and moving on.

http://blog.division-m.com/2015/11/19/all-good-things-must-come-to-an-end/

For me it is till going strong (Win10 and Win Server 2012 Essentials) and it looks like we will get another release soonish prior development ending.  While it is not an immediate concern, I'll be on a lookout for a replacement (if someone else does not pick up the code) as at some point I guess it will stop working when MS makes some Windows change.....  

[jmone]

http://community.division-m.com/index.php?/topic/5664-a-sad-day/

thesmith, on 22 Nov 2015 - 09:37 AM, said:
Hi all, and thanks for the kind words. I will be around the forum for some time to come...  and our plan is to ensure that all product will continue on, just working out the details. As for activation... don't worry, I would never leave users without the ability to user their software   Anyways we are getting close to announcing what will happen moving forward... stay tuned!

[marko]

Thanks for the heads up. It's just there, doing it's job for me and the ten terabyte drive it gives me. twenty quid well spent for me.

[jmone]

Yup - I have two pools both around 40TB each.  No fuss, just works.  Lets see how it goes

[jmone]

Still have a niggling concern at the back of my head some "Windows Update" will break my pools and given it is now donation ware support may be limited.  So I'm having a proactive look around and asked over at the StableBit DrivePool forum some Q's

[InflatableMouse]

For those looking around for a worthy alternative and not afraid of getting their hands dirty, the ZFS filesystem is really one of the best that is out there. It doesn't work with Windows though so that might be a show stopper for some people. MediaCenter runs quite well on Linux though so that's not an excuse anymore.

I've been running ZFS for several years now and have installed and configured it for friends too. It has proven to be extremely stable, reliable and saved my life data too when a power supply in my server turned out to be insufficient and caused drives to fail with sata errors (and I thought drives were failing so I kept replacing them until a 3rd one failed during a rebuild and the pool went down). During all those rebuilds and drive failures, ZFS kept my data safe and simply did what it had to do.

ZFS is actually an enterprise class filesystem by Oracle, build for their Enterprise storage servers. It is open source though, several distributions include it, like ZFSGuru, FreeNAS, NAS4Free and FreeBSD. These are all BSD-variants as far as I'm aware. On Linux, it can be built from source or installed from 3rd party repositories on Debian or Ubuntu. It runs well on ArchLinux too but I can't recommend that unless you're experienced with Linux.

Some advantages:
- zero maintenance
- self-healing
- dynamic stripesizes
- advanced snapshotting
- build in sharing on NFS and SMB
- can safely use SSD's for caching (ie, no corruption on power failures or failing cache devices - integrity garanteed)
- it's FAST!

ZFS doesn't require special hardware, it works on regular SATA disks with regular controllers (ie, no special raid controllers or NAS-approved disks required), is independent from hardware (plug and play on other hardware, import pool, done (yes, I've done that several times already)).

Some disadvantages:
- no official support unless you buy a server from Oracle (there is unofficial forum support when using a distro with ZFS support built in, like freeNAS or ZFSguru)
- doesn't work on Windows
- steep learning curve when you're not familiar with Linux (ZFS itself is quite easy though)
- memory hungry (roughly 1GB per disk is ideal, but can work for larger pools with 4 to 8 GB. More is better, though).
- no expanding of a pool by adding 1 or 2 disks. Ie, you can't expand a 3-disk RAIDZ1 (RAID-5) set with 1 or 2 disks. Basically, expansion is only possible by adding the same amount of disks (vdevs), so a 6-disk RAIDZ2 (RAID-6) will require doubling the pool with another set of 6 disks. Very unpractical so usually, this means you'll be adding pools instead of expanding them, or rebuilding a pool to expand it. Requires some planning so keep this in mind if you're making the switch.

Oke, I'll shut up now or I'll keep rambling about it. Didn't want it to become a long winded sales story. Obviously I'm a big fan of ZFS . If anyone is interested I'll try answering your questions and help you on your way. I'm NOT familiar with any of the *NAS distro's though, I've only ever run ZFS on home built Linux like Debian, Ubuntu and ArchLinux. I have no idea whether it's possible to run MC on a *BSD-like OS, problably not. If you need MC running on your storage server then you're pretty much limited to Linux, Debian for instance. ZFS runs well on Debian, MC is officially supported on it, too. It does require you to manually configure from the commandline (which is really not that hard ).

[jmone]

Converted one of my pools to StableBit Drive Pool and it was easy, quick and did not generate any issues.  It also has a bunch of Balancing Options that I'm playing with to see how they work (eg only let contents of my "Video Editing" folder be stored on my faster HDS 7200 drives rather than my Seagate 8TB Archive Drives).  So far pretty pleased but it is only Day 1

---------------------
Re: ZFS, nice write up by InflatableMouse.  After MS Pulled their Storage Pool tech from Home server and my raid card on my Main PC scrambled my HDDs a few years ago I had a look at the options and liked what I saw with ZFS for fault tolerance and performance but needed:
- Windows Platform support (both Desktop and Server OS)
- Easy way to add, remove, swap, replace drives etc all on one mount

So I settled on DriveBender and looking at StableBit Drive Pool to do the "Pooling" of the HDD.  I've also decided to switched from using NTFS to ReFS for the drives in these pools and touch wood, I've not had single issue with corruption since.  Running CHKDSK is a thing of the past!  So for me ReFS is the actual File System, with DriveBender / Drive Pool doing the HDD Pool Mgt over the top.  Nice combo but it only works on Windows and ReFS is not for all uses (eg you can not boot Windows from them) but is well suited for file storage.

[InflatableMouse]

... but needed:
- Windows Platform support (both Desktop and Server OS)
- Easy way to add, remove, swap, replace drives etc all on one mount

Not going to try and convince you otherwise so don't worry  . Important thing is that you're happy and comfortable with what you're running and that you can manage it in a way that is convenient for you.

So purely out of interest, why do you need Windows on your server? A desktop I can understand, a server however should simply be accessible for clients over the network, regardless of its own or its clients' operating system.

Regarding not being able to boot from ReFS, I wouldn't consider that an issue. In fact, I would personally never even consider placing an OS on a storage pool of disks. In any case, I consider it best practise to keep the OS separate (not just partition, but separate disk even) from storage pools. I want my OS on a basic partition, on a compatible filesystem for recovery purposes.

Did Drivebender or stablebit support snapshotting in a similar way ZFS does?

[jmone]

The only thing I use my Win Serv 2012 R2 Essentials (catchy name   ) for is backup (and I upgraded from WHS):
- Incremental Daily Backups of all my Win PC excluding my pool (unique clusters are collected each day only)
- I periodically preview then commit changes from the main pool to the one on the server
- Only on during backups (else in Standby)
- In a physically separate location to the Main PC in case of fire, theft, acidental change or deletes etc

Both Drivebender and Stablebit DrivePool support "Dupilcation" of all/selected content and some run SnapRaid or FlexRaid at the same time.  I've not looked at either as I have the full backup on the Backup server so it would be a waste of space and another level of complication.  ReFS itself is just the filesystem, with Drivebender / DrivePool doing the Pool Mgt part.  ReFS itself does no multi disc mgt but MS's Storage Spaces requires it if you want to use their attempt of Pool Mgt.  

I do have my first Ubuntu box however (dedicated Ubiquity WiFi controller and NVR) - so I'm getting to grips with that.  Still no idea how to back this one up like I can my PC's and not getting much feedback either from their community!

[RussellS]

Here is some better news regarding DriveBender, Cloud Xtender and PoolHD -  http://blog.division-m.com/2015/12/02/drive-bender-cloud-xtender-and-poolhd-live-on/

[mwillems]

The only thing I use my Win Serv 2012 R2 Essentials (catchy name   ) for is backup (and I upgraded from WHS):
- Incremental Daily Backups of all my Win PC excluding my pool (unique clusters are collected each day only)
- I periodically preview then commit changes from the main pool to the one on the server
- Only on during backups (else in Standby)
- In a physically separate location to the Main PC in case of fire, theft, acidental change or deletes etc

Both Drivebender and Stablebit DrivePool support "Dupilcation" of all/selected content and some run SnapRaid or FlexRaid at the same time.  I've not looked at either as I have the full backup on the Backup server so it would be a waste of space and another level of complication.  ReFS itself is just the filesystem, with Drivebender / DrivePool doing the Pool Mgt part.  ReFS itself does no multi disc mgt but MS's Storage Spaces requires it if you want to use their attempt of Pool Mgt.  

I do have my first Ubuntu box however (dedicated Ubiquity WiFi controller and NVR) - so I'm getting to grips with that.  Still no idea how to back this one up like I can my PC's and not getting much feedback either from their community!

jmone; incremental and differential backups are really easy on Linux and there's a huge buffet of good choices, based on your post over on ubuntu I'm not sure if you're looking for incremental backups or differential backups?  If you're looking to maintain a single backup but only want to update the changed parts after the initial sync, plain old rsync will do the job perfectly.  If you want actual differential backups or snapshots there are a ton of tools both rsync-based and non-rsync based.

Here's a partial list of backup solutions from the Arch wiki (most if not all of these solutions are available for Ubuntu as well): https://wiki.archlinux.org/index.php/Rsync

You can use one of the incremental rsync solutions like rdiff or rsnapshot, or one of the modern deduplicating solutions like attic or borg.  But if you're just trying to keep a copy in case of hard drive failure, it's really hard to beat rsync for ease and speed, the arch wiki has a good article on full system backups with Rsync: https://wiki.archlinux.org/index.php/Full_system_backup_with_rsync

I've used the instructions there on both arch and non-arch distros (rsync is the same the world over) and have successfully restored from a backup made in that way.  It also has the advantage that you can just grab individual files out of the backup if you need them.  If you need compression you can always pipe it into tar.

[InflatableMouse]

I shared something similar with jmone over PM about rsync.

I just noticed that my favorite way of backing up linux boxes is referenced from that archwiki link you shared; rsync snapshots using hardlinks using --link-dest. Didn't know that was on there . Very cool script.

The only problem with rsync I need to point out is that during the creation of a backup, the filesystem is not in a frozen state; it's live. Any program or process is able to make changes while rsync runs, which could create a problem if a program is in the process of updating files and rsync flies by creating a backup in the middle of it all. This would leave half the files in the updated state and the other half not updated. This can mostly be corrected by running the same rsync command a second time, at which time it will complete much faster (usually seconds, as it just updates the updated files since the last backup). For some applications this still isn't enough (think databases) and restoring those could break the application. I never had that problem though and my restores with rsync always worked fine so far.

Should also mention that an rsync backup does not create a bootable, functional backup of a system. Could use dd to dump the boot sector and partition tables and restore those, or (my preference) create a new partition scheme, restore the rsync backup and chroot into the restored system to reinstall the bootloader and edit fstab.

[mattkhan]

slightly OT but I read this article recently and, as a longstanding rsync user, thought it was interesting -> http://arstechnica.co.uk/information-technology/2015/12/rsync-net-zfs-replication-to-the-cloud-is-finally-here-and-its-fast/

[jmone]

thanks for all the rsync advice - looking forward to trying it out once I've finished testing StableBit DrivePool.  

DrivePool/DriveBender:  I've seen the posts about DriveBender's future and I'm sure it will work great for a long time, and Anthony will keep it ticking over....but.... I'm just not comfortable with using donation ware on my 40TB Pools holding a couple of million items.  My fear is that MS will push out some "patch" that breaks something with DB and Anthony will just not have the time or inclination to fix it.  It's probably unlikely as I've had DB working just fine on Win7, 8, 8.1, WHS, WS2012R2 etc, but I'll feel better paying for a new DrivePool licence if it all works.

I'm currently trying all the usual stuff in DrivePool:
- Moved content from the DriveBender to the DrivePool (easy, quick, no probs)
- Added Drive
- Remove Drive (underway)
- Rebalanced files across different HDD in the pool
- Tried Rebooting in the middle of these activities, renaming disks etc to see if I could cause any issues

I keep checking the contents against my backup pool and 100% good!  It is nice and stable - no crashes, fuss etc

My current impressions of DrivePool vs DriveBender is:
+'ve: Much faster to add a new Drive (it does not need to copy the whole structure into the new drive first)
+'ve: It has "Balancer" Plugins that give great flexibility to setup what you want where.  I'm going for a:
         Fill Drive 1, then 2 etc (instead of balancing).  Fill up my 8TB Seagate Archive Drives first (as they are slow writes but fast reads).  
         Keep the TV Recordings / Video Editing / Virtual HDD etc on my faster HDS 4TB 7200 drives
         No Duplication across drives (I have a second pool that I backup to - but most seem to duplicate)
+'ve: When rebalancing, changes drives etc you can leave it in a "background" thread or press a button to "increase priority"
+'ve: Much Much better Memory usage on my system (I have an additional 10GB Free!)
+'ve: More active forum with daily answers to Q from the main forum admin "Christopher (Drashna)"
+'ve: Better handling of when a Drive is considered "Full" and can set it as low as 1GB or 1% without issue

Same: Seems just as fast when streaming, copying in / out of the pool

Different:  Like all new progs it takes some time to workout what it does etc compared to your last one
Different:  You use normal windows tools to name drives, assign Drive Letters / Mount in folders etc instead of the UI
Different:  If you use the default "Remove" drive function it puts the pool into a Read Only State.  Given the size of drives this could be a couple of days which is less than ideal (MC complained the TV Recoding folder could not be written to).  There are a couple of options:
     - There are a bunch of "Advanced Options" such as disabling this "Read Only" behaviour in a config file (not through the UI)
     - Other option is to use the "Rebalacer" to move the files first and when mostly empty then I'll "remove" the drive

-'ve: Their UI reports less statistics on what it is doing when doing a Rebalance or Drive Remove.  To see what is actually happening I'm using Windows Resource Manager
-'ve:  No explicit support for ReFS formatted drives.  It is working for me just fine, and they have done some testing but warned it was not fully tested.

I'll update this post as I find out more

Thanks
Nathan

[jmone]

FYI I asked Christopher (Drashna) at StableBit DrivePool to check my post above and he responded with a bit more info:
======================================================================
Well thank you!
 
A couple of things to clarify, when adding a disk to the pool, StableBit DrivePool doesn't do ANYTHING with the existing data. Meaning that you can continue to use the disk as it was without having to dedicate it the pooling software.
 
If you do want to move data into the pool, we do have a "Seeding" guide, which is basically moving the contents into the hidden PoolPart.GUID folder and remeasuring. http://wiki.covecube.com/StableBit_DrivePool_Q4142489
 
Also, StableBit DrivePool doesn't require drive letters or mount points on the drives. You can remove any mount on the drive, if you want, and the software has absolutely no issues with this.

But yes, you have to manually change things around for this. (but a big part of our design philosophy is to be very "hands off" on your system, and to allow users to use the system the way *they* want).

And we do have a guide on how to mount the drives to folder paths, if you do want to remove the drive letters: http://wiki.covecube.com/StableBit_DrivePool_Q4822624
 
 
And while we don't officially support it, we do have a number of users that use FlexRAID or SnapRAID to get snapshot parity support. And have posted about it: http://community.covecube.com/index.php?/topic/52-faq-parity-and-duplication-and-drivepool/
The second post contains a list of threads that talk about how to set it up.
 

[jmone]

So I've finished testing on my "Main" pool and am happy with the DrivePool results (especially the much lower RAM usage and the ability to prioritise what goes on what drive).  Anyway, I've now moved onto testing on my Win Server 2012 R2 Essentials "Backup" Pool as part of a new recase of the existing HW into a Norco 4220 rack mounted case.

[jmone]

Now moved to DrivePool for both pools.  All going fine so far.

[marko]

Thanks for the running commentary, it's appreciated.

It also serves as a reminder for me that I need to attend to this too.

Do you know if DrivePool supports VSS?

[jmone]

I'm not sure, I take you want to use windows to backup your pool and hence need VSS?

[marko]

Backup manager, which insists on VSS.

I'm getting round it atm using GoodSync, but would prefer to use the other.

[Hilton]

I'm still using hardware RAID 10 with full and incremental backups to 4TB drives in a 4 bay dock.  Thinking about moving to a drive pooling solution with data duplication instead of backups. The problem with duplication instead of backups is that dupe programs are very good at copying corrupt or virus infected data, which doesn't help you!  Backups you can at least roll-back.

Nathan why didn't you consider F-RAID or T-RAID which has pooling, parity protection and snapshot schemes?  You'll have to excuse my enterprise storage bias from working 10 years with $1M dollar storage arrays.  FlexRAID seems like the best of both worlds to me.  From my understanding you can pool individual drives and create a parity drive (JBOD + Parity or multiple parity drives) You can create storage tiers too.

PS:- is this a fair comparison of the available software based parity solutions?

http://www.snapraid.it/compare

[jmone]

Good Q: I had a look at the various options a few years back after my HW Raid Controller, Windows, and CHKDSK had a fight and decided to reallocate the blocks of all my files.  It was a fun month re-ripping all my Disks, and decided that it was with paying the extra to have a real backup.  Prior to that I'd tried single drives with Mount Points, Windows JBOD, Windows RAID-5 SW hack, none of which were a great soln.

For me, all I wanted for my Media was a Pool that had the:
- Ability to have multiple HDD of any type size and from any mix of controllers
- Ability to add a new drive, move stuff off old drives etc
- Present as 1 logical Drive in Windows but have the ability to look at the files on a per drive basis as needed
- Preference for a Physically Separate 2nd Pool to periodically sync to (protection from fire, theft, HDD / Pool failure, unintended deletions, additions, modifications etc)
- No SW or HW Raid, as I had 100% drive overhead already for the Backup Pool so no desire for the extra cost in $ or performance of Parity Drives as well (I can live with the time it takes to copy back a drive failure)

So I have two pools, one on my "Main" PC that servers the media to the house.  The other is on a Win Server 2012 R2 Essentials Box that after I've made any significant changes on my "Main" pool, I'll run FreeFileSync to see what has changed and if happy make the commit.  

This is just the setup for my Media Holdings (40TB).  Nice, simple and it works without any complication.  

BUT:  The Win Server ALSO does traditional historical backups of all my PC's in the house for both file and bare metal restores (and I've had to use on several occasions to recover "good versions" of both whole HDD as well as individual files).  The HDD this is stored is also backed up by the OS.  I deliberately decided to exclude my Media Pool from the Win Server backup as I don't want to keep historic versions of changes I make to these files (eg if I update the Tags on 1,000 files I'm comfortable I don't need this history on this) and I want to see that only what I expect has changed.

I also see that others have combined the use the Drive Pool and FlexRaid etc to get Parity which is interesting.

Each to their own I guess, but this combination has worked really well for me to date.

Thanks
Nathan

[RoderickGI]

I like your setup jmone. Nice and simple. I may need to look at Drivepool at some stage.

As per my signature, I wasn't convinced that Flexraid would recover all my data based on a parity drive, back when I looked at it, and I didn't want to use the snapshot functionality as it had too many compromises in the assurance it provided.

[jmone]

Do you know if DrivePool supports VSS?

Nope neither DrivePool or Driver bender supports VSS: more from DrivePool after I asked the Q:

And no, StableBit DrivePool doesn't support VSS.  And for the same reason Drive Bender doesn't. There is absolutely no information on how to implement it, so it requires reverse engineering the feature, and extensive testing.
 
I know Anthony (drive bender dev) was working on support for it, but even so.... it was very unstable, at best.

[Hendrik]

PS:- is this a fair comparison of the available software based parity solutions?

http://www.snapraid.it/compare

I haven't found an actual fair comparison.
I use FlexRAID RAID-F myself, mostly because its a parity and pooling solution in one, and I have used it for years now, but you could probably do the same with snapraid and StableBit DrivePool without any differences in functionality.
I've had a single drive failure with FlexRAID before and restoration was flawless for me.

If you look at that comparison on the snapraid page there, FlexRAID and SnapRaid fair quite similarly, so...

[Hilton]

Just wanted to say thanks for your input guys.  Hendrik I've decided to take your lead and go with FlexRAID RAID-F.

I've got a bit of drive shuffling to do with my new WD Red 8TB drives which will hopefully be a one off shuffle.
Need to copy everything over to the 8TB drives (using Richcopy 4) so I can disassemble the RAID arrays and turn them into single disks for drive pooling.

I'm going to go with a solution which is a combination of both Hendrik's and Jmone's but using RAID-F snapshot parity on pooled disks instead of stablebit drivepool and then do scheduled periodic back up to another pool of disk.


Hendrik are you using realtime RAID-F yet or still using snapshots?

[Hilton]

Lucky I'm doing a disk shuffle...

One of my WD green 3TB drives is on the verge of failing after 3 and a bit years and the Samsung EVO 840 is down to 25% wear levelling after 3 years.  The SSD will hang in there for a while longer yet, but that wear levelling is a result of me having my Backup dedupe database and logs on the SSD to speed up the backups.  Not too worried about the WD green as it's part of a mirrored pair that I was going to break as part of moving to FlexRaid anyway. Good timing though!

3TB WD Green
HD-Sentinel-1 by Hilton, on Flickr

120GB EVO 840
HD-Sentinel-SSD by Hilton, on Flickr

[Hilton]

Update.

First RAID volume of 6TB copied no problems. Though I've had another WD green 3TB drive drop to 70% health during the copy, 300 sectors marked bad and 200 weak sectors.

Think I'll go for double parity for the 3TB drives when I convert them to another FlexRaid pool.

A bit more setup and configuration for FlexRaid RAID-F than I was expecting but it's not too bad.
ie setting up maintenance tasks and email alerting.

You would think that configuring a Cruise Control FlexRaid volume would automatically generate the best practice maintenance tasks to do the parity sync and verify etc, but you have to manually configure them.

So far FlexRaid RAID-F looks ok to me.  I did a couple small test rebuilds with a sample set of data on there so I understand how the software works incase I actually need to replace failing drives. It's pretty easy but I can see why some people would prefer hardware RAID to take care of it - Look for flashing alert on the failed drive and replace - Array automatically rebuilds.  

A bit more human intervention required with software RAID but I still think there's a much lower chance of loosing all data if you get a failed software parity rebuild because the parity and data isn't striped across multiple disks.  Worst case, I think it would be pretty hard to loose even 1 disk worth of data which is manageable for a media collection.

I contemplated T-RAID but in the end I figured if I wanted real-time redundancy I'll stick to hardware RAID-1 for data that's valuable enough to warrant it.

Pretty happy so far with the combo of FlexRaid with Hard disk Sentinel - The FlexRaid built in SMART monitoring doesn't pickup my RAID attached drives but the Hard Disk Sentinel does.

This is my new storage/backup regime - Media backups I'm still undecided as it's pretty expensive to backup 20TB

OS drives/documents/photos etc - Fast single SSD - GFS Backup with Acronis 11.5 w/ daily incrementals / weekly full retained for 12 months
Documents/photos/other important files - synced to onedrive (in addition to backup)

General Data/Gaming Drives - 4 x WD Black 1TB RAID 0 hardware RAID - Weekly full retained 3 months

Media (movies/music + less important archive data/games) - for now FlexRAID RAID-F pools with 1 parity and 1 URU (parity recovery drive incase of parity failure during rebuild) - Backup?? hmm to TBD

To be honest - Pooling is almost not necessary with 8TB drives - it's more the parity protection that's important.

[blgentry]

Media (movies/music + less important archive data/games) - for now FlexRAID RAID-F pools with 1 parity and 1 URU (parity recovery drive incase of parity failure during rebuild) - Backup?? hmm to TBD

You're using software RAID, which historically is NOTORIOUS for failures that are unrecoverable.  ...and you're wondering *if* you should do backups?  You absolutely, positively need backups.   Unless you don't care about losing the data on those drives.

Parity is not a backup.  Software RAID is worse than hardware RAID.  I think you'll probably be fine for a while.  But who knows what might happen if your machine does a hard shutdown the wrong way (pulled power cord, failing power supply, etc).

20 TB is "too much" to back up?  "With great power comes great responsibility." 

Brian.

[jmone]

I'm with Brain Brian!

[mwillems]

I've never really understood the case for RAID in a home NAS context.  It introduces a lot of failure modes, isn't a backup, and unless you happen to have multiple gigabit ports teamed on the NAS (or very slow drives) it doesn't even present a performance advantage (a modern hdd will more or less saturate a gigabit link just fine).  There are lots of drive pooling solutions that don't require RAID.  I guess it helps uptime?  If you have a next-gen filesystem it notionally allows for self-healing arrays, but even without RAID the next gen filesystems will alert you to bitrot and you can just restore from your actual backup. 

I think most people would be better off from a data integrity standpoint (if you can afford half again your total number of disks for parity) to just buy the extra disks you'd buy for RAID, but use them as offline backups for the important parts of your data, or spend a little more and have a full offline backup. 

[Hilton]

You're all correct in your observations and comments and I've struggled with this myself for quite some time debating the merits of hardware vs software RAID vs Pooling vs backups.

But I've decided that this form of software RAID is the safest with large volumes of data. FlexRAID doesn't actually change the data, or stripe data, all it does is calculate parity and put it on another disk. You can even have multiple parity disks if you want extra protection and an "undo" parity drive option as well (URU)

The source data is unchanged. Think of it as an additional layer or redundancy, it's certainly not backup.

As far as pooling goes, it also doesn't change the data it hides the volumes and uses a similar technology to junction links to just point at the file on a virtual disk.  The underlying files and individual disk can all be accessed by just plugging them individually on another computer.  So in the event of worst case you could loose a couple of disks worth of data as opposed to a hardware raid corruption (which I have experienced) where all data becomes inaccessible.  

Performance is more than adequate with native read and write speeds between 100MB/s to 160MB/s to a single disk.

I also tried to break it in several ways and couldn't in the couple hours I spent on it.
Unplugging power while copying data
Forcing a blue screen by overclocking the CPU while copying data
Stopping the FlexRAID service while copying data
Pulling a drive while copying data
Pulling a drive and inserting again while copying data
Moving drives
Relabelling drives
Changing drive letters
Putting disk in another machine and added and deleted information and putting it back in the pool

In all cases only the current file being copied/deleted was lost - all other data was intact or able to be rebuilt from parity.
Chkdsk reported no issues in all cases.

It also has built in bit-rot detection and a powerful scripting engine that you could do some interesting things with.
Technically it works with reFS but the developers recommend sticking with NTFS.

I can't see adding a software parity disk as a risk under these circumstances as Windows has as much chance of corrupting data at the filesystem level and I've lost a complete Hardware RAID array before because of disk signature corruption TWICE from failing disks and windows destroying the disk RAID signatures! (lucky for backups!)

As you can see my storage regime has my most critical data backed up twice with multiple restore points and I can rerip worst-case.  FlexRAID isn't the rolls Royce solution but it's very cost effective protection against drive failure with great flexibility to allow you to just add single additional disks instead of having to buy sets of disks and re-laying your data all the time.

I'll certainly be looking at what I can do to create some sort of backup solution though. It maybe that I have enough disk capacity when I break up my RAID 10 disks to create a backup pool with RAID-F so I can just buy disks in pairs going forward.

[Hilton]

It took 33 hours to generate parity for 12TB of data, about what you'd expect even from hardware parity rebuild.  Lets see how it goes from here.

I have enough disks left over to create another backup pool of RAID-F disks and still have a few TB free space so the next job is to create a backup.
The current RAID box doesn't do JBOD so I'll be getting something else to allow pooling of individual disk.

Co-incidentally one of our HP ESX Servers in our UK office destroyed the office data with RAID controller corruption of the volumes.  HP spent 16hours trying to recover the data and had to give up. We're now moving that environment to Azure rather than rebuilding and restoring the data.

Makes me happy I've made the choice to move off hardware RAID for my media collection!

Snapshot by Hilton, on Flickr

[Hilton]

Just an update on my experience with FlexRAID.... 

FlexRAID failed after a couple days to do a parity update and didn't notify me even though I had it setup to send emails.  It's not reliable enough for me to warrant the change from hardware RAID to JBOD pooling with software parity.
None of the data was lost but I just don't trust it.

I've gone back to 4 x 8TB in Hardware RAID 5 Direct Attach external esata/USB3 boxes and external backup.  I'm happy enough to keep buying 4 and 8 bay Hardware RAID boxes and just backup often.
I looked into all sorts of rack mount drive enclosures and I can buy 3 x 8 Bay Hardware RAID boxes for the same price as a rackmount system.  22TB usable with 4 drives is plenty and the cost per set isn't too bad compared to other options. (apart from pooling)

My RAID boxes also do JBOD spanning so I'm going to give that a try for backups. At least then the data from individual drives is still accessible even if a drive fails.

[RoderickGI]

Thanks for the update Hilton. You have validated my decision on FlexRAID again.

[jmone]

Both my DrivePools still going strong here without issue.

[marko]

I see Drive Bender is back in development again... no more donation-ware...

Since August last year, apparently...
https://blog.division-m.com/2016/08/22/its-hard-to-keep-a-good-app-down/

Finally, we are planning to up the price on both Cloud Xtender and Drive Bender.

Guess who's been clearing out some bookmarks this morning

[jmone]

Good to see DriveBender is still alive.  My DrivePools are also working well.

Of interest is that ReFS is now available as a format option in the latest version of Win10.  It appears to be a different version however to that in WS2012R2 as drives formated as ReFS in Win10 are not recognised in WS2012R2 (while those formated in WS2012R2 are recognised in Win10).  Still loving the combo of Drive Pooling and ReFS = 0 probs.