Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[6233638]

I feel like I have been pushing my luck for too long now. What started as a gaming PC is now mainly being used for media playback.
Because I wasn't running any real Media Center software before (just using Windows Libraries and MPC-HC) I did not have a very robust system set up, and just kept adding drives to increase the storage as required.

But since moving to Media Center I've now taken it upon myself to actually rip all media that I own and put it on this machine, now that I am using software that can manage it properly.
So in the couple of months since using MC, I've managed to just about fill my drives, and am having to move from ripping my DVD/BDs as ISO (which is easier) to converting them to MKV.
It has freed up a surprising amount of space, but it's only a matter of time until I run out again.

And as I said - I have been pushing my luck. I have an external 1TB disk that I copy important documents and all of my music to once every couple of weeks, but that's the extent of my backup system. (at a minimum, I'm going to pick up another one and rotate them on a weekly basis now)
What started out as a single 1.5TB drive has grown to about 10TB in total over the last few years - Blu-rays just eat up disk space.

So I really need to get a proper storage & backup solution in place - I don't even want to think about how many hundreds - if not thousands - of CDs, DVDs, and Blu-rays that I would have to rip again.
While there are clear benefits to running a HTPC, I would probably just end up looking into buying disc changers and video distribution units, leaving the HTPC for music only.
It's fine when you are just ripping a couple of discs as you buy them, but having to rip them all at once would be a daunting task.


But while I know a bit about PC hardware, I really know nothing about storage.
All I know is that I hate the Marvell SATA controller that my motherboard uses - it's caused me a number of problems. (can't do a UEFI boot off it, stalls during large transfers if you don't have the right drivers etc.)


I don't really know where to start though. I want something that is as "future proof" as possible - right now I have about 10TB but I want to have it backed up, and be able to expand well beyond that.
I tend to buy drives in "2 for ___" deals when they show up, so I suppose it should be able to survive at least two disk failures at the same time.

Beyond that, I don't know what I need - hardware RAID, software RAID, ZFS, ReFS, SAS drives/controllers etc?
I don't know whether I just want to build a machine that has a lot of storage in it, or a networked storage server.
Things like RAID always seemed like a scary proposition, because you could lose the data on all drives if you have a couple of failures, rather than just losing the data on that disk if you are using them as individual drives.
I don't know how things like RAID handle expansion and upgrades either. (e.g. your case holds 10 disks, it's full, and you want to upgrade some 1.5TB drives to 4TB ones)

[glynor]

We actually have quite a few well-versed storage experts here, as you might guess (we've all hit the same problems before).

I've, personally, done quite a few posts on this board covering many of the issues you raised.  There are, essentially, three choices:

1. Build a real, locally attached RAID volume.  Go with RAID-5 or RAID-6.  These RAID levels spread parity data across the disks in the array to ensure that if one disk fails you don't lose all data on the volume.  RAID-6 has more redundancy than RAID-5, but "costs" additional space for more parity data (and is typically only available on higher end RAID controllers).  RAID is FAST, and is done at the hardware level, so it doesn't require any filesystem fanciness.  It is NOT a backup though, as it will happily let you delete files or corrupt them like you could on any disk (and delete or corrupt the associated parity data).  And, the main downside is that it requires you to use matching disks.  Any disks are "treated as though" they are the "lowest common denominator".  So if you have a RAID of 3 1TB 7200RPM disks and 1 2TB 5400RMP drive, the extra TB on the bigger drive will be wasted/unusable and the entire array will perform as though it is an array of 5400RPM drives.  Most RAID controllers also don't allow you to increase storage space without reformatting the volume.  Some RAID controllers support online volume expansion or RAID level migration, but Windows won't recognize the additional area and you often need to reformat to get access to it via the same drive letter (or use the built-in or a third-party volume "expansion" tool which is always dangerous).   It does provide, however, a good, fast, and well-supported mechanism for unifying multiple physical disks into one volume, while ensuring that a single drive failure doesn't wreck you.

If you are buying a RAID controller, and you can swing it, I'd look at Areca or LSI controllers as opposed to the cheap HighPoint cards.  Not because of performance or reliability (the differences here are generally pretty small with modern cards and computers), but because the HighPoint drivers are laughably bad.  There's a reason they're so much cheaper.  That said, I still use a middle-level HighPoint card and it has been great for a long time.  It just makes my eyes bleed when I need to tweak something on it, and it was a bit fiddly to get set up and working well at first (involving setting esoteric settings on the card via a BIOS flash procedure to get it to work with my EFI-based motherboard, for example).

2. Use a "smart RAID" filesystem "overlay" application in Windows like FlexRAID or DriveBender.  These generally trade performance (and, often, reliability) for the ability to overcome those two big issues with "real" RAID:  using dissimilar disks and adding capacity "on the fly".  However, I'd read through some of the posts on this board about DriveBender in particular before deciding to go with this option.  These options can work quite well, but it isn't just a little performance that it trades, and they can be... fiddly.

3. Build (or buy) an external "smart RAID" box (optionally a NAS as well).  Some options would be Drobo, UnRAID (hah, their website seems to be hacked right now), or FreeNAS.  These generally do the same type of thing as Option 2, but without messing around with the way Windows works, and putting everything in an external box.  Most of these systems are custom Linux builds with fancy UIs on top.  They can perform pretty well (not anywhere near as well as a local fast hardware RAID, but plenty for media playback and whatnot), and are generally pretty stable and featureful.  Be careful with some of the NAS features, as they can be missing features in Windows that allow things like the filesystem modification events to work (which means Auto-Import stops working).

For the record, I'm in the #1 camp, but I also edit video professionally, and performance matters when you're trying to play 6 simultaneous streams of ProRes 422 HQ 1080p video on top of one another (or a stream or two of RED Scarlet 4k footage).  It isn't just that, though... I like the reliability and simplicity of having the local disk, really formatted to NTFS, directly attached.

Other options, including better filesystem support ala ZFS would be awesome, but Windows isn't there yet, and all third-party drivers I've seen are alpha-quality and flaky.  Unfortunately, we don't have anything as cool as CoreStorage on OSX.  ReFS looks promising, but is quite limited right now.

[glynor]

Search for posts with my name and "RAID" or "storage" and you should come up with all kinds of stuff and options.  Nathan (jmone) also has a lot to say on the topic, and many others here too.

[DarkPenguin]

If you want to push off buying that Bently for another decade we (Dell) make some nice storage products.

[glynor]

If you want to push off buying that Bently for another decade we (Dell) make some nice storage products.

Hah.  Agreed, there are tons of highly priced enterprise focused storage systems out there.  Some of them are even worth the money.

If you want something a little more affordable, but still quite powerful, I've had good luck myself (and have heard from tons of others) with Small Tree.

[rjm]

Don't rule out a simple low tech solution.

I have 7 media drives: Media1... Media7. For each media drive I have 3 copies. One in my main system. One in my old system that I converted to a headless server. And one in an external drive enclosure. The 7 external drives fit nicely in a toolbox which makes them portable.

Once a week I back everything up using Syncovery.

Simple. Cheap. Portable. Bulletproof.

If you search this forum and others you will find that people who use complex RAID solutions more often complain about RAID bugs and failures than they do about RAID saving them from failed drives.

[glynor]

If you search this forum and others you will find that people who use complex RAID solutions more often complain about RAID bugs and failures than they do about RAID saving them from failed drives.

With the software solutions (particularly Drive Bender style on top of Windows), this can be true.

For a large media collection where you have lots of drives spinning?  RAID really can't be beat if you can't afford to have redundant copies everywhere.  It has certainly "saved me" from drive failures a number of times in the past 10 years or so that I've been running a RAID-5 of some variety.

My setup has changed a bit since this, but I wrote up my home storage and backup strategy a while back, and most of this is still true (the drives have long-since been swapped out and the total volume is much bigger, which requires multiple backup drives, but mostly it is the same).

For backup, I use a combination of:  SyncBack SE (local backup to separate disks on a schedule), GoodSync (for "cloud syncing" to other machines and online storage), and Macrium Reflect (for imaging).

Though, I agree with rjm conceptually.  Don't worry so much about needing to get everything on the same logical volume (drive letter).  MC doesn't care, and aggregates the content anyway.  It does make developing a robust backup strategy more challenging when you have stuff spread everywhere, though.  And once you have a "single set" (say, your TV Shows) that is more than around 3.5-4TB, you can't keep things on single drives anymore, without arbitrarily splitting them up by series or date or whatever.  Balance is key.

Another thing I do is use a set of Archive Drives.  So, for example, when we finish watching the current season of Game of Thrones, I'm not going to delete them, but I will archive them off to one of my TV Show Archive Disks.  These are just big 3.5 inch internal disks that I use with my various SATA docks to store and access stuff that doesn't need to be "online" all the time.  MC still has the files in the Library (though they're hidden from most of the "main views") so if I want to go back and re-watch something again, I just have to plug in the drive on whatever computer I happen to be using, and the files are still there in MC.

This strategy allows me to keep my overall always-online storage requirements in check, while allowing me to keep "old stuff" that I won't access that often, but don't want to delete either.  Right now, I have five Archive Drives (two for movies, and three for TV Show episodes) on the shelf in my basement.  These, of course, aren't as well backed up (though I do have additional copies of some of it on "old disks" at the office).  But, they're also not spinning very much, so... Barring flood, fire, or cosmic rays, they should be fine for a long while.

[6233638]

It is NOT a backup though, as it will happily let you delete files or corrupt them like you could on any disk (and delete or corrupt the associated parity data).

I am aware of that, but when you start looking at redundant storage and buying more than one or two disks at a time, the costs go up a lot quicker than you expect - I don't know that I can afford to have a "true" backup of all my data.
I'm fine when it comes to music, because I can just buy a single portable drive, and that has enough capacity to store it all.
If I currently have about 10TB storage, and I'm looking to increase that by about 12TB or so (for now) I can't afford to buy an additional 22TB to have a backup of that.

And, the main downside is that it requires you to use matching disks.  Any disks are "treated as though" they are the "lowest common denominator".  So if you have a RAID of 3 1TB 7200RPM disks and 1 2TB 5400RMP drive, the extra TB on the bigger drive will be wasted/unusable and the entire array will perform as though it is an array of 5400RPM drives.  Most RAID controllers also don't allow you to increase storage space without reformatting the volume.  Some RAID controllers support online volume expansion or RAID level migration, but Windows won't recognize the additional area and you often need to reformat to get access to it via the same drive letter (or use the built-in or a third-party volume "expansion" tool which is always dangerous).   It does provide, however, a good, fast, and well-supported mechanism for unifying multiple physical disks into one volume, while ensuring that a single drive failure doesn't wreck you.

RAID sounds like it's focused on performance - which is good - but it also seems very inflexible.
My biggest issue is that if I were to buy a case with a lot of hot-swap bays - say 24 - I would only fill some of them now. (I couldn't afford to fill them all!)

Even if I'm buying all new drives for it, rather than re-purposing the drives I already have (which range from 500GB to 4TB) lets say I buy 6x3TB disks. (3TB are the cheapest per GB here) With RAID6, that gives me 12TB of storage (18TB-6TB parity) which seems like the minimum amount of drives to make RAID6 worthwhile.
There are reports which suggest that 6TB disks might be available next year, so when it comes time to add more storage, the best price/GB will probably be 4TB drives.
And capacity is going to keep increasing, so by the time I get to filling those 24 bays, who knows what will be available - you might have disks with 10TB or more.
But with RAID, it's only ever going to use 3TB of whatever I buy, because that's what I started with, so the maximum you would ever get in that system is 66TB with RAID6.
That sounds like a lot today, but in a few years everything should hopefully be 4K, which probably means that films will take up 2-4x the space they do now, and eventually we will be moving to 8K, so storage requirements are going to increase considerably.

So RAID sounds like a good solution if you need high performance and moderate storage requirements for video editing, but not as a long-term storage solution.
Unfortunately the software solutions sound just as problematic as RAID - but for different reasons. Is ReFS no good then? Being a part of the OS seemed like it might be easier to deal with. (unlike ZFS and the other tools you're suggesting)

Search for posts with my name and "RAID" or "storage" and you should come up with all kinds of stuff and options.  Nathan (jmone) also has a lot to say on the topic, and many others here too.

I'll have a look for that.

If you want to push off buying that Bently for another decade we (Dell) make some nice storage products.

See, this is my problem - there doesn't seem to be many options between simply buying a small number drives and using them as individual disks, and enterprise-class storage.

I just want my media to be safe from a hard drive failure (or two) but I don't want to suffer degraded performance either. The fastest HDDs in my system are above 150MB/s on their own, which exceeds what gigabit ethernet is capable of, and seems to rule out networked storage.
It sounds like software-based solutions are slow, and I don't really want anything less than I would get from simply plugging the drive into my PC and using it as normal.
While I'm looking for a high capacity solution - it's not just going to be for media storage/streaming, so the performance is a consideration.

Don't rule out a simple low tech solution.
I have 7 media drives: Media1... Media7. For each media drive I have 3 copies. One in my main system. One in my old system that I converted to a headless server. And one in an external drive enclosure. The 7 external drives fit nicely in a toolbox which makes them portable.
Once a week I back everything up using Syncovery.
Simple. Cheap. Portable. Bulletproof.

This works up to a point. The problem is that I'm now out of SATA ports, and space to add any more drives. (funny that the largest case I've bought, has the least amount of space for HDDs)
I don't really want to have external drives kicking around all over the place - part of the reason to move to an HTPC is so that I could get rid of all those discs and store them away, rather than keeping them within reach of the TV.
I don't want to replace shelves of discs with external HDDs and end up like this.

If you search this forum and others you will find that people who use complex RAID solutions more often complain about RAID bugs and failures than they do about RAID saving them from failed drives.

This is why I had not done anything about RAID now. At least if something goes horribly wrong when they are treated as individual drives, you are only ever going to lose what is on one of them - there's (almost) no possibility of losing everything.


All of that said, I'm going to do some searching and have a good read through the forums to see what has been suggested before.
I have a better idea of what my options are, but I still don't know that I actually like any of them.

However, converting a large number of my Blu-rays to MKV seems to have been all I needed to free up enough space that I don't have to make a decision right away.
I only actually care about the main feature anyway (I can always get the disc if I really want to see the extras) and it seems to be saving me a minimum of 5GB per film, and as much as 20GB in some cases. (more than that if it's 3D)

[InflatableMouse]

I know this is nitpicking 

But ...

Drivebender is not RAID, mainly because its not redundant. It does striping over disks on file level (keeping files complete and intact on a disk) and uses regular NTFS formatted drives that can still be accessed as single, independent disks. But it does not do any redundancy, does not calculate parity blocks or any of that stuff. It can do file replication on configured folders and although this makes certain files redundant, that doesn't turn AID into RAID    .

The thing with Drive Bender is that a failed disk will never result in the loss of the entire striped volume, only what was on that single disk. At any time, you can pull a disk from it, hook it up to another pc and access the files on it. If you grab all the disks, shuffle them around and install them in another pc and install Drive Bender, it will detect each disk and recreate the volume. As it stands now, DB is stable, mature and good looking. It's fast too. With smaller files I've hit speeds of around 400MB/s and with a single large file its as fast as the disk its writing to/reading from - there is no noticable overhead. I've pushed the system where both reading and writing were hitting around 200MB/s simultaneously during which the system remained usable and responsive. For personal, consumer level stuff, I find that impressive.

But its not all cheers. I've been using DriveBender for a while now, first on 2 disks, then on 4 and now on 5 disks (2TB). I've come this close to deinstalling it on several occasions due to "issues" ... non-destructive issues I might add but still, time consuming and above all, very frustrating.

It's still running though and I've not had any issues in recent months. They've fixed many issues, from the patchnotes I've learned they fixed some of the issues I had as well and (knock on wood) the last few versions have been very good and very stable. The recent reinstall of my HTPC was completely painless and detected and auto-configured Drive Bender volume without any hassle at all - I was surprised it was that easy.

There's been talk to incorporate Parity into Drive Bender, you can read more about that on their forums. I've no idea how and even if its going to happen, but I think its most likely they will do something similar to SnapRaid. I don't think they will do real time parity calculations to make a full blown software raid solution, but who knows. I might even try it out if they do  .

Another thing they've done is included a light version of HD Sentinel for SMART monitoring & notifications. Via DriveBender you can get a discount on the full version of HD Sentinel as well.

Lastly, as Glynor rightfully points out, RAID (and Drive Bender even less so) is not a substitute for backup. I can't stress how important making backups is. Typically people don't seem to get that until its too late and many people unfortunately have to learn that the hard way, myself included in the past. If you are going to invest in better storage, think about how you would like to make backups as well.

I don't backup my videos (movies, series) ... It's a calculated and accepted risk ... Music files, pictures and MC library backups are copied (using Allway Sync) to a dedicated volume and then again to another PC in the network. All the PC's in the house are storing their images on that dedicated backup volume as well (with Macrium Reflect v5 Pro, 3 full once a week and incrementals on the remaining days, after the 3rd succesful full it deletes the oldest full + linked incrementals).

Lastly, I've had a NAS. A Thecus N5200XXX Pro or something. I did not like it at all. It was noisy (fan noise), and only fast for single user purpose, on large filecopies. Smaller files slowed it down to a crawl (more so than usual). Also, accessing the NAS from several clients could result in lockups that could take up to minutes to release. I've been told Synology is miles better and typically doesn't suffer from these issues. I've been thinking about getting a DS1812+ and moving the disks to that, and migrate it to a RAID volume. I don't know yet ...

[6233638]

I think you must have edited your post as I was replying, glynor (or I just managed to skip over it)

Though, I agree with rjm conceptually.  Don't worry so much about needing to get everything on the same logical volume (drive letter).  MC doesn't care, and aggregates the content anyway.

Well that's what I have now, and I was using Windows Libraries before MC to keep track of content, so I'm not concerned about that.
It was more about having the redundancy/parity drives, and running out of SATA ports/space for drives that got me thinking about moving to something a bit more advanced.

Another thing I do is use a set of Archive Drives.  So, for example, when we finish watching the current season of Game of Thrones, I'm not going to delete them, but I will archive them off to one of my TV Show Archive Disks.  These are just big 3.5 inch internal disks that I use with my various SATA docks to store and access stuff that doesn't need to be "online" all the time.  MC still has the files in the Library (though they're hidden from most of the "main views") so if I want to go back and re-watch something again, I just have to plug in the drive on whatever computer I happen to be using, and the files are still there in MC.

I like the sound of this, and it's not something I had considered until now. I suppose it's not too far separated from using external drives, but without all those external power bricks etc. to deal with, and it will take up a lot less space.

The thing with Drive Bender is that a failed disk will never result in the loss of the entire striped volume, only what was on that single disk. At any time, you can pull a disk from it, hook it up to another pc and access the files on it. If you grab all the disks, shuffle them around and install them in another pc and install Drive Bender, it will detect each disk and recreate the volume. As it stands now, DB is stable, mature and good looking. It's fast too.

That sounds considerably more flexible than RAID and a lot of the other software-based solutions. Might be worth giving it a try.

Lastly, as Glynor rightfully points out, RAID (and Drive Bender even less so) is not a substitute for backup. I can't stress how important making backups is. Typically people don't seem to get that until its too late and many people unfortunately have to learn that the hard way, myself included in the past. If you are going to invest in better storage, think about how you would like to make backups as well.

I don't backup my videos (movies, series) ... It's a calculated and accepted risk ... Music files, pictures and MC library backups are copied (using Allway Sync) to a dedicated volume and then again to another PC in the network. All the PC's in the house are storing their images on that dedicated backup volume as well (with Macrium Reflect v5 Pro, 3 full once a week and incrementals on the remaining days, after the 3rd succesful full it deletes the oldest full + linked incrementals).

I currently have things set up so that important files automatically get backed up to other drives in the same system via Acronis, and every couple of weeks I back them up to a portable 1TB USB3 drive.
I'm now planning on picking up one of the 2TB WD drives (same cost/GB as 1TB) and alternating them on a weekly basis.

I was wanting to move to RAID - or something like it - to at least give me some protection against drive failure for my videos.
I know that it is not a backup system in that if I delete a file by mistake or it is somehow corrupted, it's gone.

Lastly, I've had a NAS. A Thecus N5200XXX Pro or something. I did not like it at all. It was noisy (fan noise), and only fast for single user purpose, on large filecopies. Smaller files slowed it down to a crawl (more so than usual). Also, accessing the NAS from several clients could result in lockups that could take up to minutes to release. I've been told Synology is miles better and typically doesn't suffer from these issues. I've been thinking about getting a DS1812+ and moving the disks to that, and migrate it to a RAID volume. I don't know yet ...

I should have said - I've had a couple of NAS devices in the past, and that's what has put me off the idea of networked storage altogether - that and that fact that 10gig ethernet is still extremely expensive.

[glynor]

As it stands now, DB is stable, mature and good looking. It's fast too. With smaller files I've hit speeds of around 400MB/s and with a single large file its as fast as the disk its writing to/reading from - there is no noticable overhead. I've pushed the system where both reading and writing were hitting around 200MB/s simultaneously during which the system remained usable and responsive. For personal, consumer level stuff, I find that impressive.

I haven't fully dug through the new stuff in this thread yet, but I did want to mention two things about this...

1. I haven't personally used DriveBender, so I can't comment on it directly.  I've just seen lots of "weirdo issues" posted here and other places that traced back to DriveBender getting borked up over the years.  From what I've gleaned, it can work extremely well, but it takes some care and feeding, planning, and keeping an eye on it.  Since that adds overhead (work I have to do) it negates somewhat the increase in flexibility.  Of course, my information might just be old.

2. Speed is in the eye of the beholder.  My RAID here at home can easily do 450MB/s sustained across even very large files (15GB+), and for small burst-mode stuff, it can hit 800-900MB/s, though this doesn't matter to me much.  More important than sustained throughput for me, though is IOps performance, and this is where many of the "overlay" systems fall down and wet their pants (just like early SSDs).  Because my RAID is used as the main media storage AND recording location, for multiple client machines, it needs to be able to service multiple, simultaneous requests without freaking out.

Essentially, no video file I ever play at home will need more performance than the ~120MB/s I can get over the network.  But IOps can hurt, if you're ripping some discs, recording three separate shows, and a client or two is playing content, all simultaneously.  Mine handles this without even thinking about it, and it is a 4 year old, $100 Highpoint card, and a bunch of cheap HGST drives (though they are 7200rpm models).

IOps matter.  For this kind of use, sustained throughput?  Meh.  That helps when you're doing backups is about it.

Lastly, if you're buying drives new, take a nice, hard look at the new WD Red series drives.

[6233638]

Lastly, if you're buying drives new, take a nice, hard look at the new WD Red series drives.

Aren't they limited to five drives in a system? I seem to recall people reporting problems with lots of them connected.

[glynor]

Aren't they limited to five drives in a system? I seem to recall people reporting problems with lots of them connected.

Not that I was aware of, though I could be wrong.  AnandTech didn't mention anything about this when they reviewed them.

My completely uneducated guess from similar kinds of things I've seen reported elsewhere?
People don't understand how bad vibration issues can be with a bunch of drives in a stack in a small box.

But, perhaps WD did implement some kind of sneaky limitation on those to protect their higher-end RE drives.  In any case, I'm using 3TB HGST drives, and they're great.  I have an array of them at the office that has 12 disks and it has been reasonably trouble-free (I've had a couple of initial bathtub curve failures early on, but nothing out of the ordinary).  My home array has 6 of them.  WD gobbled them up though, and they're hard to find now, unfortunately.

[glynor]

My Archive Drive system works pretty well.  It is not completely without issue, though you could design yours differently to mitigate this, I suspect.

Here's how I do it:

1. Custom fields.  I have two custom fields for this system.  The first is called [Archive Drive] and is a simple string field with a pre-defined Acceptable Values list:
arch01-tv;arch02-movies;arch03-tv;arch04-tv;arch05-movies

My drives are labeled thusly, using Masking Tape and Sharpie.

2. The other is a Calculated Field called [Offline] that uses this expression:
ifelse(IsEmpty([Archive Drive], 0),0,IsEqual([Filename (path)],X:\,8),1,1,0)

That outputs a 1 if the file is on an Archive Drive, and a 0 if it isn't.  Handily, it does that "double-search" in order to result in 0 for any file that has a valid [Archive Drive] field entry, but which isn't on drive X.  That allows the files to remain in my lists until I actually move them over to the archive drives physically (I'll get to that workflow in a bit).  But, re-using Drive X creates the issue with MC, which I'll get to in a minute.

3. Mounting Drives, Naming, and Lettering.  I use USBDLM on all of my systems to recognize these specific drives when I plug them in and always assign them to Drive Letter X.  This works for me for a number of reasons, but mostly because the letter needs to be fixed to be imported into MC and kept working (without breaking links), and it doesn't use up all of the drive letters.  I have lots of drives (my media array, a work array, a system drive, a users data array, a whole swath of backup drives, thumb drives, camera card readers, two BluRay drives, and a few network drives).  With my "archives" each set up to use their own drive letter, I'd quickly run out.  USBDLM lets me set this up once, always use the same simple drive letter everywhere, and I can just copy the USBDLM ini file around if I need to add a new archive drive to the system (I actually use GoodSync to sync it between my systems automatically).

My USBDLM ini file looks like this (for the relevant section):

Code: [Select]

[DriveLetters2]
DeviceID1=*DISKWDC_WD20EARS-00S8B1_____________________80.00A80\8&B5A4A39*
DeviceID2=*DISKWDC_WD15EADS-00R6B0_____________________01.00A01\8&343B639B*
...ETC...
Letter=X
This is easy to set up.  You just copypasta the string out of the included scanner utility thingy.  If you use that style, it works on whatever system with the same string used to detect the drive.  There are other ways, though.  You can have it set the drive letter based on the presence/absence of a particular file, so you can just add an archive_drive.txt file to the drive root or something like that and use it as a flag.

This leads to two problems, though.  First, I can't use two of them simultaneously on the same system.  I should note, this isn't a USBDLM limitation, it'll automatically "fall back" to normal behavior for in-use letter, or check the next section, so you can still use two at the same time.  But, MC's database won't see them because they'll be on the wrong drive letter.  This isn't a huge deal in practice, though.  If I'm watching something from Archive, I'm... Watching it.  Not it and something else from the archive at the same time.  It just never happens.

The second problem is more serious.  Basically, re-using the drive letters like that breaks MC's Fix Broken Links feature.  You have to set it to No.  With it set to Yes (protect files on missing drives), MC will remove the files for all of the other "Drive Xs" as soon as you plug in one particular Drive X while MC is running.  As long as they're all unplugged, the protection works.  But if you plug one in, it can see that some Drive X actually exists, but those files aren't there, so it assumes they were deleted or moved (assuming, naturally, that Drive X is one specific volume), and removes them all.

An alternative mechanism that would work better with MC would be to mount them all with mount point folder paths.  This would take a bit more setup if you use multiple machines (and I use a bunch, so that's annoying), but if you have a manageable set of systems and use USBDLM to script it, it wouldn't be too bad.  You could mount them as both Drive X (for using in Windows and whatnot), and as a folder path, for use in MC.  That would, I think, work around the removal problem (especially since you could add them all to MC via a UNC path, which makes MC think it is a network drive even if it isn't).

I just keep Fix Broken Links disabled, and then I manually remove "real" broken links via a Smartlist and a script.  If you don't have automatically-created broken links (I have MC monitor and use some "transitory files" that get auto-deleted eventually), then this probably doesn't matter much to you.

4.  Views.  This is easy.  Essentially, my top-level Video view has a [Offline] is not 1 filter applied.  And then, each child view automatically inherits this.  But, my main Movies and TV Shows views, I just give them a child-view that is identical (drag drop the view onto itself, choose Copy), except that it ignores the parent view filters.  I do have to recreate a few other filters for these from the top-level parents, but this generally just means adding a [Media Type] filter in addition.  I call these "Child" views "All Movies (Including Archive)" or "All Series (Including Archive)".  I have similar views in Theater View.  Unfortunately, you can't make Theater Views ignore inheritance like this, so I have to filter [Offline]=1 directly on those views.  But this isn't a huge deal.

I also add [Archive Drive] as a category to my Standard View views, and Theater View info templates, so that I can see where the files live when they're on Archive (if I don't know which disk to grab off the shelf).  And, of course, all of the files also show up in my "Advanced" top level View where I keep my "maintenance and tagging" views.

5. Archiving Stuff.  When I want to archive something off to a drive, I simply plug in the physical Archive Drive I want to use.  Then I find the files I want to move wherever they live in MC, select them (you can do a bunch at once), and tag it with the Archive Drive tag for the drive I picked.  Then, I do Rename, Move, and Copy and use a preset like this:


My Archive (Movies) Preset.


My Archive (TV) Preset.

Then, when you hit OK, they get moved over to the Archive Drive, which automatically sets [Offline] to 1 (since [Archive Drive] is already filled), and they get immediately hidden from my "main" views, but I can still get to them from those "All Media" views.  MC takes a while to write the changes over to the disk, but I just walk away.  Lastly, before I eject the drive, I open a view showing all of the files on the inserted disk, select all, and do Update Tags (From Library) on them to make sure the current tag data is written out to Sidecar files or in-file tags.  Done.

[glynor]

Oh yeah.  And these:
http://www.bhphotovideo.com/c/product/882757-REG/CRU_DataPort_3851_0000_11_Anti_Static_Drive_Case.html
http://www.amazon.com/WiebeTech-DriveBox-Anti-Static-Hard-Disk/dp/B004UALLPE

[InflatableMouse]

I haven't fully dug through the new stuff in this thread yet, but I did want to mention two things about this...

1. I haven't personally used DriveBender, so I can't comment on it directly.  I've just seen lots of "weirdo issues" posted here and other places that traced back to DriveBender getting borked up over the years.  From what I've gleaned, it can work extremely well, but it takes some care and feeding, planning, and keeping an eye on it.  Since that adds overhead (work I have to do) it negates somewhat the increase in flexibility.  Of course, my information might just be old.

Drive Bender was supposed to be set and forget but it wasn't like that. I had lots of issues in the past. Security permissions not propagating over all drives for instance, I had this happen on several occasions. Or not mounting a drive, hanging services or not properly restoring a volume after a reinstall. Oh and of course the first issue I ran into, complete drive locks taking 60 seconds or more to release for no apparent reason and clients using a fileshare on that drive locking up as well. All of these issues were fixed though and I've not seen them anymore. It has never corrupted or lost data though.

Like I said, I've come this close to dumping it but for now, it seems to be stable. My tolerance for issues with DB is still close to zero so at the next sign of trouble its probably going out the door.

DB has some great advantages and its as flexible as can be. I think its a great product but like you said, it needs to work well or it negates having these advantages. It really looks like its coming around nicely though, so for now it can stay.

IOps matter.  For this kind of use, sustained throughput?  Meh.  That helps when you're doing backups is about it.

Yeh, you're absolutely right, you have mentioned this before. i've not measured IOPS but it could very well be a problem with Drive Bender. I've noticed an occasional hickup during playback, or some stutters and when I checked there was something doing some heavy reading/writing in the background. I think I don't care enough, it doesn't happen often enough and when it happens, I've already kicked back having a beer.

[jmone]

So my take on it:

Media Serving: You need a bunch of disks to hold all your stuff, enough bays to hold the HDD, and SATA ports to connect them.  You then need enough spare bays and ports to expand your storage for some reasonable time.  I'm sure you can work out how many of these bits you need but a couple of points to consider may be:
- HDD Choice :  I've slowly replaced all my smaller HDD with 4TB ones due to issue of never having enough bays or ports.  I just buy another 4TB hdd when they come on special and it spreads the cost of many months and keeps my available storage in advance of what I need.  I like the 4TB Hitachi 7200rps Deskstars but also have some of the 3 and 4TB Seagates
- HDD Bays : The "cheapest" is having a case with lots of bays, though a good case still costs plenty of $ (eg I use the Lian Li PC-Z70B and a Lian LI PC-Z60B and even have one of the 5 Bay in 3 5.25" Slot modules in one). Another option is an external HDD product like http://www.hotway.com.tw/products/h82-su3s2.htm but I've never used them so can not comments
- Sata Ports : I select Mobo's with lots of SATA ports and when you run out put in either cheap PCI Sata boards or a more expensive 8 port SAS/SATA (I went this route with the Highpoint but would struggle to recommend it).  

Presenting the Media:  So you now have a heap of drives and have three basic options in level of complexity
- Individual Drives : MC will abstract the drives as a part of the library but you still have to stuff around with drive mapping / UNC paths esp if you want to make it available over your network.  
- Pooling SW : I too use Drive Bender and while I've had some issues it has been good in allowing you to present one logical pool yet swap in and out physically HDD.  It's cheap and it has been working fine of late
- RAID : I've not used this for years but my experience was not good due to the relative poor level of flexibility in the consumer grade implementations.

...and on performance all are fine at pushing out multiple media streams.  I can saturate my Gigabyte NW and since when will I need to stream 20 BD's anyway.

Backup:  No pooling or RAID solution is a backup.  The level of backup you want will be reflected by your costs VS effort to restore.  The good thing is you already have a backup for all your MKV's in the original BD disks.  The bad news is it is a pain to restore if you lose a lot of data.  You would then "backup" the other stuff you can not on physical disk to an external HDD and store that somewhere else.  To balance the cost vs ease you could copy ALL to a second server/pool but how often will you lose all disks?  .... well turns out I did (I trashed the data on all the discs that I moved from the MB SATA ports to the Highpoint controller....and had to re-rip the lot)!  I now backup everything (including my BD) to a second Server/Pool but this cost me $$$

Monitoring HDD Heath:  Buy Hard Disk Sentinel and keep an eye on the HDD heath.  HDD will fail, esp when you have a lot of them.  You will also tend to have multiple failures at the same time.  Murphy's Law.

PS I now have two 30TB(ish) pools.

[6233638]

- HDD Choice :  I've slowly replaced all my smaller HDD with 4TB ones due to issue of never having enough bays or ports.  I just buy another 4TB hdd when they come on special and it spreads the cost of many months and keeps my available storage in advance of what I need.  I like the 4TB Hitachi 7200rps Deskstars but also have some of the 3 and 4TB Seagates

This is what I had been doing, but it's now at the point where the smallest drives in use are 1.5TB. (I have a few 500GB/1TB drives that aren't being used for anything right now - but I'll be buying a USB3/e-SATA dock)
Because HDD capacity hasn't increased for about 18 months now, and they're still only talking about 4TB drives with 4x1TB platters rather than 5x800GB to be released later this year, it's really not cost effective for me to do this any more, which is why I'm wanting to look at adding more drives now.

- HDD Bays : The "cheapest" is having a case with lots of bays, though a good case still costs plenty of $ (eg I use the Lian Li PC-Z70B and a Lian LI PC-Z60B and even have one of the 5 Bay in 3 5.25" Slot modules in one). Another option is an external HDD product like http://www.hotway.com.tw/products/h82-su3s2.htm but I've never used them so can not comments

Unfortunately the issue for me is that I bought a high end Silverstone case, which comes with five hotswap bays (and only a backplane for one of them ) Adding bay adapters only seems like a temporary fix, and throwing good money after bad.

- Sata Ports : I select Mobo's with lots of SATA ports and when you run out put in either cheap PCI Sata boards or a more expensive 8 port SAS/SATA (I went this route with the Highpoint but would struggle to recommend it).  

What are the differences between SAS and SATA cards? One person has recommended that I look into SAS cards with expanders, but it doesn't mean anything to me.

- Individual Drives : MC will abstract the drives as a part of the library but you still have to stuff around with drive mapping / UNC paths esp if you want to make it available over your network.  

I like the idea of having a storage machine in something like a Norco 4220 or equivalent, but the cost of moving to 10gig ethernet is far too high right now, and gigabit just seems really slow compared to local drives. I've also been considering just moving my PC into one of those cases, but I hear they're loud, and being a rackmount unit, they're very deep and I don't know that I have the space for that.

- Pooling SW : I too use Drive Bender and while I've had some issues it has been good in allowing you to present one logical pool yet swap in and out physically HDD.  It's cheap and it has been working fine of late

It seems like this is probably the solution I will go with rather than RAID.

The bad news is it is a pain to restore if you lose a lot of data.

This is why I at least have a backup of all my music - those are the worst because you're only looking at maybe 500MB/disc at most using FLAC/ALAC, so a terabyte of CDs is 2000-3000 discs, a terabyte of Blu-rays is 20-30.
Now that I've just thought about it in those terms, I think I'll be consolidating all my DVD rips on a single drive, and having an external backup of them now too. I can't have more than 4TB now that a large number have been replaced with Blu-rays. When you consider that most are 5GB or less, you're looking at 200-300 per TB.

So it's still a pain, but if you lose a single drive with Blu-rays, it's not the end of the world to re-rip those. I have multiple optical drives so I can get through about four an hour.
If that happened to me, I would probably just buy two or three extra optical drives to speed up the process - Blu-ray readers are relatively cheap these days, and even if a cheaper drive is only going to rip one disc an hour instead of two, it still speeds up the process considerably.

how often will you lose all disks?  .... well turns out I did (I trashed the data on all the discs that I moved from the MB SATA ports to the Highpoint controller....and had to re-rip the lot)!

This is my biggest fear with moving to external controllers/RAID etc.

Monitoring HDD Heath:  Buy Hard Disk Sentinel and keep an eye on the HDD heath.  HDD will fail, esp when you have a lot of them.  You will also tend to have multiple failures at the same time.  Murphy's Law.

Doesn't Windows keep track of SMART status these days?
And someone mentioned that Drive Bender includes HD Sentinel - should I just buy that, or is it a stripped-down version and I would want to buy the professional version of HD Sentinel anyway?

[glynor]

What are the differences between SAS and SATA cards? One person has recommended that I look into SAS cards with expanders, but it doesn't mean anything to me.
I like the idea of having a storage machine in something like a Norco 4220 or equivalent, but the cost of moving to 10gig ethernet is far too high right now, and gigabit just seems really slow compared to local drives. I've also been considering just moving my PC into one of those cases, but I hear they're loud, and being a rackmount unit, they're very deep and I don't know that I have the space for that.

SAS is a newer "version" of (though it is really quite different from) the older parallel SCSI standard.  It is backwards compatible with SATA drives.  Essentially, a SAS card can connect to a SATA drive, but you can't put a SAS drive on a SATA card.  The benefits of actual SAS drives are dubious, though, aside from better warranties and MTBF ratings (which, themselves, are dubious).

SAS cards often have more features for building arrays, and more importantly, they offer much more flexible connection systems.  Modern SAS cards generally use a variant of miniSAS (8087 for internal and 8088 for external) though the companies all do still build cards that use the older "big SAS" connectors as well.  Essentially, you can think of a single 8087/8088 cable as a "bundle" of 4 SATA cables.  There are breakout cables for them that go to 4 individual SATA connectors, in fact (I linked to a particular one before, but they have them on monoprice now).

So, if you buy a SAS card that has 2 8087 or 8088 connectors, you can connect 8 drives to it directly.

SAS cards also always support port expanders, and much more complex "networked" arrangements with actual SAS switches.  This is a simple concept.  Take one SAS port on the host adapter, and split it into more than 4 ports at the "hard drive end".  Most SAS host adapter cards support huge numbers of drives (I think my little cheap Highpoint card supports something like 128 drives).  Of course, the more drives you add to any particular port, the less total throughput that drive can use.  In practice, though, this is less limiting than you might think... It is incredibly rare that multiple drives on the same channel need to all do sustained reads/writes simultaneously, as long as you don't go too crazy with expanders.  That's because of the striping pattern from a RAID array, which ensures that the data block is spread across multiple disks, read/written in succession.  And, more drives means better IOps performance, so long as your controller isn't brain dead.  It also means more chances for failure, though, requiring additional redundancy.

miniSAS is length-limited, though, so it doesn't work well for putting your array of drives far from the host PC (in a closet).

For longer runs, the two best options right now are:

1. iSCSI
2. Fibre Channel

iSCSI uses regular Cat5/6 twisted pair and ports, and essentially turns your network port into a specialized, local storage connection system (using SCSI style signalling internally).  This allows you to eek out a bit more performance over twisted pair than you'd be able to through a network drive, as it takes the whole networking stack out of the equation.  It follows roughly the same length limits as Gigabit ethernet.

Fibre Channel uses optical fiber, and is essentially unlimited in length.  Fibre Channel systems can be VERY fast (if you use FC native drives), but are Bentley-cost, as referenced above.  You can build a kind-of hybrid with FC + SAS, and use cheap SATA drives, but the host adapters and converters are all still pretty expensive.  Not typically worth the effort unless you have very specific needs.

If you have an extra available network jack on that motherboard, iSCSI might be an option, but it still doesn't offer much more sustained throughput than Gigabit ethernet to a network drive, with a well-tuned and behaved network.  Might bump sustained throughput to something like 140MB/s or so.  (EDIT: Actually, I looked it up and you can't get anywhere near that performance.  You're basically just making up a portion of that network overhead from using TCP/IP and Ethernet frames.  iSCSI is around 98% efficient on GBE hardware in most tests, so you're looking at ~126MB/s peak throughput.  I get ~118-119 on my home network drives with a relatively simple setup, fed by a fast, local array.)  IOps performance is generally much better on iSCSI, though, and they do have the benefit of being "local" drives (as far as Windows is concerned).  Essentially, you just install a driver in Windows, and designate a particular network jack as an iSCSI jack.  The main benefit is easing the length restrictions for SATA/SAS, though.

Coming soonish?  Thunderbolt.

Thunderbolt, of course, is just PCI Express on an external cable.  It is mega-fast, and super flexible.  The copper version is length-limited, but optical versions are now (finally) starting to show up, which alleviate this.  Over the next few years, I expect Thunderbolt to essentially displace SAS as a low/medium-end connection spec.  SAS will likely survive for high-end applications where you do use a SAS Switch to connect tons of drives, but for everything else, Thunderbolt is much simpler, faster, and more reliable.  The main problem thus far has been licensing and Intel's ability to produce large quantities of the controller chips.  The currently shipping versions of the Thunderbolt controllers (we're now on Gen 2 of the controller ASICs) are getting higher yields than the originals, and a new version is coming with Haswell (or so) that will be even cheaper/easier to produce.  For a while there, the problem was that essentially Apple was buying ALL of the controllers Intel could put out.

SATA itself is probably not long for this world.  The engineering group in charge of the spec has essentially abandoned it, and they aren't likely to produce a "next gen SATA" spec.  That's because SSDs really need an order of magnitude more capacity than SATA is capable of, and PCI express is just a much better choice.  In the future, storage is likely to be connected directly to the CPU via PCI express (or some future variant) internally, and via Thunderbolt (or some variant) externally.

If I ever replace my miniSAS setup, it will almost certainly be with a Thunderbolt system using optical cable to put the RAID box away in a closet.  But I have time.

[6233638]

Thank you for the detailed post on SAS vs SATA.

So for consumer use, it basically seems to come down to: SAS controllers can handle far more connected drives via port splitters (though it may reduce performance) and they let you use breakout cables rather than wiring up a SATA cable for each drive, so in a case with 24 drive bays, you are only hooking up 6 SAS breakout cables rather than 24 SATA cables - and less if you have a SAS backplane. (I assume just using straight-through SAS cables rather than any kind of breakout)


I bought a copy of Drive Bender, and HD Sentinel today, and it's almost worrying how easy it was to set them up.

Other than a couple of mistakes; I didn't change the path of Special Folders (fixed via Regedit) and ending up with Media Center complaining that all files were missing, and re-importing everything again from the storage pool (deleted everything in the library that it had imported off the pool, then used the rename tool to change the paths to point existing entries to it) and fixing some NTFS junctions, it basically seems to have taken care of itself. Most of those would not have been issues either, if I had used the same drive letter as the previous drive when setting up the pool. (Special Folders and NTFS junctions were all pointing to K:\)

Right now I'm just leaving it alone to balance across the drives, and duplicate files that I want an extra copy of.

It's really nice just having the pooled disks show up as a single 10TB drive in Explorer though.

[6233638]

OK - things are not going as smoothly as I had first thought.

I can't use MakeMKV to save anything to the storage pool, and Internet Explorer refuses to download anything there.
EDIT: Reinstalling fixed MakeMKV. Scratch that, it's giving me errors again. Just fails immediately.
I have no idea what to do about Internet Explorer.

[glynor]

So for consumer use, it basically seems to come down to: SAS controllers can handle far more connected drives via port splitters (though it may reduce performance) and they let you use breakout cables rather than wiring up a SATA cable for each drive, so in a case with 24 drive bays, you are only hooking up 6 SAS breakout cables rather than 24 SATA cables - and less if you have a SAS backplane. (I assume just using straight-through SAS cables rather than any kind of breakout).

Right.

Physically, SATA and SAS are identical.  The drive connectors look EXACTLY like SATA connectors, and they're electrically compatible.  If you hook a SAS drive up to the SATA ports on your motherboard and PSU, the drive will spin up, the controller just won't understand what language it is speaking.

SAS (when using "real" SAS devices) does provide a few extra benefits that I didn't go into, because they're unlikely to apply for most consumer applications.  Because it is based on the SCSI signalling spec, while SATA uses a modded version of the venerable ATA spec, SAS has "newer, fancier" capabilities.  Or, at least, it did.  Lots of what SAS can do over SATA has been later added to the SATA spec (SATA 6G includes full expander support, native command queuing, and a bunch of the rest).  SAS has much, much better "SMART" reporting than the old ATA SMART system (though, when the drives cost 5-10X as much, is this really a benefit?).  The expander support for SAS is much less "dumb" than what is available for SATA.  For example, those SAS Expanders can connect a storage pool to more than one host simultaneously, they support daisy chaining, and other similar capabilities.  Modern SATA can be considered basically SAS-lite speaking an old, hacked-together, weird language.

For high-end applications, SAS has some great benefits.  It is a way to implement a storage network with 10gbps performance with lower cost than real 10GBE, and better performance for storage-specific needs.  Infiniband mesh networks are built on top of SAS hardware, for example, and that's a way to have distributed, super-high-performing massive storage systems connected to (and accessible from) multiple high-needs consumers (HPC clusters and whatnot).

EDIT:  I also didn't mention before, but iSCSI is also available in a 10gbps version, using generic 10GBE host adapters.  This can be a pretty cost-effective way to get high-throughput data connections without having to pay the massive cost to upgrade a network switch.  You'd just need two 10GBE NICs then, and appropriate twisted pair.

There's also the new Fibre-Channel over Ethernet standard (FCoE).  I know essentially nothing about this system, though it seems odd as the FC data signaling system is a bit less efficient than SAS or iSCSI, so why would that be a benefit unless you have existing FC infrastructure to support?

[6233638]

Uh... I thought Drive Bender wasn't supposed to touch the underlying file structure on your drives?



 

[InflatableMouse]

It uses standard NTFS formatted drives, I didn't mean to imply its not touching files and folders. Obviously it has to or it can't do its work.

Sorry for not making that clear.

If you go into one of those folders you'll see the entire structure is still there, its just moved into that folder. Once the file balancing is done, you should see the same structure on all your drives with the files spread over all the disks.

[jmone]

You are looking at the structure on the physical drive not the Drive Pool which you should use.

[glynor]

HDD will fail, esp when you have a lot of them.  You will also tend to have multiple failures at the same time.  Murphy's Law.

This is an important point to consider when building any massive storage system, and one that is often overlooked.

When people consider RAID-5 (or something similar with redundancy capable of handling a single-drive failure without data loss), they often think something to the effect of:  "Well, sure, multiple disks can fail, but what are the chances that it is going to happen at the same exact moment?"  Unfortunately, in practice, life isn't so clean.  There are a few factors that contribute to it, but multiple (or quickly sequential) drive failures are unfortunately more common than it would seem at first glance.  They are not randomly distributed.  Drive failures follow patterns.  Here are some of the reasons why:

1. When you use multiple drives in an array or storage pool, you are, by definition spreading the work out among multiple drives.  Striping does it in a very consistent manner, of course, while pooling might be less consistent, but in any case, you are spreading out the "wear" across multiple devices.  Rather than beating on one "part" hard, while others sit relatively untouched, usage and wear will be spread across the drives.  Good for longevity.  Bad near the end of the usable lifespan.

2. When you build an array or big pool, you usually buy a few drives at once, generally the same make/model, and often they end up shipping you drives from the same "batch" at the factory.  If one has a particular defect, it is more likely that they all do.  And, in any case, since wear is more distributed, they all "age" at a similar pace.  This impacts pooled drives less than drives in a RAID, but it still impacts them.

3. Newborn drive failures.  I've had this happen more than once in my 10 years.  You have an existing array.  You decide to upgrade the storage capacity.  You buy a bunch of new drives, plug them in, set up the array or pool, and then 2 of them fail (without warning and despite initial testing) in the first 2-4 weeks.  If this happens at the exact wrong moment, you can be in a world of trouble if your system can only tolerate a single drive failure.

4. SMART isn't that smart.  Google's oft-cited massive study found that if SMART errors occurred, they do correlate with future failure (a drive with a single uncorrectable SMART error was 39 times more likely to fail in the next 60 days).  However, 56% of all drive failures showed no warning in SMART at all.  I've seen this myself.  I've had drives fail in an array that pass all the SMART tests I can throw at them with flying colors, but they're still dead (and don't work reliably even as single-drives).

And, yes, drive operating temperature, up to a certain "crazy limit" was not an accurate predictor of failure.

5. Parity data rebuilding.  This is one of the things that most commonly gets RAID-5 users... Often in concert with #1 and/or #2 above.  What happens when a drive fails in a RAID-5 volume?  Well, you don't lose any data, but the array is now in a "critical" state.  Performance is reduced because the data on the failed disk has to be reconstructed on the fly from the parity data spread across the other disks in the array.  So, you plug in a new drive (or you have a hot spare waiting for this eventuality), and then... The controller needs to read every single used sector on the array and reconstruct the original contents of the failed disk before the array becomes "healthy" again.

This is a massive opportunity for additional failures, especially when you have a few "still working" drives in that array that are the same make, model, age, and usage pattern/wear as the one that failed!  The data rebuild itself is a common "cause" for additional failures.  When that second drive fails, then boom, the whole thing is gone.  Couple this with using consumer drives that don't handle error correction in quite the way RAID host controllers expect, and you can have big problems.

That's why I said RAID-6 if you can manage it.  Or, have a robust backup scheme (which is what I do).

Now, how common is this?  Not very.  I've never lost an entire array in 10 years (knock on wood), though I've had one or two scares (second drive failures just after a rebuild finished is enough to make you come up with a good backup strategy, let me tell you).  But, still, it is more likely than you might think, and my arrays have always been relatively modest (the "biggest" array I ever built myself was a 16-drive array, with awesome 250GB PATA drives).  The more drives you add to an array or pool, the risk goes up logarithmically.

[6233638]

You are looking at the structure on the physical drive not the Drive Pool which you should use.

To clarify, things have gone horribly wrong and that's what I've been left with after trying to remove my drive pools, because Drive Bender wasn't working out for me. (big performance hit, incompatible with a number of programs I use)

I can't even get to the desktop now if the Drive Bender service is allowed to run, and Windows complains about the recycle bin being corrupt on my drives if I try to access the folders. (and I don't want to delete it, as I have no idea what the 6GB that's apparently in them is)

Update: I decided to just erase the corrupt recycle bin data. I don't care about what was in there if it was actual recycle bin data, my concern is that I normally never use it, so 6GB was troubling.
I can now at least access the files on the disk. Even after removing and reinstalling Drive Bender, my system will not reach the desktop with it running now though.

[WinoOutWest]

FWIW: I am 6 months into a FlexRaid setup.  So far I am quite pleased.  Even cobbled together some of my older 1TB that I haven't used for a while and re-purposed them as a 3tb drive.  No Hiccups yet and it really is nice having all my content in one big 14tb drive

[glynor]

To clarify, things have gone horribly wrong

I'm really sorry to hear that.

It is incredibly frustrating that we don't have a better native filesystem on either Windows or OSX.

OSX, at least, has CoreStorage.  Which is awesome (and handles drive pooling brilliantly - if you were on OSX you wouldn't need anything like Drive Bender at all).  Unfortunately, they plopped this awesome, modern system on top of HFS+.  That's a little like strapping an iMac onto the side of the Titanic and calling it modern.  Sure, the drive pooling works brilliantly and you can do things like Fusion Drive, but HFS+ still corrupts itself on a monthly basis, has all sorts of performance problems, and has no data integrity checking whatsoever.

And then, on Windows, there's ReFS.  ReFS has a lot of promise, I think (it is a B+ Tree Node based filesystem much like Btrfs), but there is way too much stuff that it doesn't support right or at all.  It basically is only 1/2-way finished, and that's why they didn't ship it in a consumer-facing OS.  At least, unlike HFS+, NTFS is not brain-dead and ancient.  But, this is the year 2013.  Why the heck are we still using filesystems designed for 1.33MB floppy disks?!?

[glynor]

FWIW: I am 6 months into a FlexRaid setup.  So far I am quite pleased.  Even cobbled together some of my older 1TB that I haven't used for a while and re-purposed them as a 3tb drive.  No Hiccups yet and it really is nice having all my content in one big 14tb drive

That's good to hear.

FlexRaid always looked very interesting.  But that guy seems to be a bit of a showboater, and can't make up his mind about what he wants to do with the product.  NZFS has been "coming" for quite some time now, with no real further public info.  And, when I look at the current product, it seems somewhat... Complicated, and the easy-to-find info is all marketing-speak with not a lot of substance.  I'm sure there's a lot of detail in his forums (and they do still seem to be working on FlexRaid), but...  I don't know.

I decided a while back to wait and see how the NZFS thing works out.  Or hope for something "professional" from Microsoft or others.

[6233638]

I'm really sorry to hear that.

Well on a more positive note, everything seems to be OK. I don't know about the integrity of the files yet, but aside from needing to sort out the location of everything, I don't think I've actually lost any data - I was just a bit panicked when I saw that and was then unable to access anything because it was complaining of data corruption. (on the Recycle Bin - but at that point I didn't know if it went beyond that)

I don't know what exactly has happened to leave Drive Bender in such a state that I can't even get to the desktop with it enabled, and it just hangs if I try to start the service manually. (after disabling it in safe mode so I can get to the desktop)

But I won't be messing around with that again, and I will be ordering those external drives ASAP.
As mentioned in an earlier post, I have been keeping my music backed up every couple of weeks and I'll be adding another drive, moving that to a weekly rotation.
And at the very least I will also be consolidating all my DVDs on a single drive and buying an external disk to store a copy of them on too.

Beyond that, I'm not sure what I will be doing yet.
It's a shame, as pooled storage seemed like an easy solution if it had worked well.

I thought I was managing OK by using my 4TB drive as my main storage disk for larger files (games etc.) alongside the boot SSD, and only storing videos on the others, letting MC/Windows Libraries manage them.. but having it show up as a single 10TB disk and not having to worry about things like which drive files are located on, and how evenly spread out they were was nice.

But I was even running into basic issues like it reading from one drive, and writing to the same one, rather than writing to another one, which slows down certain tasks significantly.

And then, on Windows, there's ReFS.  ReFS has a lot of promise, I think (it is a B+ Tree Node based filesystem much like Btrfs), but there is way too much stuff that it doesn't support right or at all.  It basically is only 1/2-way finished, and that's why they didn't ship it in a consumer-facing OS.  At least, unlike HFS+, NTFS is not brain-dead and ancient.  But, this is the year 2013.  Why the heck are we still using filesystems designed for 1.33MB floppy disks?!?

Well you get it in Windows 8 if you are using Storage Spaces - the problem with Storage Spaces (at least the obvious one to me) is that it requires you to use formatted drives - you can't just convert existing disks to work with it.

What exactly would be different if you had a filesystem that was designed for today's hardware?
I was under the impression that NTFS/ReFS had been doing a good job staying "modern" unlike HFS+

[WinoOutWest]

That's good to hear.

FlexRaid always looked very interesting.  But that guy seems to be a bit of a showboater, and can't make up his mind about what he wants to do with the product.  NZFS has been "coming" for quite some time now, with no real further public info.  And, when I look at the current product, it seems somewhat... Complicated, and the easy-to-find info is all marketing-speak with not a lot of substance.  I'm sure there's a lot of detail in his forums (and they do still seem to be working on FlexRaid), but...  I don't know.

I decided a while back to wait and see how the NZFS thing works out.  Or hope for something "professional" from Microsoft or others.

Having read many of your posts I can't imagine the "Glynor" finding flexraid complicated.  I beamed with pride when I read your post.  :-)  Seriously though I am technical but more of a hack when it comes to this stuff.  Kind of a "little knowledge can be dangerous" type.  I'm sure you would sail through a setup if you wanted to and that's not to say it isn't without a learning curve.  I like the fact that I've got some parity in my setup.  I'm told it ain't a performance screamer but I don't particularly care as it is a media server not a performance setup.  My worker bee PCs run off SSDs.

[InflatableMouse]

Well on a more positive note, everything seems to be OK. I don't know about the integrity of the files yet, but aside from needing to sort out the location of everything, I don't think I've actually lost any data - I was just a bit panicked when I saw that and was then unable to access anything because it was complaining of data corruption. (on the Recycle Bin - but at that point I didn't know if it went beyond that)

I don't know what exactly has happened to leave Drive Bender in such a state that I can't even get to the desktop with it enabled, and it just hangs if I try to start the service manually. (after disabling it in safe mode so I can get to the desktop)

But I won't be messing around with that again, and I will be ordering those external drives ASAP.

Glad to hear the damage is under control. I honestly didn't expect you to try it out and I was a bit surprised to see your post last night.

It all sounds familiar, I didn't expect this kind of trouble to come up with the latest version of Drive Bender. The difference must be in the fact that I already have a pool up and running while you are starting from scratch. Even when I reinstall, the pool and its configuration is already there just waiting to be picked up.

I got to ask though, did you somehow manage to include your Windows boot drive into a pool (this should be impossible), or did you move your user profile folders to a drive that you included in your pool? It sounds an aweful lot like that is what happened because I ran into the exact same stuff when I moved my profile folders to a pool once. Not that I blame you for your trouble, DB should not allow this in the first place and give a big fat warning about it. When logging into windows, often your profile folders are needed before Drive Bender finishes doing whatever it needs to do and you will get weird errors like that Desktop folder unavailable. DB can get "stuck" on something and the services won't start or take a long time. Often a reboot will fix things and waiting before logging in helps too.

Dump it and don't look back. When I look back at my troubles it wasn't worth it and the only reason its still running is because for the moment, its troublefree. I have zero tolerance for it though and at the first sign of trouble I'll be dumping it as well.

[rjm]

Being the resident doomer I have to comment that this is an example of our civilization having passed its peak complexity. Most new technologies are so complex they require more wealth (energy) than is available to properly develop and maintain them, and
the unintended problems they cause often exceed their intended value. Declining per capita net energy means simplicity is coming fast, whether you want it or not. Better to embrace it before it embraces you.

[6233638]

Glad to hear the damage is under control. I honestly didn't expect you to try it out and I was a bit surprised to see your post last night.

Well it seemed like it was supposed to be a stable/mature product, it used NTFS (and I thought it left your file structure completely untouched) and it wasn't free, so it seemed reasonably "safe".

Won't be making that mistake again!

I got to ask though, did you somehow manage to include your Windows boot drive into a pool (this should be impossible), or did you move your user profile folders to a drive that you included in your pool?

My Documents/Downloads/Pictures/Videos were in the pool. (And Windows was pointed to the right location via the registry) Anything else was still on my SSD.

DB can get "stuck" on something and the services won't start or take a long time. Often a reboot will fix things and waiting before logging in helps too.

I did try waiting a reasonable length of time, and nothing was happening. (No activity at all) Starting up the service manually took about 10 minutes and just threw up an error about it not being able to start.

Dump it and don't look back. When I look back at my troubles it wasn't worth it and the only reason its still running is because for the moment, its troublefree. I have zero tolerance for it though and at the first sign of trouble I'll be dumping it as well.

Already have. Because it managed to get my system in a state where I couldn't even get to the desktop, even though I have that sorted now, I think I am just going to wipe the machine and start fresh.


What’s worse is that I have been doing a little reading into things like Storage Spaces (seems like it is not the right thing for me) ReFS (sounds like it handles file integrity better than NTFS, and the next version of Windows will allow it on the consumer side) and Server 2012's Deduplication - some people are reporting significant savings with it enabled. There are ways to hack ReFS and Deduplication into Windows 8, but there’s no way I will be going near that now. It's too bad Microsoft only build things like that into the server version of the OS.

Server 2012 Foundation is affordable, and has limits like 1 CPU and 15 users on it, but I wonder if its worthwhile to use for those features…

[jmone]

Stuff like this makes me feel ill thinking that all the data could be lost..... while it cost me $ i am happy to have a full backup in a seperate location that also protects from inadvertent deletion, fire, theft as well the case where I trashed all my drives....which is far to easy to do!

[InflatableMouse]

I wouldn't buy Server 2012 for storage spaces. Read up on it as there are quite a few problems with it and configuring it properly can be quite a challenge. I certainly wouldn't want to use it at home.

[InflatableMouse]

But I'm very interested to hear what you come up with, so please do keep us posted about what you decide.

[6233638]

Stuff like this makes me feel ill thinking that all the data could be lost..... while it cost me $ i am happy to have a full backup in a seperate location that also protects from inadvertent deletion, fire, theft as well the case where I trashed all my drives....which is far to easy to do!

As I said, it ended up not being too bad. For some reason I was unable to access anything on the drives because Windows was complaining about a corrupt recycle bin on all of them and once I cleared that I was able to pull my files out of the GUID folders Drive Bender had created - but every drive had a copy of the entire filestructure, and only some of them actually had the files in them. (balanced across the disks)


It wasn't that much of a problem - actually the thing that panicked me the most was when I was unmounting the drives via Disk Management. At this point it was almost 2am and I was tired.

Right-click, Change Drive Letter and Paths, Remove
Right-click, Change Drive Letter and Paths, Remove
Right-click, Change Drive Letter and Paths, Remove
Right-click, Change Drive Letter and Paths, Remove
Right-click, Change Drive Letter and Paths, Remove
Right-click, oops, it was at the bottom of the screen so I moved the menu position for you delete volume and then, because UAC has trained me to automatically hit "shift-tab, space" any time a yes/no prompt comes up when doing anything system related, I instinctively hit that without even thinking and deleted one of the drives. Oops.

Fortunately, a quick Google search pointed me to reports of people doing the same thing and using Partition Wizard to fix it (which looks like a horrible freeware tool that I would normally avoid) and it was actually a simple fix, but I could have done without that.

Right now I'm in the process of moving all the data off that disk (with my other drives and an external one, I just have enough free space to do that) and will then repartition it with Disk Management before using it again. After working in PC support a decade ago, I don't trust any partition software outside of what's built into the OS. I'm sure the tools are better now, but I still don't trust it.


But this experience has just shown me that I want to stick with the system I was using before - just keep adding drives as simple NTFS volumes, use Windows Libraries to "pool" the files together when I need to see everything at once, use Acronis to back up important files to other drives when I need duplicates, and use external drives on a weekly rotation for real backups.

It might require some extra manual work, but I don't think it gets much safer than that.

The only better option would be if I had a duplicate for every drive in my system and let it handle daily backups, or something similar. (without using RAID or anything else, just simple file copies)

I wouldn't buy Server 2012 for storage spaces. Read up on it as there are quite a few problems with it and configuring it properly can be quite a challenge. I certainly wouldn't want to use it at home.

No, it's not the Storage Spaces (Windows 8 has that) but ReFS and Deduplication that interested me - though you can't use both together.

I've had trouble in the past where using the latest drivers for my Marvell SATA controller would stall transfers or transfers would appear to work, but the data would be corrupt. (had to re-rip a number of discs as a result) So ReFS's additional checks would be nice.

But similarly, I've heard reports of Deduplication (which only works with NTFS) saving hundreds of GB on 1TB+ drives, which also sounds pretty nice. (but I guess it won't save much with media files?)

Unfortunately neither of them work on the boot drive, which is where deduplication would probably be most useful, as it's a small SSD.

[InflatableMouse]

Sorry, I used the wrong name. I didn't mean Storage Spaces I meant the File and Storage Services in Windows 2012.

[glynor]

Deduplication typically carries a fairly severe performance penalty.  Generally it isn't something you'd want to enable on a boot drive, or at least not a ZFS-style "live dedupe".

ReFS, to be clear, is currently only available in Windows Server 2012.  They did NOT ship it as part of Windows 8 (contrary to early reports before the RTM and betas).  They didn't ship it because it isn't done.  I'm fairly certain it won't come with Blue either, but maybe the version after that.  We'll see.

NTFS is less old and creaky than FAT or HFS (which are of similar vintages).  And, just like HFS, they've made substantial improvements to NTFS over the years, but it is still certainly not a "modern filesystem".  Just one example?  No automated data checksumming.  It does "protect" the filesystem metadata via journaling, but it does not protect the actual file data from corruption.  There are tons of other examples.  Basically, if you EVER have to run CHKDSK on a volume to repair a filesystem metadata issue manually, then it isn't a modern filesystem.  These problems were solved more than a decade ago by other filesystems.

[6233638]

Deduplication typically carries a fairly severe performance penalty.  Generally it isn't something you'd want to enable on a boot drive, or at least not a ZFS-style "live dedupe".

From the little reading I did on it, I don't think Server 2012's deduplication is live, and is run on a schedule instead - and only on files which are more than X amount of days old. (I think five by default)
On an SSD, I don't think you would incur the performance hit that you would get using deduplication on HDDs. (because the files being read are no longer contiguous)

ReFS, to be clear, is currently only available in Windows Server 2012.  They did NOT ship it as part of Windows 8 (contrary to early reports before the RTM and betas).  They didn't ship it because it isn't done.  I'm fairly certain it won't come with Blue either, but maybe the version after that.  We'll see.

I thought Storage Spaces made use of it, but you might be right. Early reports suggest that ReFS is included as part of Windows Blue - but that was the case for Windows 8 betas as well, I think.

I'm not sure what you mean about it being unfinished though - should it require feature parity with NTFS to be considered complete? Surely they must have thought it was ready if they shipped it in Server 2012.

NTFS is less old and creaky than FAT or HFS (which are of similar vintages).  And, just like HFS, they've made substantial improvements to NTFS over the years, but it is still certainly not a "modern filesystem".  Just one example?  No automated data checksumming.  It does "protect" the filesystem metadata via journaling, but it does not protect the actual file data from corruption.  There are tons of other examples.  Basically, if you EVER have to run CHKDSK on a volume to repair a filesystem metadata issue manually, then it isn't a modern filesystem.  These problems were solved more than a decade ago by other filesystems.

Well that's why I'm interested in ReFS. (but not interested enough to hack it into Windows 8 - even if it seems relatively safe)

[6233638]

Well my WD Passports turned up. I'm more than a little disappointed to find out that they're plastic - the Toshiba Stor.E Alu drives I bought last time around feel a lot more solidly built, and metal conducts heat a lot better.

But I don't think there are any better options if you want a 2TB portable drive.

To guarantee my data is safe, I'm wanting to run full tests on the drives before copying anything over - what would you recommend? Is HD Sentinel's Surface Test sufficient?

I'm hoping that it's just because it's testing the entire drive, but it's only going at about 20MB/s on both (about 24 hours to complete!) and I normally get around 60MB/s with file transfers to the 1TB Toshiba drives.
EDIT: Performance is probably fine. I just looked at the Task Manager and the drives are writing about 70MB/s and reading 20MB/s simultaneously.

[jmone]

Is HD Sentinel's Surface Test sufficient?

Yup - if you are really worried you can get HDS to do one of the Read + Write test on every sector but they are much slower than just read.

[6233638]

I figured why not just let HD Sentinel reinitialize the disk surface to be completely sure there weren't going to be any problems.
After all, what's the point in a backup drive if your data isn't going to be secure on it? That should hopefully be enough to see if it's going to fail early, or work fine.

[jmone]

That will work but is also pretty slow.  I've used it to "refresh" disks I pull out of my setup prior to re-using or gifting them on as it puts it back into the "original" state.

[6233638]

That will work but is also pretty slow.  I've used it to "refresh" disks I pull out of my setup prior to re-using or gifting them on as it puts it back into the "original" state.

Well that's the point, isn't it? If you just do a quick format/check you don't know whether a drive is going to fail early or not.
Anyway, 7 hours in, it's about 25% done.

[jmone]

The read test is quicker however and still makes the HDD drive access each sector and report if they are weak or bad.  The weak is done by recording how long they take to response to the read request.  I'm not sure if you get any reported weak info with the re-initialise but the good thing is you can run the read test anytime (eg with data on it) as it is non-destructive.

[6233638]

Well I don't know if anyone is still interested, but the WD drives passed testing (as expected) and peak transfer rate seems to be about 115MB/s when copying data over to them. (as reported by Task Manager)

It's amazing how far we've come in just a few years, from dealing with painfully slow USB2/FW400 drives that were bulky and required a massive power brick, to these tiny bus-powered USB3 drives which basically operate as quickly as internal SATA drives do. I still hate that weird USB3-mini connector though. It makes sense when you consider that they had to include backwards compatibility, but it just looks like it would be so easy to break.

[glynor]

I still hate that weird USB3-mini connector though. It makes sense when you consider that they had to include backwards compatibility, but it just looks like it would be so easy to break.

+1

Oh, yeah... And you asked what ReFS needs to have to be "done"?  There are a bunch of things that are missing (or not quite working right) in the current shipping version of ReFS for Windows Server 2013, but the big one before it could even be considered viable for a consumer version of Windows is that it has to be bootable.  Right now, it is not.