Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[benn600]

The big issue.  If I ever decide to upgrade my storage system, I will obviously have a few metric tons of data that needs to be "TEMPORARILY" backed up.  Any ideas on how this is even possible?!  I understand that I can re-rip a huge number of DVDs (a huge percentage of my data) but I already had to do that once for this current system.  Do you think there is a place I could rent a 4 TB "redundant" array from for a week?  It might even take two weeks!  Moving that kind of data would take forever.  Actually, it wouldn't have to be limited by the network.  I could hook it up via USB or Firewire (or gigabit networking).  That could get the data flying.

I think this kind of service will become much more common because if I want to add more storage, I'm screwed - with my method of RAID.  More and more people are getting 1,2,3 TB arrays and it is unlikely that they can just buy a second for this purpose.  I'm not really worried about backup.  This is only a temporary holding place while a new array is created.

Take 4 TB of data.  A 10/100 network can move UP TO an average of 10 MB/second.  We'll say a gigabit network (which should outperform USB and Firewire (even 800)) could theoretically get 100 but we'll say 50 MB/sec to be more realistic and probably underestimate.  That comes out to 3 GB per minute.  Or 180 GB per hour.  (that's impressive)  Much to my surprise, that would take under 23 hours.

1. That gives 1 day to move the data from the old array to the temporary array.
2. Another 1-2 days to configure the new array.  If it's RAID 5, one of those days will be for setup and the other for REGENERATING.
3. Another day to move the data back.
4. Problems usually arise so that's another day or WEEK, lol.

Theoretically, 3-4 days would be sufficient for the entire project.  That would be a good business (somewhat) to buy big arrays and rent them out for periods like 1 week.  Unfortunately, there's lots of risks.  They could break your multi-thousand dollar array.  Plus, I wouldn't really want to pay anymore than an absolute max of $100.  That actually wouldn't be that bad....52 weeks at $100, $5,200 and the array would probably be $3-4 thousand.  You could get a lot more than one year out of it, too.  Just replace individual failed drives as they occur.

[mlefebvre]

Mind boggling....

I though I was bad with a bit under a terabyte.... :-)

How many media files do you store on this array?

M.

[benn600]

Close to 400 DVDs and close to 1000 CDs and thousands of pictures.

Lol.  So where can I rent 4 TB for a small amount?

Great question: I'm really thinking that if I get the chance, this summer, I'd like to upgrade to 15-drives.  So, I'm wondering if there is a solution I can buy that will let me do this....

1) Needs to hold 15 drives minimum.  I'm not going through all this effort to add 2-3 drives.  I'm adding at least 5 more drives.  By then, the drives should be $150, so $750 for the drives...but they will probably be even a bit cheaper.

2) I absolutely must have RAID 5.  Obviously, I need some redundancy or this would be a death trap.

3) I actually need a place to rent 4 TB from.  I may call my local computer stores if I eventually decide I am willing and ABLE to perform this upgrade and see if they can throw something together for me.

4) I want it to hold all the drives, have hardware RAID (not absolutely a requirement), and it needs good cooling.  I've seen the huge 10-drive cases.  All of them usually have rails that are quite nice.  Any ideas?  15-drives is a ton!  Don't hesitate to recommend up to 20-drive capable units...that would be too much but I'd love to read about them (for future reference).

I still am reaching for that "end point" where I will have enough space to run for 4 years without the need to upgrade.  I still have a lot of analog photos to digitize--20 thousand +...not to mention the standard increases on my CDs/DVDs/digital photos, etc.  Once I get all the photos digitized and the other things laying around that I need to handle, I'll be able to slow down on the data growth factors, not worrying so much about further increasing my storage.

I can't imagine how neat it would be to say 7 TB, either.  I feel empty right now because I'm down to 1.5 TB free.  Add a few months of data and I'll be close to 1 TB.  That ain't much space!!

[hit_ny]

I think you better check out how much tape drives cost

[benn600]

I am somewhat set in my ways and don't think there is much anyone can do to change my goals here.  Where can I find a 15-drive ATX case?  Ideally it could hold 16 because here again, I need a boot drive.  I'll probably end up using Windows XP raid again since it does work so well if not abused and it's free...and I don't think 15-drive RAID 5 could be had for less than the price of a car.  I haven't even seen more than 8 port cards.

[benn600]

I found an amazing case (16 drives),
http://www.amtrade.com/computer/file_server_enclosures.htm
but the price is way too high at $600.  I was hoping for a $100 cheap case.  It just has to be a cheap case that can hold 16 drives!

[johnnyboy]

I could have sworn that I'd posted replies/msg's in this thread already and so had others yet I'm looking at it now and wondering if that was another similar thread or whats happened?

Anyway, just wanted to say, your moving way past 'home user' at this point with the number of drives you want and even by commercial standards, your moving into medium to large business with the kind of size storage requirements you are looking at. As soon as you hit this range, chances of finding cheap equipment starts vanishing. I just cant picture ANYONE making a cheap $100 case to hold that many drives.

16 drives at $180 each say is:  $2880. With that type of investment thrown into the drives themselves, its expected that you would be willing to spend money to provide the additional hardware required for them.

You seem to be willing to spend a never ending amount of money on these drives and trying to get as many as you can of them, yet are not willing to spend money on the additional hardware required to support them. The more drives you are adding to this hacked windows raid 5, the greater your chance of having a failure that is going to totally destroy the WHOLE 400Tb or whatever size you have then got it to. I think you need to step back a bit from this whole thing and think about just how much cash you are pouring into all these HD's.

I think you should start investigating proper solutions with the number of drives you are getting. Solutions that allow for multiple drive failures and solutions that let you just plug in another drive if you need it and it adds to the space without having to backup all your drives each time then re-build from scratch etc. If you plan to keep adding at this rate you dont want to have to be doing that each time you get a new drive - you want to be able to just add a new drive by itself as and when you please by just plugging it in.


Anyway, to answer your question about the cases - I'd say dont bother about a case at this point, not for what you want. Just build a custom case yourself thats designed for just basically holding tons of hard drives and providing them with great cooling - shouldn't be too hard. All you need is two sheets of metal with holes drilled through the sides to allow you to screw the drives into place in it with cooling being aimed through the middle to keep the drives from over heating. I'd use some rubber washers as well to stop vibrations and give some damping effect. Leave a couple of cm between each drive as well to allow good air flow and theres your case.


Just as an FYI - you are aware that google, who has pretty much one of the biggest HD setups around I'd imagine, just uses a bunch of cheap computers to provide all its storage - just tons of cheap computers pieced together to give huge power - just a concept that if applied to your setup might help.

[benn600]

Building a case myself is a great idea.  I will step back and just think for a while.  I am not necessarily absolutely planning on this, but it's nice to dream.  And besides, if 500 GB drives came way down in price to, say, $100?  That would be reasonable at $500.

Remember I already have 10 drives so I only (lol) need 5 more.  The 16th would be a smaller boot drive.  I have a nice 500 watt power supply that I could use.  I got it for free after rebate, though so who knows how great it will be.

[benn600]

You gotta admit that these cases:
http://www.amtrade.com/computer/file_server_enclosures.htm
look pretty amazing.  They have wheels!  They must be huge!!  I don't doubt it.  Probably twice as tall as my largest case (which holds 8-10 drives).  I like the wheels...roll the thing around.

Then check out the double wide 32 drive version!  I really would like to have the 16-drive version!

Addition: they seem to be 5.25" bays.  That would make it more than double my largest computer's height.  Does that mean I would have to buy 16 5.25" to 3.5" adapters?  I already have 4-5 of them but wouldn't want to have to mess with buying a ton of 'em!

[newsposter]

If you are 'set in your ways' then stop asking advice.......

[johnnyboy]

If you are 'set in your ways' then stop asking advice.......

Nice case, reminds me a bit of the Sun v890's we have at work except thats without the multiple CPU's and 16Gb of RAM that these babies have.

newsposter - dont say that, it's interesting hearing his little journey and adventure. He's basically doing it 'the complete wrong way' according to all accepted norm's for computers so its interesting to see how it turns out and if in fact it is a viable way of doing it.
I know I personally am enjoying following his project, I'd be interested in other peoples projects too - it's always good to get new ideas and see what can be done right now.

[jgreen]

j-hombre:

I agree, I'm learning a lot from watching benn use his skull as a battering ram.  i don't think news wants benn to stop asking, I think he wants him to stop being 'set in his ways"--but what are the chances?

To review:  all benn really needs is a mirrored JBOD and he's good to go.  MC will present the most mangled dir structure as a contiguous media collection, so there's no need for a contiguous RAID volume.  And as for reliability--HAHAHAHAHAHAHAHAHA!

Still, iowa winters can get awfully cold, and that USB rig must be throwing off a lot of heat, so there's a benefit in that. 

[JimH]

ROTFL!  Thanks.

[johnnyboy]

Just reading a bunch of Articles now on Tomshardware about Raid 5 and thought back of you and this.

One point he made got me especially thinking - with all this data your putting on these drives and all the effort and your hacked Windows Raid 5 - have you actually tested that recovery works properly with it all? After all, if your doing Raid 5 so you can get redundancy, its definitely best to test it and make sure it actually will work.

Take out one of your drives, replace it with a blank drive and see if Windows (with its hack) can actually rebuild that drive for you properly or if it just gives you a bunch of corrupted data.

As for what the chances are - anyone who will go to this much effort deserves a pat on the back even if they are going all the way around the world just to get 10m away   I'm far too lazy for that.

[benn600]

You know, setting up what I've set up is actually about as simple as it gets.  Although I must admit hardware raid solutions are even a tad easier because you don't have to rely on the OS.

But referring to the redundancy testing, I agree completely that I should have actually tested recovery.  Toms Hardware is the place (I think) I first learned about this RAID 5 stuff (in Windows specifically) and it got me started.  I immediately thought--wow, I could have RAID 5 with what I already have...I wouldn't have to buy anything more! (or something like that)

If I recall correctly, recovery did seem to work well.  I might have tested it but I know without a doubt that if a drive fails, the array will keep functioning.  That I've ran into on multiple occasions (unfortunately) but that's never happened with this current 4.5 TB array--luckily!

Again, I'll ask the question: where can I find a 10-port RAID 5 hardware controller card?  Are they even available in standard PCI (not PCI-E)?  My computer that I'm using for this doesn't have PCI-E.  Then find me a 15-port card!  I think to get RAID 5, all the ports have to be on the same card, correct?  Generally speaking, 32 drives is the maximum for any RAID array, too, right?  So I guess I should just go right to 32 drives and be DONE with it once and for all?  Another 22 drives?  hmmm.  That doesn't seem very affordable now that I think about that.  Can we dream, though?

15.5 TB - but is it enough?  TB are so huge yet seem so small.  I have a WHOLE terabyte!  lol.  I have TWO whole terabytes...lol  What sounds better, 500 gigabytes or 15 terabytes?  I personally like 500 better than 15 (in some cases) but understand the issues present.

$4000 for the drives (in addition to what I have)

Oh and that 32-drive RAID 5 card...where can I find that?  Windows would happily host 32 drives on a single array.  I would love to see that .. even just for a few minutes .. actually running.  Microsoft even says that's the limit.  Why is it 32?  Now that I think about it, at 4-ports per SATA card and ~6-7 card slots, you're only able to get 24 ports.  So you'd need 6 or 8 port cards.  So around 6 cards at $100 each...$600 for cards + $6,000 for drives + $1,000 for base hardware.

that's one heck of a server.

[johnnyboy]

At some point you are going to have to consider is this:
Raid 5 can easily handle ONE failed drive, but any more than that and it packs up. Now sure windows might be able to support 32 drives - but do you really want to risk ALL your data on all your drives in one big swoop??

At some point you are going to have to step back and say 'ok, as great as one drive sounds, its just not a viable realistic GOOD solution' and get over that.
Now consider instead you use standard SATA Raid cards - 6 drives together means that if you have 18 drives - you can have 3 seperate drives fail and the system can still keep running in best case scenario (one drive fails on each raid). Compare this to your windows setup and you can instantly see the huge advantage.
There's probably a reason they limit how many drives can go onto one RAID card and thats because no one would want to risk TOO much data all in one fell swoop - theres no reason for it. I know not a single business would do it - they'd tell their staff to get over their issues with multiple drives and just deal with it.


There is of course the other fact that you can mount drives in XP I believe as folders. You could in theory just have one as your main drive, then mount all your other drives as folders within that.


Windows can do 32 drives you say - great, windows also claims it can run stably and securely - I'm yet to see that happen and I most definitely wouldn't risk all my data on Windows claim of anything. I can just imagine windows trying to properly manage and spread data over 32 drives
"ok, now where does this 1 go, was I on drive 13 or drive 14 a min ago, that stupid svchost just distracted me for a minute and I cant remember where I was now'.

Plus your CPU will be screaming trying to split the data and write it to this many drives. With hardware windows just says 'oi raid controller, write this 500Gb file, now what was it you wanted Mr svchost" while the controller gets on with the rest.

Sure you might have to pay for some more hardware, but you know what - if your doing a project this size anyway you should expect to be doing it properly not just thinking along the lines of just only spending money on a never ending supply of drives and not having to invest in anything else.

You put 32 drives into one system that systems OS can destroy the whole lot of data, a virus on it can destroy the whole thing, the software hacked raid of the OS can destroy the whole thing.

You put 32 drives into 6 raid controlers instead and the OS can destroy the data still, but otherwise if a controller goes it only destroys 6 drives worth of data, not 32!

You split those cards into 3 systems (when I say systems I mean bare minimum - these aren't desktop computers, they're functioning boards to control stuff so you just need very low spec (and extremely cheap) motherboard, CPU, RAM - thats it, or just buy second hand old systems off ebay for next to nothing) and once you've split them into 3 systems you then have an even more secure setup - a virus hitting your OS and trashing it and deleting everything on it only takes down 1/3 of your data - the other 2/3 is completely safe still.

Instead of having to back up 52342Tb of data so you can add a new drive each time, instead you can just incrementally build up a new server box to add to the rest of your data - it can grow as you do without your current situation of having to scrap it all and start from scratch each time.

Before you start saying how expensive all these extra computers would be, a SUPER quick search of ebay finds TONS of results like this:
http://cgi.ebay.com/Dell-Optiplex-GX150-Desktop-P3-1GHz-256MB-20GB-CD_W0QQitemZ160095465377QQcategoryZ140070QQrdZ1QQcmdZViewItem
just screaming for you to make them into file servers.
I even saw some $20 machines but it just depends what spec you want your file servers to run at. Google builds their insanely huge storage infrastructure from lots of cheap machines - well I say your just as good as google and why cant you too!

Take these systems out their cases, fit them into your home made case your gonna make for all your hard drives and you could have a REALLY cool looking system:

5 drives with a motherboard screwed to the side of every 5 drives controlling them.

It wouldn't look like lots of systems, it'd look like one UBER (wow term stolen for this context) multi core AND controller computer.

[hit_ny]

If you get rack mounted boards,  it will take up less space, but then you're talking about setting up your own data centre !!

How many kWs (units)  is that going to gobble up ?

[johnnyboy]

I'm not personally talking about rack mounted boards (well not commercial ones) - I just mean using normal mother boards.

As to how many kWs - each individual machine can be set to sleep after X and wake on network traffic - to access one film therefore would only wake the server it is on.

At the end of the day though, when your talking about that many hard drives at that cost each for this overall project, the cost of some electricity really isn't going to be anything that ground breakingly bad.

If your talking enviroment then having a system always on in the first place is far worse than just popping a DVD into a player which only uses power for the time the movie is actually playing for.


If you presume each system has 3 PCI slots then you can get 3 SATA controller cards into each motherboard. If each card can hold say even just 4 HD's thats 12 HD's per motherboard - wouldn't require that many systems.
The huge benefit is that you can add a card and 4 drives at any point u want though.

[benn600]

Then you'd need RAID 6 so two drives could fail...but I don't know of a hacked Windows XP RAID 6 solution.  I'll have to find one, though.

Actually, I could use RAID 6 on Linux.  What is the probability that three drives would fail simultaneously?!  We can't always worry about what could happen.  There's more to life.  Take risks, hope everything flies.  In the event of an issue, start blaming everyone else: why didn't you guys warn me about running 32 drives in RAID 0?!  I thought I was safe.  Doesn't RAID mean redundant?!  You never told me RAID 0 doesn't offer redundancy.  Thanks a lot for all your help but I just lost 16 TB of Java code written by thousands of employees.  All that work is now down the drain.

Why didn't you warn me!?  Everyone seems to think it's fun and cool to take risks.  You guys think it's all fun and games and that there are no consequences to actions.  Thanks for NOT warning me.

[johnnyboy]

warn warn warn warn warn - watch out dude - there's a bomb under your desktop!!! (oops, did I say bomb, CIA are probably on their way to get me now!) lol.

Anyway - biggest thing I'd be wondering about if I was you doing it how your doing it is how do you plan to deal with adding drives each time? Are you seriously planning to just copy all the data off, add some drives then rebuild the whole thing then copy the data back again?

Regardless of all the safety issues with one big Raid 5 drive - this would be my most base practical worry with your current setup.

[benn600]

I've got it.  I'll need some funding to complete the undertaking, though.  The way I see it, I can keep my existing setup and build another array with 15 drives.  So I'll need a Paypal account where everyone interested in a good story can deposit donations.  If we can find around 30 people, that will only be about $80 a person.  Your reward will be a heck of a read as I journey through the world of free, user-supported, hardware.

I would really like to pursue this as I have always wanted 7.5 TB of storage space.

With this new idea, I won't have to rent another array, I'll just have two large arrays.  But wait!  What if I combined the 15 new drives with the 10 I already have?  Then I'd have 12 TB!

If you've seen the, I think it was South Park (may have been Simpsons), episode, you'll recall that time when they are about to lose the city to the casino people.  They then make a bet to win enough money to buy the city back.  They win the bet and are really happy!  They just won that critical bet and won enough money to keep the city!  But then someone realizes if they won again, they'd have millions more than they needed!  That's how I am acting here.  I would be SOO lucky with 7.5 TB but then I realize how much BETTER it would be to have 12 TB that I go insane.

Yes, insanity can arise from lack of disk space.  It is a critical dilemma.  Donate today for this wonderful cause!



 
dang, I already gave away the reward.  (what was the reward again)

[johnnyboy]

lol - glad to see you've finally cracked under the pressure lol.
You already have your multi Terabyte array - the donations can go towards mine - need some cash to buy a cheap box off ebay and then a bunch of drives to go in it!

I definitely think your better off splitting your array into a few different ones - easier to manage, better security and probably better performance too.

[newsposter]

Another thing to consider with lots of spindles is that your MTBF get DIVIDED by the number of drives you have running.

The more drives, the more likely that you will have multiple failures.

[benn600]

I completely understand the risks.  With each addition of a drive, the failure rate more than doubles--especially in RAID 0 when either drive failing means that, essentially, both drives failed.

[runemail]

This case holds up to 30 drives, and looks good doing it:

http://www.3dgameman.com/content/view/4708/48/

[robydago]

as far as i'm concerned I always have double the capacity I need: that way you can perform regular backups.

it's not a matter of a drive or two failing:
you have to consider human errors: even with RAID systems if you mistakenly delete a file you can still loose valuable data (and undelete works only if you perform it asap).

if you want 4TB, very very soon you'll be albe to build up that amount with only four 1TB disks, so I'll go like this:

- a library server MC12 PC with 1 small disk for the O.S and four internal 1TB drives, joined in a single 4TB volume courtesy of XP (no RAID)

- if you want something faster than 1Gb ethernet, to the above PC I would connect an external esata unit (3Gb if I remember correctly and no protocol conversion) with 4 discrete esata interfaces and filled with four 1TB disks joined in a single volume by XP (as above)


so the PC will have two 4TB volumes and you can backup all data from one to the other

apart from the disks themselves, the other things you need (controller and external unit) shouldn't be expensive because you won't use RAID

this setup won't give you any single disk redundancy, but if you schedule an automatic backup of all your data, you won't have any problems


with such a setup you can upgrade your system gradually: when 2TB disk will be available you won't need to replace all 1TB drives at the same time:

before upgrade:

-internal -> 1TB-1TB-1TB-1TB
-external-> 1TB-1TB-1TB-1TB

after upgrade (adding only 1TB capacity to your system)

-internal -> 1TB-1TB-1TB-2TB
-external-> 1TB-1TB-1TB-2TB

you can perform the upgrade first to your external unit: even if you have to break the volume you still have all your data in your internal volume; once the external volume has been upgraded to 5TB you can copy all your data from the 4TB internal volume to the new 5TB external volume. once the backup has finished you can start upgrading you internal volume to 5TB; once the internal volume is up and running you can copy all your data from the external to the internal one... so now you have two 5TB volumes and all your data on each of them.
btw, you can do all the above because the OS is installed on a much smaller hd that you never touch

this is only an example of an upgrade, adding "only" 1TB, if you want you can swap at the same time two or three or all 1TB disk with larger capacity ones.


ps. i don't remember if xp can handle 4TB volumes; does anybody know what's the XP limit?