INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Virtual raid (combining disks)?  (Read 15324 times)

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Virtual raid (combining disks)?
« on: September 08, 2006, 11:36:29 pm »

I am using 3 500GB drives to store all of my data.  However, it is a pain because one drive is full and the other 2 have plenty of room.  I don't want to move data between the drives because the types of data would vary and I would have television shows spread on all 3 drives...home videos on 2...etc.  I don't really want a raid solution but am open to the idea.  The drives are USB and I cannot use the Windows raid solution because of it and I don't think it would be a good idea anyway.

The best solution would be for me to buy 2 of those Lacie 2TB drives ($1000 each!) and I'd end up with more storage and with a single drive (one for backup).  If I did this, I'd have to somehow sell all my WD 500GB my book's and then shell out another $800 or so to get to them.  I'm thinking I'd like to use the My Book's a little longer until 2TB drives come down to a more reasonable price range.

What do you guys use for huge amounts of storage like 1TB or even 2TB?  I find that buying 300GB drives really gets old and you start realizing the power consumption and necessity of IDE or SATA ports and space in your case!  That's one reason I like the external drives.

One solution might be to simply make a fake hard drive that literally just combined all three drives on the fly.  Then, I could store files on various drives assuming the folders were the same--and the files would show up with no problem.  It doesn't lower redundancy either because you aren't worrying about RAID 0 which means if one drive fails, you're done.  This method, at least, you only lose that 500GB instead of all 1.5TB you'd have.  I've considered Raid 5 but again, the USB situation doesn't work well.  I also need an easy way to back the data up to my second set of 3 My Book's--  This wouldn't be tough, though, because I'd just copy directly from the original drive and forget about the fake drive.  Then, I could access my data directly or install the "software" again and recreate the virtually combined drive.

The issue then would be if you create a new folder, where would it go.  Hopefully, there would be some general rules.  If the parent folder resides on only one drive--obviously go there.  If it's on two, perhaps it could go to the less full drive.  Lastly, to specify, just go to the individual drive as in the backup scenario and create the folder there.

This would be so nice!
Logged

hit_ny

  • Citizen of the Universe
  • *****
  • Posts: 3310
  • nothing more to say...
Re: Virtual raid (combining disks)?
« Reply #1 on: September 09, 2006, 07:56:10 am »

The best solution would be for me to buy 2 of those Lacie 2TB drives ($1000 each!) and I'd end up with more storage and with a single drive (one for backup).  If I did this, I'd have to somehow sell all my WD 500GB my book's and then shell out another $800 or so to get to them.  I'm thinking I'd like to use the My Book's a little longer until 2TB drives come down to a more reasonable price range.
You do realise those Lacie drives are RAID-0 cos there are no 1TB HDs available as yet. If you lose a HD, there is no way to recover anything as they prolly use some proprietary raid inside. You only find out about it after a yr or so which just so hapens to coincide with their 1 yr waranty only.

You would be better off i think creating a RAID yourself in a tower, do away with your external USBs, if they are not reqd. for portable use. Cases i hear about nowadays can fit an incredible amt of HDs inside. HD's use less than 25 Watts a piece. Use a watt-o-meter or some way to measure current consumption instead of tools that PSU vendors provide, which are only for maximum sizing, actual use is less.
Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Virtual raid (combining disks)?
« Reply #2 on: September 09, 2006, 08:33:20 am »

Does anyone know what it takes to open up the WD My Book's?  They look like they are pretty well secured shut so you'd have to destroy the case to get the drive out.

What RAID case would you recommend?  I would like one that has no drives and is really cheap.  Ideally, it should hold 4 drives.  I have 3 now and 2 for backup so if I purchased one more, I could have two raid boxes with 1.5TB.  If I want to use RAID 0, what if I then want to add another 500GB drive and get to 2TB?  Am I then required to reformat the array?  I could probably deal with this.  I think RAID 0 will be the best because I don't want to have to lose a ton of space.
Logged

hit_ny

  • Citizen of the Universe
  • *****
  • Posts: 3310
  • nothing more to say...
Re: Virtual raid (combining disks)?
« Reply #3 on: September 09, 2006, 10:11:27 am »

RAID-0 is worse (from a backup perspective) than what you have currently or BODs (Bunch of discs).

Why limit yourself to only 4 drives, towers can handle far more.

As for which RAID to go with, thats something you will have to read up on, there are pros & cons going with hw/sw RAID. There is even the question of whether to put it on *nix vs windows. Personally i have not bothered with RAID and just mirror stuff every month, checksum test, connect another IDE drive, syncback.Done. There is the risk of losing stuff in the intervening month but i'm willing to take that risk, my server only runs 8 hrs a day tops as opposed to 24/7.

Opening up WD books, i think they have a clause in there to void any warranty. Ideally use these portable USB thingies to carry stuff between your main backup server and another place as opposed to being your main backup. Many ppl do what you have done, its so easy, not much tech know-how reqd.
Logged

johnnyboy

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 626
Re: Virtual raid (combining disks)?
« Reply #4 on: September 09, 2006, 10:54:27 am »

Ok well some advice from me.
I've had those lacie drives and they are definitely raid.
I had the big disks and they died pretty much on the warranty and Lacie refused to be at all helpful or to replace or fix them in the slightest.
I strongly advice against using them as your main drives. I'm pretty sure it's not the hard drives themselves with the issues but the USB controllers and because they're setup with Lacie's raid it means you cant just unplug them and wire them into your PC, you just loose all the data on them.

Yeah so my advice - dont use them. I work in IT with alot of hardware too so have some background on the area.

My personal advice to you from the sounds of your requirements is that what you want to do is this:

you really have no viable and real option that's a solid reliable and stable solution than to do RAID5. I saw a card go on ebay the other day for 300 and that's here. In the states you could probably get one for quite a bit less.
You could wire it into your existing machine with a RAID5 PCI card and then the hard drives attached to it but I'd recommend building a very cheap (everything on the motherboard type job) PC and set it up on that.
You can build those types of PC's (without monitor, keyboard, mouse, hard drives) for around 100 usually if you look around.

You have one of those, you can build it as big as you want for ultimate storage capacity, the fact it's raid 5 gives you automatic redundancy against hardware failure and also lets you make one big single drive as big as you want.
If you want to backup against software issues as well you could create a backup job that backs up all the files every day/night/week onto your massive raid 5 solution so you'd have a backup copy.

It would allow you to use the hard drives that are currently best value for their size so could make it a really big server storage.

This also gives you the advantage of being able to have a full time dedicated server running.
Make sure it has gigabit network along with your main pc too.

Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Virtual raid (combining disks)?
« Reply #5 on: September 09, 2006, 04:32:07 pm »

I'm glad I have some experts here to help me.  Let me clarify my needs and then we can figure out exactly what I need.

First, I must have 1.5TB minimum.  I currently have 5 WD 500GB My Book's.  They are all USB only and work great.  What I was thinking about doing was removing the drives from the enclosures (they should be SATA) and placing them in a standalone RAID box.  Otherwise, I could use a desktop system--not the best.  I could take my current desktop but it only holds 4 drives...so 3 + 1 boot drive.  I really need 4 + the boot drive.  Therefore, I'd need 5.

I can't really get rid of the My Book's because 2 of them I got for $180 (amazing price) and the others I wouldn't get what I paid on eBay.  I don't mind voiding the warranty but I am a bit concerned becaue one of the 5 I hae failed in 1 week (I think the USB went out).

Also, I will need some advice on how to open the My Book's up.

I demand and require having 2 identical sets--ideally held by one box if raid so I don't have to worry about losing one drive.  I saw a Dlink 2 SATA RAID enclosure but I need a 4 drive capacity unit.  What I may do right off the bat is 3 RAID 0 drives and if I later purchase 2 more, then I can switch to RAID 5 because I think it only requires one drive extra (which the space is lost in redundancy).

This is a tough issue and I don't want to spend a lot more money if I don't have to.  I already need t least 1 more drive--giving me 3 + 3 for backup.

If there is an enclosure with no drives and capable of holding 4 drives and using RAID 0, 1, & 5 that costs $200 or so, I would be very happy and I'd order 2 of them.
Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Virtual raid (combining disks)?
« Reply #6 on: September 09, 2006, 06:54:15 pm »

The absolute end all solution would be a very inteligent USB software raid program that would let me use my USB drives in a RAID setup.
Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Virtual raid (combining disks)?
« Reply #7 on: September 10, 2006, 12:31:08 am »

I am looking at an external SATA hard drive enclosure.  This one holds 4 drives and has a 300W power supply.  My issue is that I then have to connect 4 SATA cables to my computer (with it powered off!) to use it and each time I might need to remind Windows about the drive--who knows, though.

I wish I could find a network or USB RAID enclosure (4 bay).  Dlink has an enclosure but it only holds 2 drives and I must have 4.

1TB just won't do it for more than another week or two.  I'm gradually encoding my home DVD movies and other DVDs to backup on my storage drive--at the moment, I'm at about 750GB but I add about 20GB a day and will probably end up stopping around 1.2TB -- when I'm finished that is.  Therefore, I must have 1.5TB with room for 2TB if I add another drive.  Then if I want to add more, I have to buy 2 new enclosures ($400+), 2 new SATA cards ($100), and 2 more drives ($300+).

This storage need really drags you down.  I also keep reviewing hard drive prices every few days and it bothers me to keep seeing them go down from what I paid...I usually buy something and the problem is solved so I don't need to keep watching prices....my problem is not solved and is in some ways worse.  I almost wish I could just get rid of my 5 500GB My Book's (and receive the money obviously) so I could purchase a NAS w/drives--they are always really easy to fine.  I can't find a NAS without drives!

Is anyone interested in purchasing a 500GB My Book from me?  I'd charge $215 + shipping.  We would use eBay to make the transaction secure, though.

This is also assuming I can get new drives before shipping the My Book's because I need to move the data off of them.
Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Virtual raid (combining disks)?
« Reply #8 on: September 10, 2006, 02:42:31 am »

I may have solved my problem.

This can be documented as working from now on.  I've read people question if this would work or not.  I took the chance and voided my warranty on 2 drives and will soon void the warranty on 4 more (too tired to finish project tonight).

People were wondering if there was a way to use USB drives in a raid array (software through XP).  Users are unable to convert USB drives to dynamic disks because the option is greyed out and therefore, are unable to use them in RAID arrays.  After opening up the My Book, I installed the drive through SATA in my computer (hot-swappable is really handy!)  I then converted the disk to a dynamic disk.  After putting it back in the USB sytem, it works great and I now have a preliminary 500+500 Raid 0 drive working.

I even ran through a series of drive failures such as unplugging a drive (not while writing/reading, obviously).  The managemnet console simply says the array has failed and that a drive is missing.  Whether you've unplugged one or two, all you have to do is reattach them and then "Reactivate" the disk in the console.  It then gives you a warning and you just say OK and everything is fine!  This should work great for me, hopefully, assuming reboots don't mess up the array since USB is a little less low level than SATA is, I would think.  However, the computer doesn't reboot much anyway since it's a server.

This setup, assuming I don't find a flaw later, will save me $360 + $100 + more.  The $360 would be for two drive enclosures (4 bay w/ power supply).  The $100 would be for 2 4-port SATA cards.  The extra would be for 6 SATA cables--which could be pricey considering I would need as long as I could get.  This solution also presents a serious limit at 4 drives where I would have to buy a new enclosure and new/more SATA cards.  USB can easily be added through hubs.  This solution may have some speed benefits but they are canceled through the slower speed USB capability--however, I have NO desire to go for speed because I use a 100Mb network for everything and don't plan on upgrading to gigabit in the near future...no worries there.

I've read that you can enable RAID 5 through a litlte hack but I'm not quite ready since that would require an extra drive--but it may happen if I get another great deal from Best Buy.  I got two of these drives for around $176...the other 3 were $255 from Buy.com a while back.  I'm gonna hold out but I desperately need 1 more at the moment so I have 3 in use + 3 backup.  Good night all!
Logged

MrHaugen

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 3774
Re: Virtual raid (combining disks)?
« Reply #9 on: September 11, 2006, 02:34:23 am »

Why you have bought 5 external USB hard drives is strange to me :)
And another thing I don't get is why you don't buy another drive and use a Raid 5 array instead of using 3 frigging disk for backup  ?

IF you are able to set it up in raid 5 that is... If you are not, there are fast PCI-express Raid controllers available that would do the trick.
Logged
- I may not always believe what I'm saying

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Virtual raid (combining disks)?
« Reply #10 on: September 11, 2006, 08:16:07 am »

I've solved the problem.

If I open the My Book's up, I can connect them directly to my computer through SATA.  This allows me to convert them to dynamic disks.  Next, I return them to the cases.  They remain dynamic disks now and I can use the software raid solution in XP to drive them--all over usb.  This is the perfect solution for me because 1) I don't care about speed...2) practically unlimited expandability.  I don't have to buy another SATA card after 4 drives...I can get a cheap hub or use one laying around the house.

At the moment, I do not have enough drives to use RAID 5 but as soon as I find a good price, I'm buying 2 more...maybe 4 more.

In house, I have 3 drives as Raid 0-- 1.36TB total space (500GB drives).  My external, offsite backup, is currently only 2 drives but that is enough for now to hold everything I have--it won't be in a week or two, though.  When I find a good price (like $180 at Best Buy that time) I will add one more to my inhouse array and switch to RAID 5--amazing solution.

Then, I'll add one more to my offsite array so I have the same total space on each.  This means that 3 drives would have to fail at nearly the same time for me to lose *all* my data.  One could fail onsite and one offsite, but I'd still have my onsite which could be rebuilt!

This is such an effective, somewhat cost minded solution.  If I could have got all my drives for $180 at Best Buy, the total cost would be around $1200 for 1.5TB of very redundant, safe data space.  I'm closer to $1,500 for the drives since I paid so much for my first 3 ($255).
Logged

runemail

  • Regular Member
  • World Citizen
  • ***
  • Posts: 205
  • Soft Rock
Re: Virtual raid (combining disks)?
« Reply #11 on: September 11, 2006, 09:36:41 am »

I have a couple of 500gb lacie usb drives. They hold 2 ata drives with a hw jbod controller. I can replace the drives and/or format them as two separate drives. I found that to be a flexible and good way to manage backups. I have one formated as a 500gb regular drive and 1 running two separate 320gb disks.
I never thought about making them dynamic disks and making sw-raid, its a pretty good idea for my little fileserver. I have a sw-raid thats been running for over a year 24-7 with no problems, its a nice solution if you have a backup anyway.

Im not shure about running this setup over a usb hub, in my experience they often mess things up and needs to be reconnected. At least avoid the cheap ones! What would happen to your sw raids if your usb hub fails during write/read to the disks in the array?

Im hope you keep posting your findings about your setup.

MrHaugen: Raid 5 is not a replacement for a real backup, it gives you reliability more than security.

jgreen

  • Citizen of the Universe
  • *****
  • Posts: 2419
Re: Virtual raid (combining disks)?
« Reply #12 on: September 11, 2006, 09:56:03 am »

Benn--
This is such a great hack that I hesitate to point out that you're worse off now than before you started.  Apart from the marginal increase in collective volume availability (and demonstrating your ingenuity), all you've really done is combine the worst aspects of RAID with the worst aspects of USB:  All the hardware vunerabilities or RAID (and then some) moving at the speed of USB. 

As for the simplicity of having a collective volume, that's what MC is for.  Did you forget that?  With MC you can order your media by type even though it's comingled on the drive.  And JBOD (with backup)  is far more secure than RAID-anything. 

However, you did prove out a pretty cool homebrew. 
Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Virtual raid (combining disks)?
« Reply #13 on: September 11, 2006, 10:42:17 pm »

Well, I don't want to say you're wrong because it might bite me later.  However, I am running two separate RAID disks on USB with no problems.  If I wanted to throw another $40 a drive into the project, I could have got firewire drives which are natively supported as dynamic disks (with a registry hack).

I cannot say how cool it is to map my network drive (or go to the server itself) and open up the properties and see 1.36TB capacity.  This is officially the largest drive I have ever used.  I'm running a little dangerous now because I'm still waiting on my next 2 drives.  Then, imagine this!  I have my 2 separate arrays which have 100% of my data.  To add a drive to my main array (on server), I have to delete the array, create a new one (Raid 5), and then re-copy around 900GB from the backup array to the main array....takes about 24 hours.  Then, I have to delete the backup array, add the drive (3 on RAID 0), and copy the data again.  I will do the RAID 5 one first because that provides more reliability in case it would fail in the 24 hours I am copying the data again.  No doubt that the 24 hour copy period is a scary time because I'm really limiting my redundancy down to one drive failure = no data.  When I am in the failure-tolerant day to day mode, I can handle any 2 drive failures and still have everything.  A 3rd failure might lose everything, depending on which drive it is.  The best solution would be to run RAID 5 on my backup too, but that would mean another drive purchase.

I don't like having any time at all where I have only 1 copy of data, obviously, and especially when 1 drive failure means I have nothing!  I am also thinking in my day to day operations and updating the backup array, I may run the drives from a UPS or my powerpack using my laptop wireless so I am completely separated from the power grid so a lightning strike couldn't destroy all 7 drives at once.  This is a good solution because it just means a little bit extra work and it really protects me in the event of power surges which could literally take everything in a second because in order to update my backup, every single drive in question must be plugged in and running.  Then there's the issue of if a drive in the RAID 5 array fails, I should have another on hand and ready to go.

It really sums it up good to say that when there is a failure but you're safe, you should remember that you are not safe at that moment until everything is back to normal.
Logged

johnnyboy

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 626
Re: Virtual raid (combining disks)?
« Reply #14 on: September 12, 2006, 12:41:16 am »

Ugh, all sounds way too complex to me and like it's relying too much on me to keep ontop of it all or it could all fall to bits.

Easiest and therefore safest way is just to build a seperate box, shove all the drives in it and have it as a fileserver. Give it read only access so your machine cant do any damage to any of the files over the file share. To write to the drive you have to create a new mapping using a different  User/Pass which means a virus couldn't create this and nor could an OS failure.
Throw an LTO tape drive in it and backup all files once a week using 3 tapes on a rotating cycle.

Worst case scenario and a drive packs up, you throw in a new drive, put in the backup tape. Go to sleep and wake up the next day with all files restored.

The file server will be on your internal network behind your firewall with no access to it other than via a machine behind your network.
Even then Remote Desktop would be required to access this machine to actually change the contents of the machine.

THAT would give you a rock solid solution, minimal work to maintain or look after and very safe.

You might be loving USB and all your drives right now as I did with my drive to begin with, but wait till down the line in a few years when the USB controller packs in (yes mine was Lacie too so same controller) and you'll be cursing it and wishing you'd used normal hard drives :)

USB drives will never be a rock solid stable solution, they will always just be a conveniant solution due to their portability!
Logged

runemail

  • Regular Member
  • World Citizen
  • ***
  • Posts: 205
  • Soft Rock
Re: Virtual raid (combining disks)?
« Reply #15 on: September 12, 2006, 05:40:23 am »

Why is JBOD better than Raid over usb when you have a backup anyway?
I fail to see why JBOD is more secure. It might be easyer to add drives, and I don't see really see the point in running raid instead of JBOD.


If a JBOD or raid array fails you still loose your data and need the backup.

johnnyboy:
Last i checked a tape bakupsolution was kind of expensive for backing up several tb.
And whats up with this automatic bakup every day/week and automatic restore? how often does a hd fail? every 5 years? I don't really see the point for personal use.

If the controller fails in my usb drives, I can simply take out the drives an plug them into another ide controller. So im not to conserned about that. Or did it corrupt the data on the disks?

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Virtual raid (combining disks)?
« Reply #16 on: September 13, 2006, 03:09:10 pm »

Precisely what I'm banking on!  You took the words out of my mouth.  As I described, I am REQUIRED to open the drives up and hook them up via SATA to convert them to dynamic disks so I can use them as RAID.  So, if the USB controller fails, I hook it up temporarily via SATA, get the data, and then figure out where to go next (new drive or enclosure?)

I'm not going to invest in a tape backup system!  They cost hundreds of dollars!  I need 1.5TB total capacity--am going to be around 1TB when I'm finished ripping my CDs and DVDs.  Then, I'm going to start scanning in thousands of family photographs from many years ago.  That will start eating up the space again and yet again, I may be buying another drive to upgrade to 2TB.  I'm going to receive my two new drives tomorrow so I'll be able to add them to my arrays--upgrading me to RAID 5 on my server which is very good!

Here is an interesting question.  Let me describe how my setup will be in a few days then ask the question.

3 drives + 1 parity = 1.5TB  SERVER
3 drives = 1.5TB    BACKUP

if I added another drive to the backup, upgrading to RAID 5 instead of RAID 0, would there be any way to make them mirror drives?  In other words, say I had 4 drives on each--RAID 5.  Then say:
Drives A & B failed on array 1
Drives B & C failed on array 2

This is obviously SO unlikely to have 4 drives fail, but I'm curious.  How would you have to update the backup array to make it so each drive is mirrored?  If this was true, you could take drive A from array 2 and use it in array 1 to bring it back to life...
Logged

runemail

  • Regular Member
  • World Citizen
  • ***
  • Posts: 205
  • Soft Rock
Re: Virtual raid (combining disks)?
« Reply #17 on: September 14, 2006, 09:45:42 am »

Quote
This is obviously SO unlikely to have 4 drives fail, but I'm curious.  How would you have to update the backup array to make it so each drive is mirrored?  If this was true, you could take drive A from array 2 and use it in array 1 to bring it back to life...

You could try and switch one drive, maybe both arrays will be destroyed but hey, thats life...  ;)

Sounds like a risky operation to me.
The backup is "offsite" and not running when not needed I presume?

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Virtual raid (combining disks)?
« Reply #18 on: September 14, 2006, 06:13:50 pm »

My question is not understood and I really don't care about the question so forget it.

The plan is to have the backup offsite, yes.  I, at the moment, just keep it in my car trunk which is in a garage detached from the house--but still close to the house.  I am working to find a location to store the drives offsite but our bank safety deposit boxes (the size I need) are all taken.  I need one large enough to store 3 My Book's and a 200 CD/DVD Case Logic cases which I keep backup DVDs of our home videos (have a set at home, too) and this video is also ripped to the hard drives (will be in a few weeks when I'm finished).
Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Virtual raid (combining disks)?
« Reply #19 on: September 15, 2006, 12:33:26 am »

After following the tutorials spread around the web, I updated XP to allow RAID-5 arrays.  I then took my 4 drives and started the process.  After doing some research, I learned that the "regenerating" process is very time consuming and quite a pain.  In an attempt to push my length of time before I need to re-do the RAID 5 array once more, I decided to move both new hard drives I just got today to the RAID-5 array, meaning that my backup array is still limited to 2 drives and therefore, only 1TB, which is not much larger than my total data space.  I decided that this array is much quicker to update and it does not need to be the same total space--just large enough to hold the used space on the main array.

The new array is currently working on the rebuilding stage.  I'm guessing this is what it does to actually get the array ready to be used.  It will take quite a while and I will need to let it run overnight--hopefully I can get the data copying to the new array tomorrow morning.

I'm very excited with this decision to use RAID-5.  It really seems like a good move to ensure I won't lose data--outside of other types of failures which I need to watch out for, such as viruses and other bad things.
Logged

newsposter

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 789
Re: Virtual raid (combining disks)?
« Reply #20 on: September 15, 2006, 09:37:11 am »

Before you start stacking up USB drives, do some research on the USB data protocols and data integrity.

Those issues are the main reason why 'pros' continue to use Firewire instead of USB2.

On paper, the raw speeds of USB and 1394 appear to be close.  1394 does nearly all of it's device and transaction (r/w activity), arbitration and error recovery in hardware (the interface chipsets) while USB tries to take care of those functions in software (the OS and drivers).  Usually not a good thing.

Imagine the data integrity problems you'd have if PATA or SATA chipsets were stripped of everything except their electrical interface functions and all else was handled in the host OS.  That's the diff between 1394 and USB. One was designed from the beginning for disk interconnects and data integrity.  The other was designed for keyboards and game controllers.  You pick which one you want to trust your data to.

Look at the writeups of the various disk interfaces at www.granitedigital.com before you commit to any kind of USB-based 'raid' array setup.

And regarding the XP raid-5 hack.  You have to remember that it is likely that you will 'lose' that hack (and all access to your drives and data) if MS ever updates the relavant files via MSUpdate, etc, etc.  This has already happened about 5 times in the past year.

You are a lot better off building up a 'real' raid server that doesn't depend on unsupported / unsupportable / unstable OS hacks and connection protocols that aren't really up to the work you expect from them.  The end cost is the same or less than what you've already spent and you'll have a more reliable storage farm.
Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Virtual raid (combining disks)?
« Reply #21 on: September 15, 2006, 02:42:22 pm »

I've solved every problem you listed and have no worries.  First, if the USB interface fails, at the very worst case scenario, I have my second raid backup.  If both fail at the same time (one won't be plugged in most of the time) then you'll be right.  Second, I would most likely be able to pull the drive and hook it up via SATA in an emergency situation.  Therefore, I seem to be OKAY.  I really like the fact that I'm saving $400+ on SATA cards, cables, and enclosures.  All three of which would limit me to 4 drives unless I spend $600!  Plus, I decided to update my RAID-5 to 5 drives because it looks like it will take 30 hours to "Regenerate" it.  After creating the array, it goes to that stage and is giving me a percentage.  I can write to the drive but it is VERY slow--8% of my network connection.  I usually get 80% of my network connection.  I'm going to wait until tomorrow morning and then start the transfer of 900GB.

Does anyone know why the RAID-5 regeneration is even required when the array is empty and freshly formatted?
Logged

newsposter

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 789
Re: Virtual raid (combining disks)?
« Reply #22 on: September 16, 2006, 10:38:38 am »

ah, you haven't 'solved' the problem of MS obsoleting your XP raid hack with file updates.
Logged

jgreen

  • Citizen of the Universe
  • *****
  • Posts: 2419
Re: Virtual raid (combining disks)?
« Reply #23 on: September 18, 2006, 01:11:04 pm »

I'm gettng a little worried about benn600.  It's been over 24 hours since he's posted, and I'm afraid that means he's lost all his data.  Benn?  Benn?
Logged

mashedpotato

  • Member
  • *
  • Posts: 2
Re: Virtual raid (combining disks)?
« Reply #24 on: September 19, 2006, 04:31:32 pm »

We appreciate your journey, Benn600.

I would particularly like to know how you managed to open the WD enclosure. That thing is killing me!
Logged

johnnyboy

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 626
Re: Virtual raid (combining disks)?
« Reply #25 on: September 19, 2006, 08:47:27 pm »

lol.
This has got to be the most complex thing I've followed in a while ;)
For someone wanting that much space and spending that much and wanting that rock solid a backup you wouldn't have thought the outlay for a tape backup drive or a stand alone seperate file server would have been an issue.

http://cgi.ebay.com.au/ws/eBayISAPI.dll?ViewItem&item=8755748193&rd=1&sspagename=STRK%3AMEWA%3AIT&rd=1

For instance supports up to 8 drives.
Shove that in a very cheap basic case and you're talking about no more than an outlay of $200 pretty much.
Fair enough the way you're looking at it is money spent that isn't buying you hard drive space but you buy that, shove in the drives on RAID5 and you're off. Hardware wise you're sorted, software wise the files could get deleted so make it a linux file server running SAMBA with read only access and boom that's solved too. We do this at work at enterprise lvl and we consider it solid enough a solution that will last rock solidly for 3+ years.

Think about the added use you could get out of a stand alone server as well and it makes it highly worth while.


Basically, what you've put together is a real hack job type solution from what I can read and while it may work, your giving yourself 10x more headache and hassle than just doing the job properly.
Not only can the USB controller go, but it can mess up the actual drive so it's unreadable quite easily.
I've had this happen personally more than twice and that's with light infrequent use.


Anyway, all that said, thanks for the update and please keep us informed.

Very curious how this whole thing is going to pan out for you.

If I get time I might spec/cost up a "proper" solution and show you what it'd cost to do it "properly".
Logged

mashedpotato

  • Member
  • *
  • Posts: 2
Re: Virtual raid (combining disks)?
« Reply #26 on: September 23, 2006, 02:40:58 am »

Just a bump to beg Benn600 for good instructions (and maybe even a pic) on how to open the WD My Book. Would really appreciated it!
Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Virtual raid (combining disks)?
« Reply #27 on: September 23, 2006, 09:50:01 pm »

Lol--sorry.  I'm in college and am quite busy with homework, not to mention my usual computer work fix.

I have opened 7 Western Digital My Book's so far.  Please be sure to at least test it somewhat because supposedly, you violate the warranty the second you open it.  I looked all over the net for some info on opening the My Book's, but didn't find anything...so I'll now post my steps.

1. You must remove the screw in one of the 8 indentations on the sides.  Just look around and figure out which one it is--should be obvious.  One will be much softer than the other.  You can either spend some time cleaning the entire sticker out or you can just put a medium (not too small) sized philips screwdriver down in there and, while pushing strongly, just start turning.  The screw will begin to lift out and break through cleaner than the "trying to remove the sticker method" which I did on my first drive--that just made a bigger mess.
2. Now you need to be prepared to devote some time to the project.  At this point, the warranty should be violated, so you want to be extra careful to not drop the drive (as I have done twice).  I used a butter knife because a screwdriver is much too strong and sharp.  You really want a putty knife or something that doesn't have a lot of sharp edges that might force defects on the case.  Start on the side containing the open plugs and move around.  Once I got each of the top spots pried open, I stuck a pen in there to hold it open because while opening the other 3 spots, it can snap back in.  Remember that the lip goes under the drive so be sure to pry the correct way--just think about it and pry in the direction it should release.  It's hard to explain exactly the direction.
3. Once all 4 areas are released, you need to apply quite a bit of force to pull the outer case off.  By this time, I usually ended up breaking a tab or two on the back side but it doesn't matter because the top and bottom hold it quite well when snapped back together.  Careful pulling the outer case off because it has tabs as well, so gently pull and tweak the case to get the tabs (near the front of the drive) to release.
4. To get the drive out, I find that you just have to remove the four bottom screws (in a square), one and only one near the power button (the one holding the metal piece attached to the drive), and then I think there were 4 more--all in all, 10 screws had to be removed for each drive.  That is the bare minimum.  This allows you to pull the drive (still in its metal holder--remove a few more to release that) and then you can unplug the cables.
5. I then hooked the drive up to my SATA port on my computer and was able to convert it to a dynamic disk quite easily.
6. Replace the screws and close the drive up, adding the final screw once more which should cover the rought edges of the sticker, only leaving the silver color of the screw.

Unfortunately, I don't have a drive open so I can't give really accurate detail nor can I film the process--I'm not opening them up just for fun.

My Raid5 array has 5 drives= 2.0TB but my backup drive, Raid0, has only 2 drives= 1.0TB.  I'm basically to about 2GB free.  Therefore, the second I can get these drives for $200 or less, I'm going to pick one up, add it to my backup array, and recreate the backup.  Copying 1TB of data over a 100Mb network takes quite a while--I'm thinking around 24 hours!

Also, please don't misunderstand me.  I TOTALLY understand and know how to do RAID5 right.  I was ready to click the BUY button on 2 4 drive enclosures + 2 4 port SATA cards + 8 long SATA cables.  That would have cost me around $500.  I am saving that much money and am QUITE happy because this means that I can fit my backup array in a bank safety deposit box--a RAID enclosure is frickin' huge and would never fit in the size boxes we are offered--unless I paid a lot more (maybe).

Both arrays are running great.  My 5 drive array in our basement seemed to get pretty hot so I bought a notebook cooler w/ 3 fans for about $18.  It seems like somewhat of a piece of junk!  It doesn't have much flow in it at all.  Does anyone know how to convert a STANDARD PC fan to a USB fan?  I have an extra fan (the larger one) which has the standard 4 pin connector for inside a PC.  Can I easily convert that to a USB power source?  Another option would be to run a cable from the inside of the PC to the fan which would be easier, but a hole would have to be open in my server PC.
Logged
Pages: [1]   Go Up