INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Raid5 -- part 2  (Read 5757 times)

robydago

  • Citizen of the Universe
  • *****
  • Posts: 518
Raid5 -- part 2
« on: May 28, 2007, 07:14:43 am »

benn600,
you locked your thread on RAID5.
I only wanted to add a note to my post: when I suggested to you to go for a cheaper solution and not implementing RAID, is because RAID is not really meant to protect data in all situations! It will only safeguard your data against hard disks failures, but the MTBF of hard drives is very very long these days... If I were you I'd invest my money to get a backup solution because that's the only way to protect you against human or software errors that are much more likely to occur.
That's why business\corporate users use both: RAID, basically to reduce risks of downtime, and Backups to reduce risks of loosing data.
In my home system I always have double the capacity I want; for example now I have 800GB of space, but to have that space usable plus its backup I bought four 400GB disks (IMHO tapes are not that good for home backups); To have a backup without spending a fortune I have to save money somewhere and that's why I don't use RAID (basically RAID controllers are too expensive)

Ciao
Logged

LonWar

  • Citizen of the Universe
  • *****
  • Posts: 2874
Raid5 -- part 2
« Reply #1 on: May 28, 2007, 07:55:49 am »

Benn, Did you install the Foxy plug for IE like I said in this post?
http://yabb.jriver.com/interact/index.php?topic=37549.0

Afte I installed this, I no longer get any runtime errors or FF crashes... Whatever they did with the IE plugin, fixed FF.
Logged
-

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Raid5 -- part 2
« Reply #2 on: May 28, 2007, 09:38:52 am »

I didn't believe/understand what you were saying.  I guess I will try it to see if it makes any difference.

lol...back to the RAID issue.  It ain't too easy to backup 7 TB.  Besides, I have over 1 TB of hard drive backup space.  Remember that most of the data I store is simply personal DVD rips--giving me convenience to access my DVDs at any moment.  Therefore, I don't need a backup.

The other big portion of my data is FLAC.  Same situation here--not quite as simple, though.  I theoretically could re-rip everything but considering all the tagging and ratings I've put time into, I make sure to back this up.  I am using RAID because I want one huge drive.  And so far I haven't made any stupid mistakes with accidentally deleting data.  And the server is battery backed up which helps keep the data clean and the MFT safer, right?
Logged

robydago

  • Citizen of the Universe
  • *****
  • Posts: 518
Raid5 -- part 2
« Reply #3 on: May 28, 2007, 09:50:19 am »

Benn,
we all have different phobias: mine is that I will lose my data for a software or human error.... or for a natural disaster or theft: my secondary hardisks are inside a different pc which is located in another room\floor of my house...

At least for the capacity I'm dealing with right now, I can get a single data volume using only XP built in disk manager. As for DVD\CD rips etc., my plan is to get rid of all physical media so I will always need a full backup.
Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Raid5 -- part 2
« Reply #4 on: May 28, 2007, 04:34:36 pm »

Yes but I cannot afford double 7 TB of hard drive space.  This 7 TB server is costing me right around $5,000.  I cannot afford to build a second for a backup.  I have about 1 TB of backup space for all the stuff I really need.

By the way, having your backups in the same house is alone a big mistake.  It is best to store them in another house, ideally at least a few miles away.
Logged

robydago

  • Citizen of the Universe
  • *****
  • Posts: 518
Raid5 -- part 2
« Reply #5 on: May 28, 2007, 04:40:25 pm »

benn,
that's why I suggested to you to spend as little as possible to get one 7TB server, so that you could afford two of them...8)

i know an off site storage policy would be much better, but putting my backup pc somewhere else would also mean backing up my data via the internet... way to sloooow right now, but when super fast internet connections will be available all I'll have to do is to move my backup PC somewhere (if I can find a friend willing to house my PC for free...)
Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Raid5 -- part 2
« Reply #6 on: May 28, 2007, 06:11:05 pm »

lol.  I am pretty much spending as little as possible.  $2,000 right off the bat for the hard drives (16 RAID 6).  Then add a basic computer $200.  Then you've got to get a good case--mine is $700.  Then the controller card for the RAID $700.  You're pretty close to $4,000.
Logged

robydago

  • Citizen of the Universe
  • *****
  • Posts: 518
Re: Raid5 -- part 2
« Reply #7 on: May 28, 2007, 06:32:39 pm »

benn,
my idea was that if you use less drives (using 1TB models) you could, I think, save some money because:
- using a good but kind of "standard" $250 case, you save $450
- not using any additional controller and using only the one provided with the motherboard: you save $550 (i'm keeping $150 so that you could buy a better 8 sata ports motherboard)
 
So you can save approx, $1000 (but to be more accurate you have to calculate how much more or less seven 1TB drive cost instead of sixsteen 500GB ones...)

Anyway if a 7TB RAID system will cost you $4000, to have it non raid but 7TBx2 you should strip the basic system down to $2000...
Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Raid5 -- part 2
« Reply #8 on: May 28, 2007, 11:24:10 pm »

Yep, that's a moderately good idea.  However, I am upgrading FROM the model you described.  I started with a software RAID NAS that ran from a simple case.  I'm ready to upgrade to a server-quality hot-swappable case.  There's no denying that the case I purchased is incredible (lol).

The reason I want a good hardware raid controller is so the OS is more separated from the RAID control and I don't have to deal with the sorta hardware raid issues.  This should really be "IT!!!"

And 1TB drives are quite costly.  Yes, it's a big issue!  What I find HILARIOUS is that when I built my previous server everyone laughed at me for making it so horribly, being held together with bubble gum.  Now that I'm trying to do it right, everyone who posted the negative comments seem to have disappeared and the people who supported my previous actions appear, complaining!

So I guess people only speak their mind if they are upset!  LOL.
Logged

johnnyboy

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 626
Re: Raid5 -- part 2
« Reply #9 on: May 29, 2007, 11:49:33 pm »

*ta da* (as if by magic he re-appears)

I think you'll find the real issue here is that you only started talking about your new system less than a week ago and in that time a few of us haven't checked in yet you've had 3000000 posts and 3032342352 thoughts and figured we were no longer interested in your exploits when in fact we are!!


First point - your case looks pretty awesome, your amount of space is insane and I'm jealous, I've been getting all the shows I like recently and my space has been disappearing rapidly so seeing where your thinking is coming from although I dont really want to fork out that kind of cash for it.

Anyway - well done on going hardware solution - excellent choice and I fully agree with your current solution and choices - expensive but a rock solid very profession and reliable way to go thats going to make your life alot easier and worry free.

Your choice of 500Gb HD's is also a good choice - 1Tb drives right now are the very limit of whats available and so are NOT good bang for buck - 500Gb is the sweet spot right now so your doing great there.

As for the needing a backup - I agree with your point - totally unnecessary, everything you have on those servers is easily (although time consumingly) replaceable (movies, music, etc). All the photos, documents etc obviously should be backed up seperately but they'll also easily fit onto a single 500Gb drive.
There's some software called 'fileback PC' that I use sometimes personally - it'll backup your files, but better still, it'll keep up to X copies of the files when backing them up so you could use it to save say up to 5 different copies of a file, document, edited pic etc just to really give you a safe, thorough backup. With your music as well - you dont need to backup all the actual files to backup all the data - just backup your MC Database and you can then use that to auto tag re-rips of your files if the worst should happen.

Realistically anyway - if your house burns down and you loose everything you own - are you really gonna care at all about your music? Re-ripping some music is gonna be the last of your worries.

If you want to give file safety as well, you could consider and probably should consider making the shares read only - all your films dont need file tagging - just database tagging, your music doesn't need tagging - just do it in the DB and say once a month/week whatever map an identical share to the files except one that has read permissions as well and apply all DB changes to the files then re-map back the read only share.

The only files that need read/write are your documents really and what you can do with those (and your MC DB) now you have a rock solid system, is setup a scheduled job on the server that creates localised copies of all those files to another folder that isn't shared every night.

With the Xeons, and Server motherboards etc in general - they're classed as enterprise level so they have better error correction built in I believe is the main difference:
http://en.wikipedia.org/wiki/Xeon    <-- there's more info about the Xeon for you though.

Not sure if you've bought it all yet but this is an awesome project for you to read up and learn all about it for with a great hands on to really give you a feel for it. If you get this solution worked out and properly learn about everything your doing, there are a LOT of well paying IT jobs out there specialising in Data storage and backup - think about someone like google, gmail, hotmail, yahoo, etc (just the obvious ones) - they all need guys who specialise in what your doing now!

Anyway yeah - I'm back on board and you have a full supporter of the way you're doing it now (ps we never laughed, just cringed!! lol)
Logged

jgreen

  • Citizen of the Universe
  • *****
  • Posts: 2419
Re: Raid5 -- part 2
« Reply #10 on: May 30, 2007, 12:04:15 am »

Well, i'm the source of the "bubble gum" quote, but i meant that as a testament to your ingenuity.  While I agree with johnny, especially all the cool gear you're accumulating, I think roby's point is more to the target:  you really don't need a RAID or a contiguous volume, all you need is a JBOD.  The costs go down considerably at that point.  As for backup, if you really don't mind re-ripping your DVD's, then you've already got a backup, subject to the media not disintegrating.

Most of all, have fun!  School (and life) will get stressful, but A/V ought to be a break from that.   
Logged

johnnyboy

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 626
Re: Raid5 -- part 2
« Reply #11 on: May 30, 2007, 12:20:31 am »

JBOD has been suggested before but his main goal is he only wants one single drive (share too I guess).

This new solution however 'should' let him now consider his storage as physically secure with deletion etc (ie software/file system) issues now being the only remaining problem. Getting the server machine to locally create a backup of the files that is unaccessible over the share should fix all deletion type risks.

I think trying to have an additional backup at this point of all these ripped disks is just pointless - as with all computers and computer systems there has to be the point where you as an individual draw the line and say thats secure 'enough' and I am now going to get on with life and not worry about it any more.

That 'enough' point is i think what you just learn with experience to computers as you become more familiar hands on with what 'can' happen and what does in practise tend to happen in 'normal' terms.

You have the physical disks and thats your backup if all else fails which in itself isn't that likely at this point.
Logged

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14463
  • I won! I won!
Re: Raid5 -- part 2
« Reply #12 on: May 30, 2007, 07:03:23 am »

I gave up with M'Board based RAID-5 for two reasons - 1 it was always rebuilding, but more importantly it not not support RAID Expansion (could not grow the array by adding additional drives).  I then looked at adding a "real" raid card but the cost drove me to having two sets of disks in two differrent PC that back up to each other (each just using Spanning under Windows).  I've only got around a couple of TB but the benefit I see of having two PC's each using spanning is:
1) Real Back Up
2) No Raid-5 Overhead
3) Can use different sized disks yet one logical drive
4) Cheap
5) Expandable

Given that - what I think I'd really like is a well priced enclosure that would take say 8 Drives, was quite, JBOD support that I could plug into the PC using E-SATA....
Logged
JRiver CEO Elect

robydago

  • Citizen of the Universe
  • *****
  • Posts: 518
Re: Raid5 -- part 2
« Reply #13 on: May 30, 2007, 07:23:35 am »

jmone,
that's my opinion and solution as well.
But to save space and reduce noise, I think I won't ever use more than 4 disks. I think that my next upgrade will be two sets of 3 data disks 1.5TB capacity (total space 3 TB) plus two small fast system disks.
The benefit of the secondary\backup pc is also that it can serve as media server to all media player PCs\devices in the house, while I can use my primary PC undisturbed.
Logged

johnnyboy

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 626
Re: Raid5 -- part 2
« Reply #14 on: May 30, 2007, 08:49:55 am »

are you saying that you have basically two computers which mirror the data between each other?
Logged

robydago

  • Citizen of the Universe
  • *****
  • Posts: 518
Re: Raid5 -- part 2
« Reply #15 on: May 30, 2007, 02:25:32 pm »

are you saying that you have basically two computers which mirror the data between each other?

yep,
I have two computers, but one of them is read-only: it's both the backup destination and read-only media server
Logged

johnnyboy

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 626
Re: Raid5 -- part 2
« Reply #16 on: May 30, 2007, 03:13:08 pm »

To have a complete mirror for all your data (when your hitting that kind of total size) is a pretty big resource waste IMHO.

As a corporate company we would never do it, only non-replaceable data needs that kind of backup.
Logged

robydago

  • Citizen of the Universe
  • *****
  • Posts: 518
Re: Raid5 -- part 2
« Reply #17 on: May 30, 2007, 03:22:16 pm »


as stated before, I plan to not have any other source of all my data than the copy I have on my harddrives (I want to get rid of CDs\DVDs).

so, as far as i'm concerned, ALL data is non-replaceable; even if in theory I could try to recover most of the data, the time it will take make having a complete backup well worth it.

but in the end it's all up to how much you value your own free time...



Logged

johnnyboy

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 626
Re: Raid5 -- part 2
« Reply #18 on: May 30, 2007, 03:30:27 pm »

Wanna give me all your CD's/DVD's seeing as you want to chuck them all? ;) lol.

Just make sure you have all your receipts then in case the RIAA or anyone ever comes knocking on your door, without any original media it'd be hard to prove you own it all.
Logged

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14463
  • I won! I won!
Re: Raid5 -- part 2
« Reply #19 on: May 30, 2007, 03:37:30 pm »

To have a complete mirror for all your data (when your hitting that kind of total size) is a pretty big resource waste IMHO.

As a corporate company we would never do it, only non-replaceable data needs that kind of backup.

Yeah that was what I Originally thought, right up till I had to re-rip all my Audio CD's......I then decided that at under $200 per 500GB Drive seemed sort of Cheap! (I note however your on a different scale for volume).  The PC I back my HTPC Media Up to is my the Households Main PC - It is also good in that I can play around with Views, Meta data and other testing on this PC without stuffing up my HTPC, then just copy the relevant files back....
Thanks
Nathan
Logged
JRiver CEO Elect

robydago

  • Citizen of the Universe
  • *****
  • Posts: 518
Re: Raid5 -- part 2
« Reply #20 on: May 30, 2007, 03:44:56 pm »

if you buy two multi TBs systems. it's not going to be cheap.
but if you have a lot of TBs of data, it'll take you a loooong time re-ripping if you loose your data.

Logged

johnnyboy

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 626
Re: Raid5 -- part 2
« Reply #21 on: May 30, 2007, 03:48:05 pm »

Yeah but he's mainly doing DVD's. The bang for buck value drops HUGELY.
A CD ripped as lossless = around 350mb per disc. Doing it as mp3 or something is even better.
A single DVD movie though is around 6000-8000Mb - obviously no where near the same scale of economy/effort.

500Gb drive
For music (lossless) at 350Mb per disc thats:  500,000\350 = 1430 albums per drive
For music (mp3) at 100Mb per disc thats: 500,000\100 = 5000 albums per drive

For a film at 7500Mb per disc thats: 500,000\7500 = 66 films per drive

Obvious HUGE difference there :)

Anyway, if you can really afford it then either way it makes no diff. Benn was just asking for advice and suggestions about his Raid 5 though which is how this all started so we were commenting :)
Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Raid5 -- part 2
« Reply #22 on: June 01, 2007, 08:14:48 am »

Thanks for all the info.  Sorry for the delay--big family issue.

You all have great information.  I especially liked the reminder that in the event of a fire, the last thing I'll worry about is where my music or DVDs are backed up.  In addition, using read only for the server is another great idea.  I always worried about the potential for a problem and you're right that I only need modifications rights on certain folders.  Since I'll probably use Windows, would I just want to modify the permissions on certain folders and then can I still use one top level share?

Hey, what should I call the server and/or share drive?  I like to use the computer/case manufacturer and in this case it would be Chenbro.  Otherwise, I used Server in the past.  And for the share name, I could use Book1 which is what I used originally.  This came from the fact that I started with the WD My Book drives and Book1 seemed reasonable cause I could add Book2, etc.  lol.  If I would have stuck with that system, I'd be at Book16.

I received the case but that family emergency led to another opportunity which takes my time and I haven't had a chance to look at it.  Such a nice, HUGE, box just waiting to be looked at.  I am going crazy to look at it.  I have the Newegg order ready to go and will place it soon.

Overall, this identical box/server could be built by anyone for under $3,000 using some spare parts.  Isn't that amazing?!  That's CHEAP!  Lots of 1 TB readymade solutions (which have slow processors) are over $1,000!  It's amazing these drives are down to ~$115 at Newegg.  I remember paying $100 for 20GB--yes then I've heard about 5MB for $500.  I also agree with not using 1TB drives.  Remember when I started, I bought ten drives.  I am literally stuck with those drives--at 500GB--or I'd have to buy 10 more, different sized drives.

The RAID card I'm considering supports adding space, etc., but I am maxing the array out...both the case capacity and the RAID card capacity.  Great ideas and I can't wait to continue this project.
Logged

johnnyboy

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 626
Re: Raid5 -- part 2
« Reply #23 on: June 01, 2007, 04:51:03 pm »

Good luck with the family issues - they always come first.

I sometimes call my boxes things like 'server' but its a bit boring.

At work we have the following setup for servers:

In London they're named after tube stations: waterloo, aldgate, bank, etc
NY - boring corporate logical type names
Australia: Dingo
Toronto after greek gods,

And so it goes on - I'd say come up with something creative and fun thats also easy.

Box name: sunFun
Share name: media

Easy enough and logical :)

As for the shares - you can just change the user permissions on the share and then have two different users setup that you use to connect with.

The default one everyone uses is read only.
The one to save changes to DB is read/write/execute

And then yeah - you basically change the permission level on a per folder basis so that some are only read.

Between the two users then you'd get the full control you want and you already have hardware safety, this then gives you software safety.

For the nuke/fire/WW17 safety I recommend grabbing family, giving the server a passing kick as you run past it and heading for the hills!!
Logged
Pages: [1]   Go Up