Topic: Testing hard drive speeds (Read 5339 times)

benn600 · « **on:** March 09, 2007, 11:26:47 pm »

Naturally, with a 10-drive RAID 5 array, I absolutely want to test its performance! Does anyone know of a "good" and free utility that will simply test drive speeds for writing, reading, etc at various file sizes, etc? I had a nice utility at one point and, saving everything I ever install, somehow managed to misplace it (even with good folder structures).

I want to see how quickly data can be read from the drive even though the bottleneck would be the network. I'm mainly interested in a number for explanation purposes. I'll then compare my results to a standard HDD I have. Thanks!

hit_ny · « **Reply #1 on:** March 10, 2007, 01:43:17 pm »

HD Tune

benn600 · « **Reply #2 on:** March 10, 2007, 07:00:47 pm »

Unfortunately, I think that was one I found in my searching and it doesn't really help in this case. It appears to only show individual drives and not drive letters. That would only test one of my 10 drives on my RAID 5 array. I need a program that will test the speed of the array overall (by drive letter).

benn600 · « **Reply #3 on:** March 11, 2007, 12:04:47 am »

Theoretically speaking, alltogether I should have 160 MB of fresh, high speed cache at 16 MB * 10 drives. I'm thinking only 9 would be effective since the 10th is simply redundancy and would require calculations (thus slowing everything down) to utilize. This would mean that a huge amount of data could (theoretically) be written instantly. I'm looking for a program that randomly writes 1's and 0's so it really could test the maximum write / read speed. I wonder if 160 MB of binary could be written in a matter of nothingnesses.

newsposter · « **Reply #4 on:** March 11, 2007, 11:23:21 pm »

Ah, the grasshopper starts to see the differences between OS-hacked fake raid and hardware-controller based real raid....

benn600 · « **Reply #5 on:** March 12, 2007, 12:26:59 am »

Lol. How much would I have to pay to get 10-SATA drive RAID 5? So far, it cost me $100 in controller cards. To get RAID 5, don't all the drives have to be on the same card? For hardware RAID 5, I'm thinking you'd absolutely need a 10-port RAID 5 card, right? I don't think I've hardly seen any larger than 8-port cards. How much would it cost me for 10 drives? Plus, I don't have PCI-Express in this particular computer.

Addition: I mean so far, the array has been running amazingly well and flawlessly! A few hard reboots (mistakenly) have also had no visible affects on the array, either. It seems great so far! Remember I have backups of most of my data (the most important stuff in multiple places and types of media).

Last time I said everything was fine, I think something suddenly went wrong and I had some issues? Let's not discuss the issue too much or something will start messing up!

newsposter · « **Reply #6 on:** March 12, 2007, 03:09:54 pm »

More drives DOES NOT equal more throughput. The more drives you have, the harder the controller card and OS have to work to create stripes and calculate checksums when writing and to reassemble and error check data when reading.

Larger arrays DOES NOT equal easier managability, recoverability, backup-ability.

benn600 · « **Reply #7 on:** March 12, 2007, 05:07:47 pm »

lol. go read my Casually backup 4 TB thread and help me figure that issue out. It's a good point but we're all going to be dealing with it sooner or later. The point is that having ONE "drive" is easier for managing on the software end. I agree it sucks for backup and such but it's great to have one single area for data.

hit_ny · « **Reply #8 on:** March 13, 2007, 10:42:59 am »

Don't think i will be

see, i use a few partitions on 3HDs, as i run out of space i get bigger HDs, so i went from 120GB->250GB several months back. When that runs out of space i will throw in 500GB ones. It's timed so the $/GB is the least for a particular size of drive. Currently i think its 400GB. The older ones get tasked for offline backup.

MC is none the wiser, as all paths are identical. No need for RAID in my case at all.

Having one big drive is nice & all, but i value backups over ease of use.

benn600 · « **Reply #9 on:** March 13, 2007, 04:05:04 pm »

I must say that I looked at the prices of hard drives and at $170 for 500GB, how can you get much cheaper!? Lol. I don't think it's worth saving a few bucks to settle for smaller drives. Problem with having 10 500 GB drives and using them individually is that I will get the first 7 to 95% capacity and they will slow way down. It just creates big problems.

johnnyboy · « **Reply #10 on:** March 13, 2007, 08:28:59 pm »

To be honest, I dont see many other people ever sharing your situation.
Most people will just add drives and not try get them all setup in RAID 5. The extra hassle it causes is just not worth it to most people.

Sure it's working for you and your enjoying it but thats because your enjoying the technical challenge of it and trying to solve the countless issues it causes you.
Your constant posts here about new issues or questions about it just prove how much hard work it is.

There are probably several people here with huge amounts of storage, maybe as much as you, some possibly even with more but not a word from them. The main reason I'd imagine is that they are simply adding new drives to their systems and leaving them as stand alone drives. To add more space, they dont have to rent 4Tb of storage, they just add another drive.

The additional hassle you give yourself is all so you dont have to worry about 'multiple drives' but to be honest, its pretty much a void arguement.

You have one drive so you have:

d:\music
d:\films
d:\whatever

most people will just have

d: <-- is their music
e: <-- is their films
f: <-- is whatever

and so dont need the extra folder you are needing anyway.

You save yourself a drive letter but gain a folder. They just use the drive letter.
And as for spanning drives - thats not really that big an issue - you can either sort alphabetically or just by genre or whatever you need.

When they're all loaded into whatever media jukebox you use as well (guessing MC) then it becomes even more of a mute point because of the fact that MC doesn't care where the files are saved and just gives you a general interface for them.

The issue of managing your files that you try to overcome by having one big drive, most other people are just over coming by using Media Management software.

Right now you are right, 500Gb is the sweet spot it would appear for HD sizes (http://www.pricewatch.com/hard_drives/)
a few months ago 400Gb ones were probably the sweet spot.

As for backing it all up, I think most people will consider their physical DVD collection as their permenant backup in the event the HD goes without warning. If you have warning then you just buy a new drive and copy the files over to it. The advantage of individual drives is that if one goes you are just loosing the files on that one drive and so its not that big a deal to backup.

All images and music can usually be backed up onto a single 500Gb drive.

johnnyboy · « **Reply #11 on:** March 13, 2007, 08:32:58 pm »

Oh yeah and just to clarify, not complaining about your stream of posts here about your setup, I quite enjoy them.
Just pointing out that you have these issues whereas most people simply just plug in the extra drive and are away without a second though.

hit_ny · « **Reply #12 on:** March 14, 2007, 02:58:49 am »

$170 a drive is high, i wait till they are close to $100 (i buy in multiples of 3), when the 1TB drives come out watch the 500's drop in price. So usually i'm at least 2 steps behind the latest & greatest. That cycle is usually 3-5 yrs, which is a good time to replace all the drives anyway.

Of course i don't have DVDs on HD so my space requirements are not as high as yours.

I told you previously, putting DVDs to HD is not worth the hassle or the cost atm, simple reason is you don't switch titles like you would tracks, once a dvd is playing you're sat down for the duration, unless you like jumping titles every 5 minutes. If you watch episodes, usually there's quite a few that will fit one one DVD so you'r e sat down even longer. i also don't have an urge to watch something again if i saw it recently, unlike some catchy track, that i might want to listen countless times.

I did consider setting up a RAID-5 earlier, but then some one posted here about losing 2 HDs, guess what the whole thing was hosed. It's rather unusual to lose 2 at the same time, RAID-5 only protects against the loss of 1. That there put the RAID idea out of my mind. You alos have to make sure you have a good controller, cos if it fails, your back to square one again, as has happened to some friends.

So the point of RAID is what(?) in exchange for the pain of dealing with it.

If you lose one drive you are good.

If you backup frequently then this chance is reduced, if you replace drives on a regular cycle of 3-5 yrs, it's even less.

BODs or bunch of disks, no hassles, simple. Internal HDs as oposed to USB ones and you have the best value your money can buy.

Getting back off the tangent, try out IoZone, it may be able to do what you want.

johnnyboy · « **Reply #13 on:** March 14, 2007, 06:38:52 am »

If you look at that link I posted, you'll see 500's have dropped to $133 each which means they are better value than 400Gb drives right now.

At that price as well, it actually works out fairly reasonable to backup all your DVD's onto HD.
If you take the extra step to get best value and best setup and run them through DVD Shrink when storing them, you will further reduce them to 4.5Gb per disk. If you say each drive after its formatted works out at around 450Gb then it means you can get 100 films per disk which is pretty good value - $1.33 per film - quite a reasonable price.

If you want even better bang for buck and value for your system you can compress each to a 2Gb .avi xvid film. The quality on a 2Gb xvid will be superb (most xvid films are compressed to 0.7Gb or 1.4Gb if they want to do in high quality so a 2Gb one will be awesome quality). At 2Gb per film you would be able to get 225 films per disk which is then incredibly good value for money and definitely a point most people can afford to have all their films with instant access for.

I think now is definitely a point where its fully practical to rip all your DVD's to disk (this also saves you having to use the real disks so you dont have to worry about ruining or loosing disks so works as a great value backup for them). I dont think I would personally leave them uncompressed, I'd probably use DVD Shrink just as a very fast solution, but I dont think ripping them all is that crazy a notion

hit_ny · « **Reply #14 on:** March 14, 2007, 10:59:01 am »

I will assume you are using older drives for offline backup as opposed to the original DVDs themselves. You spent time ripping, encoding to xvid + tagging, i would not want to do that again, if as you say, you can put 200+ movies per 500GB drive.

Dunno, maybe if you like watching them over & over.

A DVD-album (at 2 per) like this (or a cpl if you insist on lossless) will hold just as many. Now if you situate it close to the DVD player, how longs' it take to pop one in eh

, these would be backups, the originals stored away in some safe place.

if i'm making a music playlist or even images, i might source files from 100s of albums and want them avaliable for easy access. Movies for some reason..hmmmm. I don't see this situation improving with HDTV around the corner.

The only instance where i find putting movies to HD makes sense is in a time shifting app like Tivo etc, good luck getting anything off it.

benn600 · « **Reply #15 on:** March 14, 2007, 05:07:31 pm »

I'll read all these responses a lil' later. I'm busy with midterms right now. I'll just say (after reading a bit) that if you have lots of separate ads, you have no redundancy. With it all on a RAID 5 array, you won't lose anything if a single drive fails. If a single drive fails (in individual drive setups), then you'll lose that drive of data.

hit_ny · « **Reply #16 on:** March 15, 2007, 02:11:49 am »

Oh sure, i'm not saying its 100% fool proof, depending on how long ago the last back up was, is what you stand to lose in the event of a losing a HD. I do mine every month so worst case is i could lose a months worth of stuff. If you reduce that to every 2 weeks then its less.

It's not happened yet, but think i could recover from a month's loss without too much trouble.

johnnyboy · « **Reply #17 on:** March 15, 2007, 07:50:05 am »

the thing about stand alone drives is no matter how many fail, you only loose the data on those drives. With your raid 5 - if one fails you are fine, but if two fail you loose everything.
Also, due to the fact the disks are all stand alone, you are only evr going to be loosing one drives worth of files - esp. with DVD's that are such big file sizes, re-ripping the films that were on that drive really isn't that huge a task and is free as compared to paying for additional HD's to back all your data up to.

benn600 · « **Reply #18 on:** March 15, 2007, 09:21:04 am »

Exactly. With the single drive approach, you basically have to back it all up in a 1-1 ratio. With my array, I get redundancy for all my data at 1-10 ratio for wasted space.

In any event, I also have the important stuff (excluding DVDs that can be re-ripped) backed up. I have a 0.7 TB server that I keep off most of the time. It is actually a bunch of 100's of GB drives using a spanned volume in XP. This lets me simply add drive space and end up with a huge drive. I know the failure rate is high but that's okay. I also have a 500 GB drive I use for backup and lots of DVDs for the MOST important stuff (irreplaceable pictures).

benn600 · « **Reply #19 on:** March 15, 2007, 09:31:22 am »

After I've watched a movie once, I like still being able to have instant access. I, unlike you guys, will often pull up a DVD randomly. I will jump to a section (with a good song) and view just that area or I might just let a movie play (from part way in).

However, I completely agree with much of the notion. Music, you are much more likely to repeatedly listen to and make complex playlists pulling from many CDs. Movies require a large commitment to view. You aren't going to just throw together a playlist of your favorite ten movies! Perhaps a playlist for a trilogy or such but even that is pointless.

Don't forget that you can't do as much work while watching a movie. When I have a movie pulled up in VLC, it slows me way down. Music doesn't do so much. I can have music on and still concentrate close to my full capacity (and sometimes more) because the music is simply background encouragement with a beat to keep me going.

Movies take a 1-3 hour period of your life and throw it in a magical world. You really lose that section of your life to the plot and story. Can you listen to a hundred songs in a day? I'd say that's over 5 hours of continuous listening but its no where near impossible. It's hard to watch three movies in a day! The time comparison is similar. 100 songs would be a challenge and three movies would be, too. That works to about 5 hours of movies or 6 hours of music.

hit_ny · « **Reply #20 on:** March 15, 2007, 11:04:13 am »

Quote from: benn600 on March 15, 2007, 09:21:04 am

With my array, I get redundancy for all my data at 1-10 ratio for wasted space.

Yes, only 1 drive fails you are fine, i'd say thats your best case.

RAID is not backup

I think you already realise this but it's not clear going from what you said above. RAID only provides operational redundancy, so if your server must stay up then you can afford to lose a disk and get back soon after. You still need some way to backup what you have if anything worse happens. I heard of controllers going bad and (unless you're experienced in this), you could lose the lot, already happened to a friend, and made it clear just what RAID is. If you know what you're doing i guess you could get away, have experience recovering from this sort of thing. Building a multi-TB server is cool & all, but it comes with a price tag, if it's worth anything to you in the first place.

I don't see any way to get around the 1:1 ratio.

You could use optical media (takes very long if ur servers's in the 100s of GBs or more and also is write once only, so lags from the current state) and tape is pricey, but is standard in business.

My worst case is i lose all HDs in one fell swoop , i hop over to the store pick up another 3 and copy over from the backups, and i'm still only a month behind.

benn600 · « **Reply #21 on:** March 17, 2007, 11:57:49 pm »

Yep, like I try to overly emphasize, I have everything that matters backed up to at least another hard drive...the MOST IMPORTANT stuff is also on optical DVDs. I also carry my hard drive in my car (so it's usually offsite) but I learned a week ago that a safety deposit box opened at our bank...so I will strongly consider getting that. Then I can put the My Book (backup drive) and some DVDs in it...and it will be 3+ miles away from our house.

I'm discussing upgrading to 15-drives in another thread in this category (Casually Backing up 4 TB). I'm guessing the conversation will heat up very quickly.

In the end, I may just end up trying to buy a huge case, using standard SATA cards, and going with Windows XP hacked RAID 5. I can move my existing hardware from my current case, too, so I don't have to buy new components.

My question is that if a drive fails in my Windows XP RAID 5, I see on screen that a drive failed...but how do I figure out which physical drive it is that failed? lol. Interesting question but I can picture it being important!

In the event that a drive failed now, I'd probably just shut the whole system down, unplug it, and casually sit until I get a chance to buy another drive. The last thing I want is for a drive to fail, though! That's $150+ that just "flew out the window."

But check that thread out...it's gonna be a screamer. Imagine this: 7 TB

Now all I have to do is say 11 TB (Randomly) and my dreams of 15-drives go out the window. We went from 2.5 GB drives when I was young to 500 GB drives. What about when we get to 500 TB drives? 2 TB drives are just now becoming "somewhat" common (in arrays)...so we're at a thousand times the space. I'm getting bored. bye.

INTERACT FORUM

Author Topic: Testing hard drive speeds (Read 5339 times)