Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[benn600]

I'm referring to the 7 TB server I built about 3 months ago.  It is using a Promise Supertrak 16350 RAID card under RAID 6 with 16 500GB drives.  Core 2 Duo processor, 2 GB memory, Abit Fata1ity motherboard, RAID 5 main drives.

My complaint is that I seem to get somewhat slower and disrupted speeds than I think I should be getting.  The speed also seems very random.  During my initial testing 3 months ago, I was getting 190 MB/second duplication rate (read and write same data again).  That seems incredible considering that's 380 MB total transfer rate per second.

However, now I'm at 61% space utilization and all the while it was building up, I've noticed issues with the speed.  The main reason I'm so confused and anxious to try to solve this issue now is because with a HTPC, the video can start stuttering quite severely if someone else starts using the server.

Basically, one DVD stream has no issues at all.  As soon as I add another file copy process from the server, the video will start stuttering every 5-20 seconds for 1-3 seconds.  If I look at the network activity, it looks like each user is given full speed for a period and then the speed drops to zero.  It should try to balance speed so everyone gets and equal amount of bandwidth at once.

But what confuses me is that on one node, I have a computer and the HTPC.  Both are sharing one 100Mb connection to the server.  I then have another desktop with a gigabit connection to the server.  With the two PCs on the 100Mb end, while the DVD is streaming I can start copying files and the video still stutter even though the PCs network usage is < 60% on average.  Then, if I leave that and add the gigabit computer, it will suck 25% of the gigabit connection (~30 MB/second) and the video will stutter about the same frequency but the PC copying over the 100 Mb connection slows way down.

It's almost like if it is asked to do two things at once, it gets really confused and gives chunks of data to each user at a time.  I'm at a loss because I'm not really sure I understand much about how all this works anyway but this is horribly annoying.  With a single drive on XP (even software RAID 5 like I had for 6 months), I get amazingly well balanced speed so I can have a handful of perfect video streams while data is being copied!

I mean if this was a business environment, lol--so once you hit two connections you lose the robust, even data transfer rate?

Where was/is my mistake?  Should I have gone with the 3ware card?  Is Vista horrible?  Unfortunately, I can't do some easy tests like I'd like under XP but it might have been better under XP (ignoring the losing array issue upon reboot issue).  Weirdly enough, my gigabit transfer percentage went from 20% to 35% with Vista.

You'd think for $700 it could handle a simple 2-user setup.  My software RAID5 system worked better in regards to consistent speed any day!  I'm not after incredible maximum speed as long as I get a consistent, maintained speed!  Random speeds confused programs and seem to introduce errors more often.

I'm going to call Promise again if I can't find anything to help me.  Now I wish I could have another identical hardware set so I could mess around with new stuff.  Who knows, maybe I should just build another system except with 750 GB drives.  I really thought this would solve all my issues once and for all!  How crazy would that idea be?!

I've tried enabling/disabling/toggling SMART, NCQ, Drive Cache, Array Cache Mode, Vista Cache Modes, etc.  Nothing makes much difference except for the Vista Cache Mode.  The second checkbox helps speeds quite a bit.

I did a quick duplication test again -- remember I was getting 190 MB/second?  This time I got 50 MB/second.  A single drive might even be able to achieve that!

It's not tough to see how frustrating this can be.

Addition: I am using the onboard RAID to create a 3-drive * 320 GB RAID 5 main OS drive.  This doesn't seem amazingly fast either!  I am usually copying to or from this drive in my tests.  Perhaps this has some severe limitations since it's just onboard RAID.  But either way, it seems like all my data access is horribly slow.

[jmone]

Is the array rebuilding in the background?  I had a RAID-5 setup (Intel Mobo / SW) and it was always rebuilding giving poor throughput.

[jmone]

Also some of the whitepapers over at http://storage.knowledgestorm.com/search/keyword/ksstorage/Raid%205%20Design%20Raid%205%20Storage/DirectSSkw/Raid%205%20Design%20Raid%205%20Storage suggest that Fragmentation is a cause of progressive degradation of speed on Raid Setups (but they are written by Defrag Vendors!)

[benn600]

There is nothing going on in the background and I defrag regularly--and Vista does it every evening on a schedule.  I'm definitely going to call Promise and see if they can figure anything out.

I used Disk Bench to run some more tests.  It was varying quite a bit but I was getting up to 130 MB/second file creation speed.  So the network access speed of up to 30 MB/sec over gigabit is no where near what I should be capable of getting based on the overall speed--and I would wonder if it's capable of more because that seems a little low.

[newsposter]

cheap cards = degraded performance

And you also have to select a raid scheme that supports streaming media.  A stripped raid set that is constantly tinkered with is not it.

In the absense of industrial strength raid cards and a supporting OS & filesystem, you are probably going to have to go with mirrored drives.  Preferablly mirrored by intelligent hardware cards, not by the OS.

And this is why I build my servers with a 6 drive max.  2x 20 Gb drives to mirror the OS on (using OS mirroring) and 4 drives in a 2x2 mirrored set (using hardware mirroring).  I can still get 1 Tb usable in a 2U server and can successfully run the storage on a 300 watt PSU in a $50 case without adding a dozen cooling fans.  The worse case I've ever seen is 3x video streams out and 1x in on top of the usual file/print/mail activity.  No herky-jerky, no pauses, dropouts, etc, etc.

Everything gets bought used on ebay except for the PSU and hard drives.

$100 for a case and PSU
$100 for an AIO mobo with 10/100/1000 and cheap video
$100 for a pair of 20G drives
$150 for a cheap AMD CPU and 1 Gb ram
$500 for 4x 500 Gb ATA133 drives (you can almost get 750 Gb drives for that price now)
$200 for a pair of 3Ware ATA133 raid cards
$30- for an additional Intel chipped 10/100/1000 card

$1180 for 1 Tb of on-line storage that works for more than 60 days at a time.  By the end of this year that $$ spend or less will put between 1.5 and 2 Tb mirrored on-line.

Other than MSFT Updates, I haven't rebooted my 4 servers more than once in the past 18 months.  And I never defrag.  Never.  I don't backup anything that I have on CD or DVD.  Once a quarter I drag out the 8MM Mammoth tape drive and copy whole filesystems to tape.  Nothing fancy, just belts and suspenders.

What kind of network switch are you running?  A lot of 'cheaper' network 'switches' are really upgraded hubs that can't handle a full hit of packets.  You need to read the tech specs on networking gear to see max data throughput and packets/sec capabilities.  Hint:  If the network gear runs from a wall-wart power supply, it's definitely crap.  If the network gear doesn't cost at least $25- per GigE port, it's probably crap.  If the casework is all whooshy-cool looking to fit into a small office desktop environment, it's probably crap.  Same thing goes for your network cards.  Other than Intel chipsets, most mobo network ports are crap and can't handle constant pushing of GigE traffic.

You might have to go with multiple net cards in your storage servers and break the house up into subnets to improve performance.  Again, cheap network switches can't do subnetting very well, if at all.  I have one subnet for inter-server comms and another for the house.  I use the mobo network ports for inter-server comms and the add-in cards for house service.  The $30- Intel chipped net cards I use also come in 2 and 4 port versions (and cost $60- and $100 respectivly) so there is plenty of room for subnet growth if necessary.

Jumbo frames.  All of your GigE stuff needs to be set to use jumbo frames.  This is one area that separates crap network switches and cards from the good ones.  If your networking stuff can't handle a full hit of jumbo frames at the same rate as standard 1500 byte frames, then it's crap.

Whatever you do, do not depend on something like a Linksys BEFSRxxx as an all in one 'solution.  Combining a NAT firewill with routing and DHCP/DNS services is way more than those little boxes can handle.  I have a BEFSR41 but I use it as a NAT firewall and DHCP server only.  The default router and network switch is a TrendNet 24 port 10/100/1000 I bought on sale/closout at the local PC shop for about $225-.

DHCP and DNS.  Everything in the house is on a static IP; printers, servers, and workstations/PCs alike.  I have a small 4 device DHCP pool set up but thats for 'transitory' devices like the occasional laptop that someone might bring by.  My router uses the Linksys box as the primary DNS host and has opendns.com servers set up as secondaries with a rapid time out.  The Linksys box uses the local comcast.net servers for DNS with opendns.com as secondaries too.

Integrated solution.  Everytime you piecemeal a system together, you are creating a pile of bottlenecks.  Tweak one thing and all you do is to move the predominate bottlneck to another location.  Spend some time on design, correct device selection, and engineering and it will work not only in 'testing' but in real-world use too.

[benn600]

The worse case I've ever seen is 3x video streams out and 1x in on top of the usual file/print/mail activity.  No herky-jerky, no pauses, dropouts, etc, etc.

3x what kind of video stream?  If I was getting what I was getting at one point, that was close to 400 MB/second.  DVDs are around 10, so that would theoretically be limited by the gigabit port.

I'll do some more testing and call Promise to see if they can give me ideas.  Everything I got was less than the prices you stated and brand new / very decent.  I got two 320 GB drives for the ~$100 you spent on 2 20-GB drives.

[benn600]

I've made a very disturbing discovery.  I went to my main computer, which is utilizing a gigabit connection to the server, after traveling through two identical ~$50 7-port switches, and opened 8 DVD streams plus two file copy sessions.  I had 7 of the 8 DVD streams muted and the 8th one's audio played flawlessly, absolutely flawlessly.  It was using right around 30% network utilization.  But like I said, all the video streams were playing flawlessly.

Unfortunately, at about 6 or 7 streams, CPU utilization jumped to 100% and the bottleneck became the processor--The key that impressed me was that the streams all played without any hickups so they were being provided a very steady data stream.  Plus, I was using VLC which doesn't seem to have the best buffering abilities and it stutters much more than MC.

So then, the million dollar question.  I went to another computer...it has the same network path as my other computer...they both hookup to a gigabit switch which goes to another gigabit switch where the server connection is made.  This second computer is using a 100Mbit card.  I opened VLC and started a single DVD stream.  INSTANTLY, my other computer's 8 DVD streams + 2 file copy sessions nearly halted!  Network utilization went from 30% to less than 1% and hit 0% repeatedly.  How incredibly strange?!  I'm going to do some more testing to see if the 100 Mbit connection was confusing the network (lol).

Any other ideas?  As soon as I introduce multiple clients into the network access, everything goes crazy.  I think the second computer's single DVD stream worked fine or close to fine (while the other computer halted!)

[benn600]

I am able to get 13 DVD streams playing flawlessly.  I'm not sure what the issue is but it seems to appear in strange ways for unusual reasons.  It appears that I need all gigabit connections but am not really sure if that would fix the issue.

[benn600]

I think I figured it out!  I started changing the switches around.  I downgraded the entire network to 100 Mbit and now I am easily able to stream two streams (lol).

So I'm now going to wait until we may move and at that point, I will buy a very nice, all-in-one, 48 port gigabit switch (or two of them).  So at $800 each, at least it should work.  I do notice some lag when I start streaming 10 DVDs simultaneously but we shouldn't need to stream more than 2-3 EVER.  I'm just using this as a test.

VERY pleased to have figured this out.  IT WAS THE SWITCHES!

But my 100 Mbit switches are fine.  There is also a chance that if everything was gigabit it would work fine.  I may use my gigabit switches but force everything to 100 Mbit.