Topic: 31.5 TB Server Build Log (Read 24924 times)

benn600 · « **on:** June 18, 2009, 03:48:20 pm »

This thread is intended to show the progression as I build a new server, replacing our current 7TB system which has been reliable for just over two full years now. The drive platters are starting to wobble and the chassis is unable to support all the weight from the 7TB of data.

Planning & Ordering
Please see the previous thread for HDD capacity discussion and the preliminary specs. I may detail what I actually ended up ordering later on but I think it is mostly or completely the same as the list.
http://yabb.jriver.com/interact/index.php?topic=51273.msg357953#msg357953

A brief overview is that this will be a single Quad-Core Xeon based mostly server-grade system. Twenty four drives will be used in the hot-swap bays (maxing it out) and all will go to the major array. My goal is to use RAID6 with triple parity and a hot spare, meaning that (24 - 4) = 20 drives will be available for space: (20 * 1.5TB) = 30TB. I will add two additional drives to fixed brackets internally for the OS operating under RAID1 (mirroring). The system will have 8GB of memory and a slim optical drive only during installation. The case supports an optical + 1 internal drive, but I need redundancy on the main drive so I'll give this up. During install I'll just set the drive inside temporarily. If I can find a way to have two drives + optical I will.

Day 1
The first shipment has arrived. It consisted of two boxes and included 27 Seagate 1.5TB SATA Hard Drives, 2 packs of 2x2GB DDR2 800 Memory, and a single Quad-Core Xeon CPU. The remainder will arrive tomorrow so I can hopefully start building this over the weekend. At some point I will place a small order of Supermicro Chassis accessories to go along with the case. In order to be positive I order exactly what is needed, I decided to wait to actually look at the case. I will likely order a locking bezel, two internal fixed-drive brackets, two additional spare hot-swap carriages, spare redundant psu. I'm pleasantly surprised at the price of these accessories (Supermicro) compared to my previous case (Chenbro) which seemed to cost a lot more.

benn600 · « **Reply #1 on:** June 20, 2009, 12:42:01 am »

Day 2

The rest of the components arrived today. I have been working on the system for several hours now. My previous server met huge problems along the way and this server is no exception. Here are the problems I've encountered so far:

1. One hot-swap carriage is partially damaged so I need to order a replacement (free of course, but annoying).
2. This big server order also included 8GB of memory for my Mac Pro and, buying a server motherboard, I thought it would take fully buffered memory...nope. So I ran to the local Best Buy to get half the memory (4GB) and will order the rest later at a little better price online. I will probably end up putting the unused 8GB in my Mac, meaning I'll get the 8GB I needed plus another 8GB and the 2GB already in there.
3. I got a slim SATA drive but didn't realize it plans on an IDE slim drive. So I powered down the old server and stole it's IDE drive, only to learn this motherboard has no IDE on it (which is fine but compounded this particular problem). Of course the problem would be averted with a $6 cable--that I can't get until next week from ordering it! I don't think we have any good computer parts stores that would carry something like a slim SATA power adapter.
4. Upon powering the system on, half the drive LEDs don't function properly. This EXACT problem happened with my last server--8 of the drives didn't light up (16 total). Then, they sent me a missing cable. Here, I'm not sure what the problem is. Sideband cable problem??
5. After checking in the card setup, one of the 24 is outright failed. This isn't too bad really but does not tell me if the three fixed internal drives are alright--nor does it guarantee the others are good...not until I actually set up the array will I know.

At the moment I am trying to get my Windows install disc copied to a USB flash drive. I have no other procedure path--I don't have any full size SATA optical drives that I know of. I must say that the are equally as many pros as cons between my previous Chenbro and this new Supermicro case. Each one literally has several advantages and several disadvantages over the other. Overall, the Supermicro is a better value, I think. It's still a long road ahead but I hope to make a bit of real progress this evening. As soon as Windows is installed and the array is building then things will hopeully be mostly good from then on out.

I still don't think this card offers RAID 6 triple parity, which is going to be a big disappointment. I was hoping for 3 parity + 1 hot spare. Otherwise, I suppose I'll got for 2 parity (RAID6) plus 2 hot spare. My previous Promise RAID card did offer triple parity but I didn't use it (only 16 drives then). This is 24 and they are bigger, probably less reliable!

benn600 · « **Reply #2 on:** June 20, 2009, 12:21:43 pm »

Day 3

What a change of pace...things are going much better now. I finally tried a USB optical drive and before I knew it, Windows was installing. I immediately started running Windows update and setting up the basics of the server. Finally, I installed the RAID driver and software. Another "before I knew it" and the array was initializing. I must say that I am much more impressed with the polished Promise web interface. Areca's looks like some amateur web developer from 5 years ago. The functionality is there but in a much less aesthetic manner.

On to the array: I was very disappointed (but mostly expecting) no RAID 6 Triple Parity. Again, the Promise card did have this available. With such a huge array of huge drives, I would really have liked an extra parity (3 total). So I took the best, RAID6 (2 parity) and was originally thinking to add two hot spares as an alternative. After thinking it over, I decided that it probably can't rebuilt two failed drives simultaneously, so in the event of a two failures rapidly, I have as long as it takes to rebuild the first drive to replace the damaged drive (perhaps 5 - 8 hours or more) and generally, this should not be a problem. I plan to keep two spare drives in spare drive carriages standing by for very fast swaps. Ultimately, this is giving me an array size of: 31.5TB in addition to some fixed internal drives. I am not completely sure how to finally set this up. Without a slim optical drive--and now that I know USB optical works so well, I may drop this--so that allows to fixed internal brackets. But since this is not dual processor equipment, there is a lot of extra case room. Opposite the power supply I can easily fit three drives but not nicely in good brackets.

It should be done rebuilding soon and then I'll start copying 7TB to it.

benn600 · « **Reply #3 on:** June 21, 2009, 01:06:07 am »

Day 3

Things are going very well so now is a good time to post pictures.

Empty Supermicro chassis with hot-swappable fans removed, ready to start installing components.

Another angle of the chassis.

No motherboard.

Motherboard.

Adding hard drives. Notice 20-pack is partially gone now.

Inside chassis with more components added.

Installing Windows 2008 Server from a slim disc drive through USB. It took me 4 hours to try this -- after trying every other method of OS installation. I was limited by some hardware issues.

31.5TB array being written to. Notice the top right drive is not lit. This is a hot-spare. The entire array is 23 disks of RAID6 + hot spare. I wanted RAID 6 triple parity but it isn't available on this particular card.

Visual of copying data. About 45MB/sec. Could be a bit faster but all in all, it will take well under 2 days to copy the main folder on the server. Perhaps part of another day to finish the remainder of the data. When everything is copied, I should go from 100% usage to 22%. The new array is exactly 4.5 times larger (7TB -> 31.5TB). My calculations are updates to how many days are left at the time of calculation--1.6 days, 1.3 days.

The old server was //Chenbro/Titanium and this new one, at the moment, is //Supermicro/Beryllium -- I am considering a shorter server name such as "Tin" to speed up tying the address in.

Right now I have two 1.5TB drives (identical to others) that are mirrored for the OS in three volumes: Main (OS), Other (Backups, etc.), Shadow (Shadow Copy for Beryllium once all set up). While brackets are not available, I do have plenty of room for at least 4 internal drives but I can't figure out a high-quality mounting scheme. Plus, I can't see a need for more than 1.5TB space for all the system purposes. For two drives I have to remove the slim optical drive, unfortunately, but at least things would be properly mounted and installed rather than held into the chassis with click ties, for instance.

Including all trouble I encountered and research/thinking to resolve the problem, this took about 10 hours to build (hardware, OS install--3 times). Adding all the drives really took a while and, unfortunately, it is not all totally complete. I am waiting on a few small accessories. I don't anticipate more than an hour of extra time, though. Initializing the large array took around 11 hours and then I ran a quick format and started copying the 5.5TB folder from the existing server.

newsposter · « **Reply #4 on:** June 22, 2009, 03:09:54 am »

newegg started up a 10% off all disk drives promotion yesterday.............

benn600 · « **Reply #5 on:** June 22, 2009, 08:00:41 am »

I used it three times to buy three more drives a day or two ago. It only works once and up to $10. Basically, it means I get three more drives for the same price I got them for a week ago. $120 each. I placed an order and then canceled it because I thought I found a better product. Ultimately, I realized there is a delay in the promotion becoming available back from the void order. When it was available, I placed the order and quickly tried to place two more HDD-only orders. It worked! Honestly though, I was only getting the price I got earlier. I was ordering the correct memory for this server, some adapter cables I need, and a few other small items.

benn600 · « **Reply #6 on:** June 22, 2009, 09:12:50 am »

Duplicating an entire DVD of content. Notice a very usable 431 MB/sec. This means reading and writing that much data. It does go down a bit over time. Is there a good, free program to test reading and writing speeds alone (write random data, read existing file)?

Copied video folder to new server. If I added another drive of space (removed hot spare, for instance), it would probably show 29.9TB (maybe 30.0!) but this is actually 31.5TB of hard-drive-company rated space (1,000 vs 1,024). In the background are the other three volumes contained on the RAID1 array, 1.5TB total into three volumes. Main will be for the OS and all associated software, Other for whatever else I may want to separate out, and Shadow for Beryllium Shadow Copy (worth its weight in [insert pricey element here]).

DarkPenguin · « **Reply #7 on:** June 22, 2009, 10:23:38 am »

When I started at XIOtech we were just starting to stuff 36gb drives into the Magnitudes. At 64 drive bays that works out to just over 2 TB's of storage. That would be a two cabinet setup. Maybe the size of a refrigerator. A single cab setup would have been just over 1TB. I found it funny when you could buy drives that were bigger than that. Now you've a setup that has almost as many spindles.

Neat.

benn600 · « **Reply #8 on:** June 22, 2009, 12:05:49 pm »

I have compared this to a few items as well. It has about a thousand times the storage of my iPhone 3G S (32GB). Oh yes, and the first hard drive I ever really dealt with was our Gateway Destination with a 2.5GB hard drive. I think it's still laying around somewhere. But 12,600 times the space. Of course drive for drive it's (1,500 -> 2.5) 600 times.

newsposter · « **Reply #9 on:** June 22, 2009, 04:47:31 pm »

Use hdtune for disk testing. Google is your friend.

benn600 · « **Reply #10 on:** June 22, 2009, 07:06:43 pm »

If I'm asking a question it's because I've spent way too much time on Google already. I can't tell you how much I searched for a GOOD speed test program. I keep running into junk...of course I want free. I'll see how this works for me...I may have already tried it. Okay so is this worth $35? I do buy programs from time to time but just to speed test??

benn600 · « **Reply #11 on:** June 22, 2009, 07:42:06 pm »

HDTune on the large 31.5TB array. Another post talked about 480MB/sec so this is clearly the limit of the card's processing capability. What if it could handle RAID6 calculations faster? I mean 21 drives * their max speed! Say 80MB/sec: 1,680 MB/sec.

Daydream · « **Reply #12 on:** June 22, 2009, 08:54:14 pm »

Flying high on a happy vibe, eh?!

Can you tell how much power the whole rig draws with all drives up?

benn600 · « **Reply #13 on:** June 22, 2009, 09:32:11 pm »

I know quite accurately the power draw:

Previous Server (Chenbro) ~ 200 watts
Intel Core 2 Duo
2GB Memory
16 * 500GB Western Digital Drives
3 * 320GB Maxtor Drives

New Server (Supermicro) ~390 watts
Single Intel Xeon
4GB Memory (8GB soon)
24 * 1.5TB Seagate Drives
2 * 1.5 Seagate Drives

So the new server requires about twice the power.

DarkPenguin · « **Reply #14 on:** June 22, 2009, 10:49:40 pm »

Isn't iometer the standard HD test? It is the one we used all the time.

benn600 · « **Reply #15 on:** June 23, 2009, 12:34:45 am »

I'll try it when I get a chance. Right now I'm knee deep in switching over to the new server as efficiently as I can. I had been throwing data wherever I could for quite a while and now that I have plenty of space, I have a lot of organizing to do. Although I'm really trying to keep in mind that this is a brand new server and its reliability is less than what I'll consider it to be after it's been running for a month or two.

I finally fixed the backplane LED problem. Earlier I noted that half the failed LED lights would not function. This is used for indicating a failed drive--something I did not have on the last server (exactly anyway). There are special connectors in addition to the 24 SATA cables (6 of them) that plug into the backplane. Unfortunately, they don't appear to indicate the direction they go. Of course we all know that diodes only function with properly directioned power and so that is probably why they were failing. After some trial and error, all 6 rows of failure LEDs function properly. I can manually select to signal a drive and I have done so for all 24 drives, showing that this feature works.

In addition to the main data copy from the old server -- which completed moments ago -- I have to setup and configure apache, PHP, Perl, Logmein, mysql, hmailserver, Media Center, and whatever else I'm forgetting. The data copy took 2.5 days, a lot less than I was planning for but when I look at the speed, I feel that it should have ran a bit faster. It ran at around 45 MB/sec on the fast end.

newsposter · « **Reply #16 on:** June 23, 2009, 12:58:43 am »

re: power. I'm seeing specs that say that the new Qnap 8 Tb server draws 35 watts with a full load of drives. More than a bit less than your hand-built for the same amount of storage.

Daydream · « **Reply #17 on:** June 23, 2009, 02:50:14 am »

Quote from: newsposter on June 23, 2009, 12:58:43 am

re: power. I'm seeing specs that say that the new Qnap 8 Tb server draws 35 watts with a full load of drives. More than a bit less than your hand-built for the same amount of storage.

Well it's a cool thing in itself, but not quiet an apples-to-apples comparison. Do you plan to stack 4 of those to get to a similar space size?

Let's see 4 Qnap -> 4 x $800 = $3200. Plus 4 x WD 2TB x 4 chassis (32TB to compare with 31.5TB above) -> 16 x $240 = $3840.

$3200 + $3840 = $7040. And no Areca controller, no server MB and CPU, etc. But I'm happy all "cubes" together draw only 160W

Heck I might beat Ben on overall cost if adding power consumption annual forecast into the mix!

benn600 · « **Reply #18 on:** June 23, 2009, 10:42:54 am »

Where is all the power going in my system? Obviously the drives take a lot but the Xeon??? Aren't a lot of servers these days using dual Xeon processors? I elected for a single to save power.

I finally installed Vista on my desktop. I had purchased a copy the day it was released a few years ago but hadn't really used it much since. So I installed with the latest updates and am getting amazing transfer rates from the server. When I start, it peaks at 98MB/sec! Unfortunately, it quickly drops to around 50MB/sec but I am still installing updates and drivers so in a bit I may know more about the whole issue. This time around I'm using two 500GB drives in RAID1. Funny how I happen to have so many unused 500GB drives laying around. When the drives fail from the server, I try them in a desktop and apparently I was able to find two that work. Many of them truly have failed. But I think having a high speed mirrored drive is important to realizing the speed benefit--in fact, I was doing a copy to my RAID1 drive. If I'm not mistaken, RAID1 mainly helps on reads and not writes. So if I move data from my local drive to the server perhaps it would stay at that 98MB/sec speed! Clearly it is filling up the cache in a second or two and that's why the speed drops so rapidly. It literally goes to 98% gigabit LAN usage for 1-3 seconds and then half that.

Daydream · « **Reply #19 on:** June 23, 2009, 03:23:46 pm »

Regarding power, a normal HDD takes 11-12W (on random writes), a green one ~4W. The rest of the difference is just because of a different class of components. I'm not sure I was clear, but I'd take a unified build like yours over any other bundled solutions. Heck my desktop rig with 13HDDs in it or around it gets close to 400W (it's a good-for-everything kinda design not just a file server).

Funny how you mention the 500GB drives. I have a bunch too, don't know what to do with them

. Back up stuff to them weekly and otherwise keep 'em on the shelf?

For LAN speed yeah, obviously the writing speed at the desktop end is the problem. Unrelated, just curious - did you tweak the network settings -> jumbo frames, etc?

benn600 · « **Reply #20 on:** June 23, 2009, 11:58:26 pm »

I have not tweaked network settings just yet. There are a lot of other tasks I need to complete before the server switch over will be finished. I tend to bundle all the new hardware/software I can into a single upgrade so everything can be updated and reinstalled all at once with the most optimal configuration at the moment.

As far as the old server, with 16 * 500GB drives, I plan to use it for an offline backup server. Unfortunately, I'm literally already well past its capacity but it will be backing up more than I did before--only about a terabyte previously backed up.

benn600 · « **Reply #21 on:** June 24, 2009, 01:24:25 pm »

My Mac copies data at about 57MB/sec sustained from the server over gigabit. For some reason my Vista desktop, basically adjacent to the Mac in every way, gets 12 MB/sec, the theoretical max of 100Mb ethernet. So what's going on with this??

DarkPenguin · « **Reply #22 on:** June 24, 2009, 02:26:09 pm »

Is Vista fully patched? Is it trying to fool around with your shares in the background?

benn600 · « **Reply #23 on:** June 24, 2009, 06:58:39 pm »

It is fully patched. I don't think it's accessing shares in the background.

Funny story: after a fresh install, I elected to install all updates. It took a couple hours and then rebooted. After another hour or so, it restarted once more and said the updates were not configured properly. Reverting changes.

NICE! 3 hours of updates, which I constantly kept checking, thinking it HAS to be done by now. It was never done...and this is on a mirrored volume so it should be at least a little faster than a single drive. But all that time and then it took most of the time once more to undo everything. So I then reinstalled again (not going to risk all that activity for errors) and installed 3-10 updates at a time, saving the newly posted items (IE8) for the end. Thankfully, each round went very quickly and didn't give any errors.

newsposter · « **Reply #24 on:** June 24, 2009, 07:41:02 pm »

always try to download all of your updates before installing any of them.

Get The Important Once in place first. .Net, Service Packs, etc.

Now go cruising through hardware mfgr suppor sites and make sure all of your device drivers are up to date.

Once all that is done (and the box rebooted), point Microsoft Update at the internal files it should have downloaded.

Asking any intel/msft machine to stay up and stable (locally on the network/internet) through a 3+ hour update session is (as you've found out) a complete waste of time. It's also potentially distructive to the OS load if you get so screwed up (missing or corrupt downloads, etc) that changes can't be backed out.

benn600 · « **Reply #25 on:** June 25, 2009, 11:21:18 am »

I adjusted the registry key which throttles the network connection during media streams. I am now getting much better throughput. When I start a copy from the server, I easily start around 100 - 120 MB/sec. After a few seconds, it drops to half that. I bet it is reaching the maximum speed of my mirrored 500GB volume. Nevertheless, this is very impressive and actually peaks the network at around 95 - 99% initially--which is the highest I've ever seen a gigabit connection. I saw that on 100Mb network a little more often.

I really could use 10 Gigabit networking! That would mean a maximum speed of over a gigabyte per second. With my server at 500MB/sec I could actually realize the full potential of its speed. I would only need it on my two desktop workstations...or even one of them. 4:4:4 video anyone?

Daydream · « **Reply #26 on:** June 25, 2009, 01:45:33 pm »

120MB/s is the absolute limit of Gigabit! Any faster and the wire will snap!

10Gigabit, err... $3000 a network interface, plus the fiber setup, kind of expensive

4:4:4 hehehehe... for post-production and archival? You work with some cool equipment if you can capture in 4:4:4.

benn600 · « **Reply #27 on:** June 25, 2009, 03:51:21 pm »

I can't, I just know and have heard about 4:4:4 video. Somewhere I heard of a demo of some 4:4:4 video and I remember hearing they have a big RAID setup to provide the data rate necessary for it and I think it was somewhere close to the max of this server.

Yep, 1,000Mb/sec = 120MB/sec. I do see very close to this--and actually saw 124MB/sec at one point but that's probably some sort of cache causing the seemingly impossible microsecond speed.

When are we gonna have 10Gb network ports built in to every computer? It really needs to run over the existing Cat6 cable (cat5e). I still remember when (and much farther back TOO) gigabit was starting to be available at Best Buy (residential work) and the cost of it. Of course back then I really don't think it would have helped anything. Now, with all this video and hopefully more HD content as time goes on, it is practically the low end starting point.

benn600 · « **Reply #28 on:** June 28, 2009, 08:59:02 am »

Oh no, used space keeps creeping up. I still haven't fully switched over to the new server. I think it will be a while, too.

benn600 · « **Reply #29 on:** March 05, 2010, 10:36:47 am »

It's been eight months. I am still amazed no drives have failed. Below is the usage. At this rate, I guess it will be full in 41 months, a little over 3 years.

Do hot-spare drives remain powered on and spinning? The 24th drive is a hot-spare and I have always wondered this. If it was spinning, then it would be approaching its end-of-life when needed...

jmone · « **Reply #30 on:** March 05, 2010, 01:45:34 pm »

Good Q - There was a great thesis writtin (I can lay my hands on it) that looked at why many RAID systems failed - and they found that the change of multiple drive failure was an factor of the number of drives in they array and the age of the drives. Sounds fair but the RAID failure rate was surprisingly high (much higher than you think) as most RAID arrays had the same drives, purchased at the same time and hence tended to fail around the same time - and unfortunately many times during the parity rebuilding period trashing the who array.

Daydream · « **Reply #31 on:** March 05, 2010, 03:47:18 pm »

Ben, did you look at any SMART info for the drives? Would you trust it?

benn600 · « **Reply #32 on:** March 06, 2010, 07:55:54 pm »

The controller has this information and I will look at it again. I am merely referencing how the RAID system has not failed any drives which would automatically start using the hot-spare. It does worry me that I bought all of these at the same time. I contemplated ordering from separate vendors but that would have meant hundreds of dollars more, all for a potentially reduced risk.

INTERACT FORUM

Author Topic: 31.5 TB Server Build Log (Read 24924 times)