Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[benn600]

I have begun the moving process to switch from my old Windows XP USB Raid5 to FreeNAS.  I must say I highly recommend FreeNAS.  You can access it at http://www.freenas.org and download it.  Installation takes around 32 MB and it can be installed to a USB flash drive (freeing up an IDE/SATA channel) or any hard drive.  After installing, remove the CD, reboot, and hopefully your computer will boot to the new installation.  At that point, all you really need to do is setup your LAN IP which basically consists of pushing a number (I think it's 2) and then 'y' to use DHCP.  Then, you'll get an IP address.  Now go to another computer and enter that IP.

The Web GUI is quite impressive.  First, you must add your hard disks in the Web GUI to allow them to be formatted.  After that, you can format them with UFS or a few other common file systems but UFS is the preferred.  You can also setup RAID0, RAID1, or RAID5 all in the webGUI, completely software based.  I currently have my new RAID5 regenerating.  You aren't supposed to format the array until regeneration is complete.  Also, since it is somewhat a "beta," be aware of rules and instructions because some actions are not prohibited and they could create problems.  Also, with beta software and any NAS at all, backup!!!

My case holds 10 hard drives and I plan on building a bracket to hold the boot HDD (I couldn't get USB booting working even though my mobo supports it).  I may also add a few more drives (160GB's) as an onsite backup option for the most important data.  I did run a quick test a few days ago with smaller drives and after removing a hdd, I was still able to access the array.  I did not try replacing the drive again.

With regenerating, I have been keeping records and it looks to be running at a little under half a GB per minute.  I am predicting it will be finished 12 hours from now.

For everyone who has tons of media, this may be a great option.  The reason I'm using my 2.4GHz P4 and nicest case is because it holds the most drives with breathing space and lots of circulation.  Another case I considered keeps the drives almost touching eachother--they would get way too hot.  I also was having trouble in the beginning.  I advise not changing the auto power off, power management, or noise settings.  I set the drives to power off after 20 minutes and strangely enough, when I turned on FreeNAS, the drives spun up and right as FreeNAS was about finished booting, all 10 drives shut off with about a 1 second gap between each (it sounded cool, lol).  But then it said drives were disconnected!!  Just leave it to always on and avoid that potential issue.

Take an old PC (with at least 96MB of memory) and throw in a few drives.  I like RAID5 for redundancy.  From my limited tests, write speeds seemed decent.  When everything is finished tonight, I'll be moving lots of data back to the array and can post on the speeds.  The WebGUI has a really nice graph feature that updates live, showing network usage and CPU usage.

There are currently little or no permission features so only use this on a trusted network.  It seems nice to have one box with all my data in it.  Compared to the USB solution, now all I need is one power cord and one network cable...compared to that + 5 USB power supplies, 5 USB cables, a hub, lol...just a mess.  Now it's all inside the box.

Be careful with the SATA cards because FreeBSD (FreeNAS) has limited support.  A 2-port card I have is not supported but the 2 4-port Addonics cards I both ($45 each) have been working so far.  I'll have to wait until I start actually writing to the array before I can be sure they will work.  I also had to buy 10 SATA cables for about $6 each and they are quite long...I'm thinking 39" but that is helpful because the top drive uses a lot of that distance.

I also purchased 4 adapters to put the hdd's in the regular, large bays.  They were about $20 alltogether.  Finally, 2-4 power Y-splitters because my 400W power supply only has 7 connectors but I need 11 (10 drives + 1 boot drive).  Now I have 10 unused USB enclosures (the My Book's).  I almost wonder if I could sell them?

This is a great piece of free software to really put together a great NAS.  Enjoy!

[glynor]

Wow....  Great find Benn.  Thanks!!

[johnnyboy]

Hey nice going Benn.

You've definitely taken us on a nice little journey the whole way through your experiment and kept us nicely updated.
Looks like you finally came to the same conclusion more or less that we all suggested but you got to do it the scenic way and got to learn a hell of a lot about it all along the way.

You probably could sell those cases on ebay quite easily I'd imagine - people always want hard drive enclosures and those are quite nicely made ones.


Wish I had your resources to do all this at home, I'd love to setup a file server. I'd also set it up as a domain controller and give me a nice little home networking setup for all my computers for permissions etc to make file sharing etc really easy.

One day

Keep updating us!

[benn600]

I did learn one disadvantage to large file servers.  It seems like if you exceed 2TB with UFS, it automatically sucks 8% of your drive for inventory or something.  With 10 drives and losing 1 (RAID5), you'd think I'd get 4.5TB.  Well, here is what Windows reports.  I'm surprised it reports NTFS when it's a UFS partition but I guess Windows probably doesn't even know about UFS:

File System: NTFS
Capacity: 3.96 TB
Used Space: 324 GB
Free Space: 3.64 TB

What bothers me is that freshly formatted, it shows 324 GB used!  I wouldn't mind as much if it just hid that from me and said hardly anything used (and lowered the capacity).  Now, I have to always remember to subtract 324 GB if I need to know how much data I have on the NAS.  I understand the 4.5TB being lowered to about 4TB.  That is just the way drive space is calculated (1000 vs 1024).  When I was using 5 drives on RAID0, it took 2.5TB down to about 2.2TB, so -.3TB plus about 40% more would be right at around -.5TB.

I am absolutely AMAZED at the write speed I am getting.  Unfortunately, I have no way to test gigabit networking even though my two main computers have it because I have 3-5 switches in between the computers and the CAT5 cable might have to be upgraded if I understand it correctly.  I always judge the performance with an ALT+CTRL+DEL and then viewing the networking tab.  Right now, I'm getting a solid 70% write speed.  That is reading from a USB drive so if I read from a SATA drive, it might even be faster...I'm restoring my 275GB of FLAC now.

I should add that I have seen that percentage at 99% quite a few times.  If I'm reading and writing to 2-3 locations simultaneously, it will often hit 99% and stay there.

For comparison, with Windows hacked RAID5, I would get an average of 30% when writing and worse yet, it would fluctuate like crazy.  One second it was 30% and then it was 5%...then 50%.  I'm guessing it was a lot to do with a Windows buffer.

The system is a quite decent 2.4GHz P4 only because I needed the large case, big power supply, and onboard SATA.  2 4-port PCI cards + 2 onboard SATA give 10 total SATA.  The boot drive is on IDE.

Generally speaking, the only disappointment I have now is the used space issue.  Getting it setup was a lot of trial and error, trying different hardware and software configurations--most people probably won't have the issues I ran into.  But in the end, I think everything is working great now.

Once I get all my data restored and re-rip (sadly) all my DVDs (didn't have enough temporary backup space) I'll be a calm happy camper.

Scary addition: As I was getting ready to call it a day, I decided to take one last look at the forums (FreeNAS, MC, etc).  So I walk over to my desk and with wet hands, didn't want to touch my chair.  So, I started bumping it out of its location.  Ooops.  The leather caught (by friction) the USB/power cable from my 11th My Book and slammed the drive flat on its side--while running and restoring my ONLY copy of all my music (other than spending a month re-ripping).  You cannot imagine how scared I was.  That's the worst thing I've ever done to a running hard drive.  3 seconds after it slammed to the desk surface, it shut off.  I then turned on my monitors to see that it had stopped copying.  Oh great...now what.  So I unplugged all the cables and plugged them in again.  It turned on and thankfully, has been copying for 10 minutes no with no issues.

It's interesting because I was just experiencing a WD drive a day or two ago in that I had it hooked up to my temporary, 2nd FreeNAS and I had to rearrange the drives (All piled on the table).  Barely touching the drive caused the head to relocate to its safe location.  I wonder if this drive did that?  I should think so because the impact the drive experienced was tremendous from my perspective.  I don't recall hearing any bad platter noise, thankfully.

More or less, I'm just very thankful nothing bad happened (that I know of yet) but I was freaking out when it happened!  Imagine a My Book's cables pulled almost like a laptop's power cord tripped over..and it being flung over on its side (the worst angle for force)!

[johnnyboy]

Ikes - that would have really sucked loosing all your music too!
I've now backed all my music onto DVD's all of which are 'verified' after the burn to make sure they're safe.
This way hopefully I have a pretty stable backup as well as my hard drives (also have a copy stored on my X-Box on XBMC).

Your whole thing has got me wanting to do it I just know I couldn't get away with it without getting hassle from my room mates over the extra wires etc and to be honest, I guess I have no real need except wanting to copy all my films onto it which is a nice extra.

How are you storing all your DVD's?  As pure straight rips or are you compressing them at all?
If you use something like DVD Shrink with them all you'll find you can get them all down to 4.5Gb and loose barely any quality with them. If you encoded them all to xvid for instance on the other hand though you could probably quite easily get each DVD down to 2Gb and still have blindingly good quality (most films are reduced to 700Mb when done to xvid/divx and are pretty darn good quality so putting the file size at 3x this would give you pretty much perfect quality).
It's a bit like saving all your photo's as .bmp's or something and saying 'well I want perfect quality on them' so ignoring the fact you could save them as .jpg's at say 95% and have no 'visible' difference but save yourself around 3x or more the file size. Heck - the whole industry has accepted the fact that saying you want 'perfect' is pretty much pointless which is why 95% or so of camera's just save files to jpg's and dont use something insane on space like bmp's or whatever.

[johnnyboy]

Oh Yeah,
other last point - you should probably speak to some *nix nerds or look it up online what the different file systems are good for - you might find some are designed alot more for larger files and so use up less space.

With FAT for instance, it needs a file table for all the disk allocations. Each section it creates needs to be indexed in the FAT so if you use small allocation sizes then you end up with a larger table to point to all the segments.
With modern large files however (eg DVD's, mp3's, etc) where they are all at least 1Mb, using large segments results in smaller FAT tables so saves some space there.

The trade off obviously however is that if you (purely an example) used a 1Mb segment and your file was 3.1Mb then it'd have to use 4 segments (eg 4Mb) to store it.


Worth researching the file system to be used quite a bit as it could make quite a big diff to how much wasted space you have!

[benn600]

UFS is pretty much the best filesystem available to date.  I was researching and found that there are probably two causes to 324GB being stolen.

1) It saves a portion of the drive back for the root user in case others (me) fill the drive up (I'm also root, lol).
2) It utilizes soft writes which takes 8% of the drive.

A helpful individual on the FreeNAS forums gave me commands to enter to disable soft writes and change the percentage saved but I can't get them to work.  On the FreeNAS shell, it says the commands don't exist and on the WebGUI command page, it just prints back what I entered and doesn't do anything.

I also learned that the percentage is basically used to speed up writes and as a work space for defragmenting.  This is troubling because Windows never really took any space but it was about half as fast and I needed to defragment from time to time.

The whole reason I dropped my second direct backup array and combined all 10 drives to one array is because I was running out of space.  The reason I'm running out of space is because I had 1.6TB devoted to direct copies of DVDs (I own).  Unfortunately, I have calculated that I could run out of space again (LOL) after only I forget how many, but a small number of DVDs--2-3 hundred.  Don't forget I'm trying to get 30K-40K old analog photos/slides scanned in which could take a few hundred GB's and I keep getting CDs which add up.  I feel like even after all this time, investment, and work FreeNAS is stealing space from me.  I want all of my drive back.  People have told me that lowering the percentage of 0 makes the filesystem unusable...lol

I'm going to go check the FreeNAS forums again then study for my last test this January Term (one class in January)...then when I get back, I will continue working on this.

Addition:
Arne is a nice individual who is very intelligent and spends a lot of time helping me (and others).  Here is the response I got--

RE: URGENT: I lost 8% of my space?!
By: R. B. Riddick (riddikk) - 2007-01-25 05:06
Hmm...
Dont know where the CLI is... Shell sounds good...
 
1. it is "umount" not "unmount"
2. "umount" does not say so much...
3. but "tunefs -p /dev/raid5/Book1" says many lines...
4. U could try the "shell" again...?
5. U can do all this, while the graid5 device is rebuilding... it is fully usable, BUT it does not tolerate a further disk failure then...
6. As I wrote: 0% r not recommended... Maybe 1%... Just try it... :-)
 
-Arne

So I guess I mispelled umount but I don't think tunefs worked.  I will try it right away later today--I'm anxious.

Also, I destroyed my array and reinstalled FreeNAS (2 minutes) but then recreated the array from scratch and now it's rebuilding.  I just kept messing with it as a test and figured I don't want any low level errors I don't see to plague me so I'll just start it over again.  It takes around 20 hours to regenerate so it will be done tonight around 8-9 PM CST.

[johnnyboy]

The only way you won't lose any space is if you do them as individual drives, but then you have no hardware redundancy.
As soon as you want that, then you need the RAID5 which is going to eat up space.

Have you checked the actual price for a RAID5 card? It seems with all this cash and time you're spending, a simple card might not be that big a deal and it'd give you the ultimate stability/speed/space I'd imagine.
http://cgi.ebay.com/Sil3114-SATA-Serial-ATA-5-Port-4-1-Raid-Card-upto-4HD_W0QQitemZ230079328469QQihZ013QQcategoryZ39968QQrdZ1QQcmdZViewItem
Two of those would give you 8 drives - 3 would give 12 drives


What you actually want is one of these:
http://cgi.ebay.com/Dell-Powervault-128T-LTO-4-0-TB-Tape-Library-warranty_W0QQitemZ160075996902QQihZ006QQcategoryZ51090QQrdZ1QQcmdZViewItem

And then no raid 5 at all - just stripe them so your getting maximum speed reading/writing to those disks and maximum capacity and then your using the library for backups lol.

[benn600]

Lol.  First, I'm not spending any more on the project.  I got a great deal on the drives and I already purchased 2 4-port SATA RAID controllers for about $3 more each than the one on eBay you sent me...although mine don't do RAID5 but if you want RAID5 with 10 drives, aren't you required to have 10 ports on one card?  I don't think RAID5 can take drives from other cards in the system.

That 128T is misleading.  I thought you were sending me a 128TB NAS--lol.  That's a lot.

Addition: Like I said, I have all my most important data backed up except my music.  I would really like to backup my music but it would take around 60 DVDs.  If I could buy dual layer DVDs for about the same as DVDs, I could manage 30.  Now I don't have them backed up at the moment.  Once the new RAID5 array is up and running, I'll copy it there and my 11th drive is my backup drive...so at that point, I will have two copies (with one redundant).  All our profiles, documents, photos, web sites, etc are backed up to DVD and hard drive.  Excluding video, everything I have is right around 500GB.  That's hard to believe!  Music is only (lol) 275 GB.  I remember when I was lucky to have a 80GB drive.  Ooo, then I built a new computer and got to upgrade to 2-120GBs (3 years ago).  My current system I built a month ago is running those same drives I bought 3 years ago.

[newsposter]

In the Unix/Linux world, UFS is ancient and only suppored as a legacy filesystem, primarilly in the old SunOS (pre-Solaris, pre 1995) world.  XFS (from SGI) and JFS1/JFS2 (from IBM) are the current state of the art filesystems with Ext2/3 and ReiserFS bringing up the rear.

Not only is freeNAS the slowest of the 'storage servers' out there (by a 10x-15x margin), it has a history of data integrity and unrecoverable RAID corruption issues that are not seen in things like openfiler or NASlite.  A $30- license for either of those two products is money well spent.

[benn600]

The Open filer is free, isn't it?  Wow.  Now that I know there are alternatives to FreeNAS, why didn't I consider them earlier?  Should I switch?!  It's easy to switch now!  I haven't copied any data yet!

Please tell me which I should use.

Addition: I am looking at Openfiler and it looks like FreeNAS except it's a lot more advanced in its feature set (LOL).  It is NOT prerelease.  It also looks like it used to be a commercial product that was donated to the open source community.  I am downloading it right now and am going to cancel my FreeNAS setup and then try this out.

Thanks a LOT for recommending these alternatives.  I am definitely interested in this package.  FreeNAS is only about 30MB where as this is over 200MB so it will take a few minutes to download...waiting...lol.  I can't wait to burn and install it.

[jgreen]

poster--

For these down-and-dirty storage bins, how would you rate naslite or openfiler vs linux+samba?

[johnnyboy]

lol - dude I'm just loving your whole project.
This sudden late entry into the game by newsposter is interesting - time to start doing some heavy research - search google for 'freenas corruption' etc and see just how bad an issue it is.

A man with solaris experience is a valuable asset - butter him up for some of his huge depth and wisdom!

Howdy newsposter

[benn600]

I wish Sourceforge would use normal forums like this or others.  It uses this crazy thing that is hard to follow and keep up with.  lol

Now I found this on OpenFilder:

tried for the last 2 days to make Openfiler work but it's crap! When a LVM is made and data is stored on it and you do a clean install all the data isn't there any more!!! And this is full-blooded NAS?!??! No way, maybe I do something wrong but I can't imagine that when the harddisk is broken that you can't get the data back from the storage volume... I installed FreeNAS again and could mount all my harddisks WITH the data (I used another drive to test Openfiler)!!!! That's the way it should work, always can get your data back. Hopefully FreeNAS will get more features (like adding ACL's to maps not just the whole disk and e-mail alerting). I don't know how long it will take but hopefully it's soon. Maybe you can tell me why Openfiler can't mount it's stored data back on a clean install???

It basically gives a personal experience with OpenFiler being a bad choice.  Any ideas?  I need to know if I should stop my FreeNAS from regenerating and try OpenFiler.  It took 5 hours to get to 50% regenerated and I'll have to start it over if I try OpenFiler--not to mention the time it takes to try OpenFiler, etc.

Addition: I installed OpenFiler once and I guess I forgot to setup RAID5 before hand.  So, I am reinstalling it right now and I made sure to setup the array during the installation process.  I noticed that OpenFiler also supports RAID6 which is impressive because I rarely see that!  I'm thinking that allows 2 concurrent drive failures but takes 2 drives for parity.  I almost considered that but I'm not that worried.  I have never had a formal, true drive failure.  I've had some close calls but never been bit from a failure.  These SATA drives seem quite reliable so far.

[benn600]

My dad just recommend I take pictures of the project as I continue on.  He will be helping me gets some extra fans mounted in the case with filters to help keep the drives cool.

The only remaining issue I don't have a solution for is the 2 HDD's in the external, small bays (floppy drive holders).  These bays are perfect for hdd's but both bays are basically touching so two drives would be touching and there isn't an easy way to add a fan.  The other 8 drives have a fan directly blowing air on the drives so it's just the issue for these last two.  Lol, wouldn't that be terrible if those two failed at the same time from too much heat and I lost everything?!

[benn600]

'Nother big change.

1) FreeNAS has reports of data loss when using RAID5 because the packages running it aren't quite stable enough to rely on.
2) OpenFiler was okay...I got a RAID5 array built but I couldn't figure out how to create a volume from it and then share it.  Other people were having the same problem based on the forum posts and I didn't see any solution.
3) Now I'm trying Suse Linux.  I realized, after trying OpenFiler (Red Hat Linux), that Linux has inherent RAID5 support built in.  So, all I need to do is install Linux, setup a RAID5 array, and then configure Samba -- I'm done!  Well, I installed Suse and configured Samba--it has an amazingly simple GUI front end in Suse.  Then, however, I opened Partitioner from YaST and nothing shows up!  I have a post in the Suse forums asking why but if I can't get that open, I can't get my RAID5 array created nor can I mount it and share it.

I'm so close now!  Suse is really neat I almost want to move completely to Suse!  Is there an emulation program that would let me run Media Center in Linux lol?  I don't want Windows + MC, I just want MC in Linux...lol.  That is unlikely at this point but something to think about.

[johnnyboy]

Hey

With that forum post you read that put down OpenFiler and put up FreeNAS I'd just say one comment means absolutely nothing really. Every single bit of software on the web has some bad comments about it somewhere but the main thing to realise is you know nothing about the person who posted this.
They could just be completely stupid and not know what they were doing and so blame it on the software when in fact it was user error.
Could have been a specific bug in their hardware/software config - could be a million things and in fact if you only find one bad review about some software you should usually take it to mean that its pretty ** good software.

If you search for FreeNAS I'm sure you'd equally find people who have left it and gone to OpenFiler and sited bugs.

Gotta say - I'm impressed with how fast you seem to be learning and picking up all this new stuff - learning *nix isn't easy or fast!

And have you tried WINE or VMWare for Linux?
Not sure how well it'd work with MC but should do the job.

[benn600]

I made another change.  I really couldn't get anything working right...

1) FreeNAS took some space, used older filesystems, lots of reports of data loss.
2) OpenFiler, I couldn't even get it to work!  I set up a RAID5 array during setup and then let it "progress" or whatever (regenerate) which took 10 hours.  Then, I couldn't find any way to get a volume created and/or shared.  Other people on its forum were having the same issues.
3) Then I tried Suse.  I had the RAID5 array already created from OpenFiler's install but would have re-set it up if I could.  During install, I selected to customize partitions but in the Expert Partitioner page, nothing showed up.  Then, after installing I had the same issue.  I went to the partitioner and it just didn't show anything other than the program title and an Apply button which always said you've made no changes!
4) Ubuntu Server.  I setup the RAID5 array during the install and then it installed.  I did read a step-by-step tutorial to help me but once I realized it's all command line (basically), I decided it wasn't that great either and I'd be force to doing a lot of typing and configuration.
5) So guess what...lol!  I'm back to Windows.  I know it's sad that I have to settle for "forced" RAID5 in Windows but I think it's the best in the long run.  It means I can keep my old Windows server items running like Juice podcast receiver, hmailserver (GREAT program), etc.

Have I gone full circle yet?  I'd like to finish this project up so I can start re-ripping 300 DVDs and moving all my backed up data back.  I have regenerated the RAID5 setup over 6 times now (some partial then crash or manual stop).

I just want to finish this.  I will definitely post some pictures just for fun.  I used some of my non-existent hardware skills to add a big fan to help keep the drives cool.  I still need to secure 2 more hard drives in the case (or more) with custom brackets but that will be dealt with later.

[johnnyboy]

NOOOOOOOOOOOOOOOOOOOOOOOOO!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

lol - you were heading so well in the right direction!!!

You've experienced the normal *nix experience - it's a pain and very hard work to learn to use/do, windows is ten times easier but the difference with the *nix solution is that once you've finally mastered it and got it up and running it all just runs and keeps running and running and running forever and ever and gives you a much more stable better performance solution.

The other great thing about using a *nix solution is that it will be a lot trickier and harder to learn to use and to use - but I promise you, when you graduate, if you want a computer job the *nix solution and experience is going to get you a load further and better $$$$$$$.

In the UK at least, for a normal IT support type job you're going to max out at around £50k a year probably - thats MAX so you gotta be the best for it. For *nix support however, that moves up to more like £70k a year.

The windows solution might be up fast but it'll come down out of the blue just as fast and be alot less reliable with lower performance (as you've already seen).

[benn600]

That will be the test.  Once it's done regenerating (it's 2/3 done now), I will see how fast I can write to it.  I could write to FreeNAS (RAID5) at 70% of my 10/100 network connection.

[jgreen]

benn--

Give the *nix+Samba another hammer, and report back, if you would.  This is something I'm pretending I'll do one day, although drive capacities are increasing so quickly I may simply have to learn how to plug in a USB drive (currently I have my neighbor come over and do it for me).

I still don't know where you're getting all the cash.  When I was in college we used to steam off the wallpaper and make soup out of the paste.  If it was lumpy we pretended it was clam chowder. 

[benn600]

Sorry.  I have it all setup and don't want to go back to square one again.  The write speed is very impressive.  It looks like I get more burst speed.  About 80% of the time I am copying, the percentage is at 80%.  It only drops below that for about 20% of the time.  I'm not happy that the speed isn't consistent and steady but for smaller writes, it will probably be instant.  Overall, the speed seems to be very similar to that of FreeNAS but I no longer lose 400GB.  Here are the specs I get from the drive when mapped.  If I dare, I will post an actual screenshot (ooh, lala).

Used Space: 2.47 GB  (I just started copying!)
Free Space: 4.09 TB
Capacity: 4.09 TB

The graph still barely shows blue (used).  I love how I'll be about done with everything I have (ripping) and I'll be at or below half.

When I lower the refresh rate on my network monitor, the rate is right around 65% so it looks like it is marginally slower but has more burst speed.  I'm guessing Windows does more buffering to get fast instant writes but long term writes take a bit longer.  This is more than twice as fast as a 5 drive RAID5 over 1 USB port.

I'm thinking that the more drives you have in RAID5, the faster it will be.  Think about it.  For every 9 bits, a 10th bit has to be calculated for the parity.  So, 1/10 of the bits are calculated.  Before, 1/5 of the bits were parity and had to be calculated.

My overhead is only 10%.  Wow.  If I was at 11 drives, the overhead would only drop to 9%!!  lol, one percent.  Oh, I guess it would be 11% for 9 drives.  Here:
Drives: 3, 33%
Drives: 4, 25%
Drives: 5, 20% *my old*
Drives: 6, 17%
Drives: 7, 14%
Drives: 8, 12.5%
Drives: 9, 11%
Drives: 10, 10% *my new*
Drives: 11, 9%
Drives: 12, 8.3%
Drives: 13, 7.7%
Drives: 14, 7.1%
Drives: 15, 6.7%
Drives: 32, 3.13% (max in Windows RAID) *my desired, 15.5TB*

When you look at it like that, it seems like a numbers game.  Many of the percentage drops are only a few %!

[benn600]

I have some rate stats.  I have been copying 275 GB of FLAC music with one JPG per folder.  Smaller files often slow copy rates but these are all reasonably large files at 20-30 MB + 1-2 MB jpg.

In a period of 3 hours (+10 seconds), the copy copied 75.8 GB.  That equates to 25.27 GB/hour.  Calculating, that is 421.17 MB/minute or 7.02 MB/second.

That seems very good to get 7 MB/second.  I have read reports that software RAID5 is limited to much less than that but I have proof that it works well in my case.  It may slow down over time or as the drive fills up but I've got a long way to go.

Theoretically speaking, if I were to fill the entire drive at that rate, (4,090 GB / 25.27 GB) it would take 6.74 days--and assuming I can keep that rate.  When I rip DVDs, I find that the network slows my speed so I rip to my local drive during the day (while copying in the background ones I already ripped) and then by night time, I usually am about out of hard drive space so I just get the rest moving and it finishes only shortly before I wake up...then I have a fresh, clean system drive to rip to again.

Addition: I also want to suggest JDiskReport.  I mean anything J has to be good.  But, it will be very handy once I finish re-ripping all my DVDs.  Then I'll be able to see with nice graphs exactly where my space went when I fill it up.  But really, I wish I would have run that program before deleting everything.  There is no doubt that a few clicks to generate some amazing graphs detailing many different attributes aggregated across your data really will provide great information on your strategy.  I'm guessing that the graphs will be worthless unless I hide DVD's because those will surely take up 2TB alone when I'm finished ripping.  TB's aren't as fun as GB's.  I mean, what?!  4 measly terabytes?!  I remember our 2.5GB hard drive many years back (it's still running, lol).  Who wants 2.5 of anything--or 4 for that matter?  Not I!  I want 160 of something.  I want 160 Terabytes!  That's the ticket!  That would be a thousand times my largest hard drive ever purchased (aside from 11 50 GB drives).  Realize that other than that pile of 500 GB drives, the largest I had ever purchased before that was 160!  I was hurting!  I had held off until prices more reasonable and then what do I end up with?  4 measly units of space.  (lol--units).

'Nother Addition: lol.  Now what.  My FLAC just finished copying.  It finished around 2:30 AM and I had my podcasts download at 3:10 PM.  Now, I'm moving some DVD files from my FreeNAS box (few 120's, 160's) and I'm getting a CONSISTENT 90% write speed!  Before I was getting 55-65%!  It is not dropping at all and I have the update on HIGH!  lol.  That is INCREDIBLE.  Theoretically, the max is 12.5 MB/sec and 90% of that is 11.25 MB/second.  That is amazing to say the least.  See, I have seen 99% on that before but it usually takes two simultaneous writes or reads to max it out.

I also tried copying a folder of FLAC and it stayed at 70% but when I copied a 2nd simultaneously, it went up to 90%.

I did order a cheap (with rebate) 8 port gigabit switch that I want to put toplevel where the server is.  Our house is broken down to about 5 separate branches.  Of which, another switch exists down the pipe.  So, to get gigabit around the house would require a lot of switches.  I think the only two gigabit capable machines are the server and mine.  I feel that this way, each branch will get a full megabit of speed because the switch can get a gigabit into it and then split it out.

How do I know if my cable is CAT5e?  I know CAT5e & CAT6 work with gigabit.  If it comes with one cable, I'll be fine.  I just need one to go to the server.  My problem was that if I was copying lots of files from the server on the upstairs branch and went to mainfloor 2nd branch to play a DVD on my Xbox, it would stutter.  This way, theoretically, the server will have more bandwidth available to send out the data.  I would think the read speeds could be amazing because it doesn't have to calculate anything and should just work amazingly.  I've never experimented with gigabit either.  I would like to run a temp cable to from my desktop to the switch just to see what kind of speeds I can get.

Generally speaking, I get very good speeds over the network.  That percentage meter (Alt Ctrl Del) seems very accurate and, lol, it is averaging 87-89% very solidly.  That's great!

[johnnyboy]

U got one hell of a home network!!
Keep us updated

[benn600]

Now I have a broader question.  Before this NAS, I was using a Gateway as my server with an old box for IPCop.  Now, I have taken a powerful machine for the server but the Gateway I replaced is very similar in performance so I'm not terribly concerned.  Worst case, I can put server NAS (my code name) where server OLD is and then use server OLD as a workstation.  The problem is that my top level switch is a floor up--but only one "run" apart.  At the upper floor, I have to put the gigabit switch.  Well, obviously I need the server directly connected to that.  I can either run a second cable from upper floor to basement for the server (one now goes to a 100mb switch and splits to server + IPCop).

Otherwise, I could keep server NAS on upper floor by the new switch, leave it running 24/7, and actually consider using it as a workstation.  I have an older computer at this location now so it would be a great upgrade but my concern is if running it as a server and light workstation is a good idea.  I don't have any concern about capabilities but most of all, I am concerned about the issue of switch users...basically, administrator will always have to be logged on and users will have to remember to log off vs. turn off.

Is there a program I can install that will disallow shutdown without a password?  A warning (like other users are already logged on) is useful but doesn't necessarily help...people (including myself) may just click ok.  I need something that will make me stop and think and not let others shut down.  I'm not worried about holding the power button.  They won't be that ignorant.

I need administrator running mainly for programs which are not capable of running as a service.  Mainly, that is MC UPnP & Library server & Juice Podcast Receiver.  There may be others, though.  The reason I wouldn't request a service option is because it never fails--I will run into other programs I want to run that aren't service capable so trying to get everything to run as a service is a lost cause.  I know there are service emulation programs but I'm not interested in messing with those.  I think the switch user model works fine but also, upon startup, I know I can have an autologon but is there a way to autolog off immediately?  I bet a batch script in the startup folder would work....but details?  It would need to switch users, though...and ideally add a bit of a delay to let everything startup and calm.

I will have this on a UPS.  I'll find my biggest UPS.  Now that I don't have 5 USB hard drive power supplies, it will be nice because one cable is all I'll need.  That gigabit switch I purchased is only $40 after rebate!  Talk about bargain.  Most 10/100 switches aren't much less.  It's D-Link brand and has 8 ports.  I currently use 6 ports if I'm not mistaken so I must have 8...but beyond that, I have lots of switches with a few free ports each.  This is just the top level switch which connects our various floors.

We ran cable many years back and just kept adding short runs here and there.  We bought 1000 feet for around $55 many years back.

[johnnyboy]

http://www.softheap.com/how-to/disable_the_shut.html


For your usage I'd say its your call - no hard and fast rules.
Anyone using the machine is going to effect overall system stability in the long run, they will also effect performance while they are using it and there is the possibility of windows getting corrupted, your hacked RAID5 getting messed up and all kinds of issues. Leaving it stand alone as a dedicated file server means it should keep going forever.

Ideally I'd say dont use it for anything else, real life where things like it costs money for another desktop to use for other people I'd say use it. In practical terms, if you are going to let people use it I'd make sure you are very fussy with who you do let use it and what they use it for.

http://www.softheap.com/how-to/disable_the_shut.html

Think thats what you want.

[benn600]

It's a tough issue but I definitely need to make a decision.  I really would like keeping everything separate but then again, would like to utilize this computer.  I checked the server and during a write, the CPU usage hardly even goes up.  I noticed it stays well under 20% pretty much all the time--including when I'm copying or reading files.

[johnnyboy]

Alternate solution you could do which would give you some more security at price of some performance...
Setup VMWare (their player version is free) on the machine and make anyone using the machine have to run Windows (or Linux or whatever) inside of the VM image. If itc crashes or corrupts the system or whatever it makes no diff as its not your server they're crashing/corrupting but rather just the VM Image.

Best of both worlds really.

[benn600]

That's not a bad idea.  Remember that Microsoft Virtual PC is now free.  I have tried that a few times and it seems to work well.  That isn't a bad idea.  The only drawback is that it would really suck the resources, especially ram.  My family is reasonably secure and problem free even with no security software (no anti-virus, spyware detector, firewall, etc).  I do run IPCop at the modem but that's about it.  We all run Firefox, use Thunderbird, and other than me, my family is not very demanding.  We don't download any illegal music or movies nor do we have any of that junk software installed.  All these things really help keep our computers lean and fast.

As an update on my array, I'm here now:

Used Space: 532 GB
Free Space: 3.57 TB
Capacity: 4.09 TB

It's definitely getting filled.  I'm still well under 1/4 full but I also have a huge pile of DVDs I have to rip.  I'm currently ripping and will continue for the next few days to hopefully finish.  I'm scared for the final numbers.  I'm estimating around 2.1 TB used in the end which will leave 2 TB free...just over half used.

I'm amazed how cool the computer NAS runs.  It has a big exhaust fan and power supply fan.  The big exhaust got pretty hot when I used to use the computer for DVD encoding but it seems quite cool and the power supply isn't bad either.  I love how people always recommend huge power supplies for projects like this.  I had all 11 drives with a 600 MHz computer running great on a 300 W power supply yet everyone always goes crazy and wants to recommend 500-600.  The computer I'm using now does have a 400 W supply but it also has a very power hungry Prescott P4.

[johnnyboy]

VMWare's player is free also

[benn600]

Here's the new image.  I should point out that I am running Vista (on my 2nd day today) and finally was able to enable Aero Glass.  I will try to post some pictures of the NAS later on since it looks pretty neat.  Notice the filename: tb.jpg.  I am disappointed the graph looks just about the same as it did in XP.  I figured they'd pick new colors--more modern colors and enhance the graph a bit.

[ThoBar]

If you're interested in the VM route, look at VMWare Server. It's completely free (Just requires registration), can be run on Win and Linux, and has the advantage of being able to start VMs at boot time, under an account you specify (or Local System). VMs can also be accessed via the VMware Server Console accross the network. It's what I currently use and recommend for gateway boxes / special purpose machines where there are hardware limitations... VMware also has a range of "appliance" machines available for free download from their website.

Cheers,
C.

[benn600]

Wow.  That graph looks a LOT different now.  I have finished ripping everything that I can think of.

1) Stats:
Used Space: 2.34 TB
Free Space: 1.75 TB
Capacity: 4.09 TB

2) Defragment
I am running the Windows defrag utility and it gave this error on the details bar:
System resources are too low to display the graphic.
lol.  I guess it can't be scaled down to a few hundred KB's.

3) Scan for errors.
I'd like to scan deeply with the windows scanner.  If I check both boxes, it says it needs to scan on reboot.  Does this pose any risk?  I'm worried because perhaps as Windows is booting up, it may not recognize it is a RAID5 array (hacked RAID5 at that!) and it might think the whole drive is garbage and erase it or "fix it" by erasing everything.  I don't think it's worth the risk.  Any ideas?  Only reason I would want to do it is because I had around 2 hard reboots (one while writing).  Both seem to have been a stupid mistake and they shouldn't happen again--it's on a nice 725 VA battery backup which should keep it up for a reasonable amount of time--but I don't even want to unplug it to test it...it's not worth that, lol.