Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[benn600]

I really need to upgrade to 2GB.  I have a Core2 Duo and it's quite fast, but I see the hard drive accessing data ALL the time.  I multitask a lot so I definitely need more RAM.  I'd like to buy some more...that will be my next computer components purchase.

[InflatableMouse]

I checked and my slow media drive is not fragmented. So it must be the fact that it only has 13% free space remaing.

-How big is the disk?
-What size are its clusters?
-What type of disk is it?
-If its a regular ATA disk, is there a block device (CDROM/DVD or Tape drive) attached to the same cable?
-Again if its a regular ATA disk, is the disk jumpered for master/slave operation or on Cable Select?

Just think about this:
Fat16 (MS-DOS) had a maximum of 2gb partitions. Why? Because it could allocate 65,536 clusters with a maximum size of 32,768 bytes.

The default cluster size for NTFS is 4,096 bytes resulting in a max partition of 17Tb. Therefore, for any harddisk we can buy Windows XP will format it on 4,096 byte clusters unless you change it. (NTFS can allocate a maximum of 4,294,967,296 clusters with a size of 65,536 bytes resulting in a theoretical maximum partition size of 281Tb )

So, your 250Gb harddisk will format with 4,096 byte clusters, and it will have 65,536,000 of them. This is 100 times more than the MS-DOS disk, and it will use 100 times more memory. it won't be 100 times slower but I promise you, it IS noticable. As a rule of thumb, I try to stay below 10 million clusters on a partition, so the 250gb disk will be formatted with 32Kb clusters. Anything larger will be formatted with 64kb clusters. From there, I start splitting disks up in multiple partitions to keep the number of clusters down.

It is true that a larger cluster size will result in more diskspace waisted. Depending on your average filesize, this may result in a few gigabytes of slack space. Personally I'll gladly sacrifice a few gig for the gained speed.

[benn600]

Wait...so for a 4.5 TB partition, you'd recommend formatting with a cluster size other than 4 KB?  I think when I formatted the drive I just left it at default.  If you use a large cluster size, such as 64 KB, does that potentially speed access up?  Does it waste 63 KB if you write a 1 KB file?

[InflatableMouse]

Wait...so for a 4.5 TB partition, you'd recommend formatting with a cluster size other than 4 KB?  I think when I formatted the drive I just left it at default.  If you use a large cluster size, such as 64 KB, does that potentially speed access up?  Does it waste 63 KB if you write a 1 KB file?

I would NEVER use a 4,5tb partition, unless I have files so friggin' large it wouldn't fit otherwise.

What I would do with that space is create between 500-1000Gb partitions (depending on its use) with 64Kb clusters and use a mount points much like the way unix systems work. But that's me, I'm a crazy dude that believes in conspiracy theories

I guess this doesn't really work in your situation because from what I remember you are using striped USB disks? I wouldnt think cluster size is going to matter in speed in that situation but if its slow for you, it might be worth a shot.

And yes, with 64Kb clusters you waste 63Kb if you write a 1Kb file. But who cares with that much space anyway?   

[rjm]

I have a very simple configuration. It is a 750GB SATA drive with one partition formatted as NTFS using the default cluster size.

[hit_ny]

So, your 250Gb harddisk will format with 4,096 byte clusters, and it will have 65,536,000 of them. This is 100 times more than the MS-DOS disk, and it will use 100 times more memory. it won't be 100 times slower but I promise you, it IS noticable. As a rule of thumb, I try to stay below 10 million clusters on a partition, so the 250gb disk will be formatted with 32Kb clusters. Anything larger will be formatted with 64kb clusters. From there, I start splitting disks up in multiple partitions to keep the number of clusters down.

That's interesting, i tried testing some time back the 64k-cluster-is-faster-than-4k cluster with IoZone and my results were inconclusive in terms of read speeds and write speeds.

Did i miss any other important variable(s) ?

Wanted something more than it feels faster, could not find any supporting docs on the net arguing in favour of 64k clusters vs. any other size.

Not knocking your explanatoin at all, just was not able to prove it.

I have 3 x250GB for media, spread out on different partitions varying in size from 40GB  to the biggest which is 232GB (most you get from a 250GB HD).  I tested using Iozone on the smallest to the biggest partition and the differences in read/write speed was minimal between 64k and 4k clusters for each partition.

The theory at the time was it would take fewer operations to read/write using bigger clusters,  therefore faster. But i found that XP's caching made this a moot point.

The biggest notceable difference affecting read/write speed is de-fragmentation, if that's done on a regular basis, all is good.

[benn600]

I am not using USB discs anymore.  They are SATA.  I setup RAID5 because I absolutely MUST have redundancy and I want everything on one drive so my space usage can shift between data types, like music/video, without having to remake the partitions.

I'd say the quality is great.

[runemail]

I am not using USB discs anymore.  They are SATA.  I setup RAID5 because I absolutely MUST have redundancy and I want everything on one drive so my space usage can shift between data types, like music/video, without having to remake the partitions.

I'd say the quality is great.

what controller are you using? and how many disks is that 4.5TB array?

impressive

[benn600]

It's 10 WD USB My Book's that I opened up and took the SATA drives out of.  They are each 500GB.  It is great having so much space but with all my DVDs, I'm down to 1.6 TB free!  That's hard to believe.

Opening the My Book's was tough but I got it done.  I have 2 4-port SATA cards and my motherboard has four total, of which I use two ports.  That gets me to 10 ports with two more.  The case is currently holding 9 of the drives with the 10th and a boot drive simply laying in the case.  I know it's not the best solution but it works and I don't move the case around ever.  I also added a nice, large fan to blow directly over the top 4 drives (bottom already have a fan) and the 9th gets cooling simply by the pressure which pulls air in the opening.  The drives stay impressively cool.

I tried FreeNAS but canceled that because it used up around 400 GB even when it was freshly formatted.  That's a ton of space!  Plus, I've heard of reliability issues.  Then I tried various Linux distributions but nothing worked well for me.  In the end, since I still need a Windows server (to run MC UPnP server, and a few other Windows only programs) I decided to go with Windows XP "hacked" RAID5.  It is so impressive I would recommend it highly!  I have a 700 GB backup server and an 11th 500 GB My Book so I have two BACKUP copies of my most important data + lots of it burnt to DVD, too.  So I have plenty of backups of the most important data.  DVD Video and podcasts are the only things I don't keep backed up because I can always rip them again easily.  My 300 GB of FLAC is backed up, though!  That preserves my ratings and all the work ripping 1 thousand CDs.

Yesterday we had a terrible ice storm.  (Iowa)
At around 6 PM, the power went out!  Oh great!  Luckily, I had this server on a UPS.  I actually have 5 of them...3 of which are 600 VA or greater.  So, I calmly went to the server and worked on it for a few minutes.  The monitor is on a separate UPS.  Anyway, I was able to safely shut it down without potentially corrupting the drives.

I have the other smaller battery backups around the house.  This is a good idea for anyone who has power outages:
1) I put a 350 VA UPS (plenty of power) on our main cordless phone so we didn't have to use our wired phone.  This is a GREAT idea.  With lightning, I didn't want to use a corded phone anyway but it was also much more convenient using our cordless.
2) Another was at our main IPCop & modem...this let me use my laptop with internet.
3) This was hooked to our wireless AP on a different floor.

This covers the internet but I did have some issues.  The issue is that IPCop runs on a big computer and takes a lot of power.  I need a convenient way to switch to the plain old router so I don't need IPCop and therefore, can run a lot longer.  A modem and router take so little power!  That UPS also happens to be a 725 VA (quite large).

In the end, I'd love to get a 1500 VA or larger, but since these outages are rare, I think this works fine.  The first UPS we got was $5 (Black Friday) and the second was $20 (Black Friday).  The third was a good price too, but then the last two were closer to normal price.

They work great in situations like this!  80,000 homes were without power because of this storm.

[KingSparta]

My first backup system i got about 16 years ago after living in korea for a year, you just got to have it there. the power goes off maybe 2 to like 4 times each day in korea when i was there.

Currently I have Two APC RS 1500 Power Backup Systems, And Both Have The Axillary RS/XS External Battery Packs.

The Load Is Split Between The Two Battery backup Systems.

I Can Basically Run My Whole System Including My TV\Tivo\DVD Players For Over 4 Hours.

If i turn off the extra stuff it will run a bit longer.

I have 360 Gigs in the computer case, 100 gigs is raid

I also have 4-each 300 gig One touch firewire\usb drives

2-each 1TB one touch III turbo drives Firewire 800

1-each 1.5 TB one touch III turbo drive Firewire 800

I may get a new drive soon maybe 2TB

[benn600]

The reason I wanted all in one is because I want one drive.  It makes it much easier for me and everyone else around here.  It's nice only dealing with one network drive and then I keep a nice folder structure there for my data.

Here's a question for you.  I realized that my 725 VA APC UPS will only run my main server for 10-15 minutes.  So, I just added another UPS (650 VA) in the same path.  Basically, one is plugged into the second.  My theory is that if the power goes out, the first will power the second and the second will think the power is still on fine.  Then, when it dies, the second will kick in.  This will give me more time.  The best solution is to just get a bigger one, obviously.

Now what if you have a 350 VA or so and want to power something that it can't power alone?  There really isn't a way to add their power is there?  If I put two in line, the first would die immediately and the second couldn't power it either.  So they would stop providing power (because they can't) but they would still be full of power.

[KingSparta]

i did that once thinking the same thing, the problem was i was trying to pull too much from one of the backup systems and the overload protection kicked in and shut the whole system down.

So in other words it did not work for me, but in theory it seemed OK

1\2 of the drives are used to backup the other drives, so everything is backed up, including the user database for my ftp server.

I have about 400,000+ mp3's

330,000 of them being radio shows

[benn600]

I figured there wasn't a solution to using smaller UPS devices.  But that's not a problem because the smaller ones are great for the DSL modem or cordless phone. 

350 APC
350 APC
650 APC
725 APC
750 Other Brand

For a while, I had one of these in our theater and that was great.  I unplugged it once and it ran for 15 minutes before I plugged it back in.  I didn't realize at the time that they beep constantly when they are almost depleted or I would have left it running longer.  I was actually in our theater when the power went out.  It was tough getting upstairs because it was pitch black...and I was in a hurry to get to the server and check on it.

[hit_ny]

I Can Basically Run My Whole System Including My TV\Tivo\DVD Players For Over 4 Hours.

APC don't rrecommend you put a TV on thier RS series, saying an inverter would be better and also lot cheaper than the more powerful APC UPS's that do shaping.

Did  this work out, for you ?
I suppose you are using a digiital one LCD maybe instead of a CRT tube. Tho i cant imagine why this would be any different than a CRT monitor. I think it has to do with the lack of shaping on the wave, The RS series outputs a square wave only instead of a sinusoid. A PSU in a PC can deal with this better than the one in a TV.

a TV draws a lot of power, i suppose with the aux batteries it might work for say an hr or so.

Also the don't recommend daisy chaining models in series. The way to do it is to get an aux battery, but thats only available from models RS-1000 & up. So before purchasing one i checked this and got the RS-1000.

[benn600]

That's good info.  I am actually running a NEC Projector, Sony subwoofer, Yamaha 7.1 amplifier, and the usual stuff (DVD, VCR, CD, gaming systems).

I would like to return one of the big UPS devices to the theater but having two on the server is appealing.  I need a way to have the server shut itself down automatically if the power goes out.  The problem is that I know it has software and hooks up via USB but I leave it on switch users with "Serve" logged on.  So, only I can logon to Serve, logoff, and then shut down safely.  I need the auto shut down software to logoff Serve before shutting down.  Maybe it does but it seems unlikely.  This is important, obviously, if we are gone or even if myself alone is gone.

[hit_ny]

Can't you call a batch file that does all the graceful closing before it powers down.

[johnnyboy]

With regards to the original point of cluster sizes, these days the larger the size you can use the better off you are.
Sure if you have a 1kb file you will loose most of the extra space, but considering hard drives of these sizes are mainly used to backup large files (smallest usually being say a 2Mb mp3 and on average probably being 5Mb+ files or even several Gb).

With smaller cluster sizes you are wasting more space telling the system where the next part of the file is on large files.
Imagine a 100Mb file for instance.

With a cluster size of 50kb for instance (using easy numbers for illustration purposes), to store that file would require 100x20 clusters to store it. That's 2000 clusters to store your 100Mb file.
If you use cluster sizes of 4kb though your changing that to 100x250 clusters = 25,000 clusters for that same file.

Now depending on your file system being used it will control and manage these clusters and keep information about which is where in different ways. Regardless of which it uses however, whether a big FAT table at the beginning of the drive, or linked list so that each cluster tells where its next part is or a combination of these methods (which is what I think NTFS does) storing all this additional information about where the extra 23,000 clusters are is going to waste space.


Now sure you might have some tiny files that would result in the extra cluster space being wasted, but the real question is how much space is being wasted in all that additional information about all the clusters.

Now with your Linux partition you had a large initial amount of lost space, the question is though would it have saved more space overall by then not wasting any more space after this initial loss. With your windows setup, it initially showed less lost space but is it then going to be wasting more as you are using it.

There is a program called 'Treesize Pro' I use alot. It shows you folder sizes and is a great way to see wasted space. It also has an extra feature however that will let it show you real file sizes / used file space. Its a great tool to show you wasted space by the difference in file sizes and used file space.

I'd recommend running it on your Windows file server to give yourself a heart attack when you realise just how much wasted space you have.


File systems are an extremely complex thing once you start moving into large scale storage systems and choosing the right setup isn't an easy task at all

[benn600]

Well, then it probably got set to the default when I formatted it and yep, I bet it's wasting a ton of space.  I wish I would have investigated that a little more before formatting the array.  At this point, there isn't any issue because I'm not out of space but that's too bad.  What I really want is a place I can borrow a 4 TB array from for a week at a reasonable price.  Then, this summer when I have time, I could reformat the array and reload all my data.  I don't want to re-rip 300 DVDs just to reformat the array.

I understand that description on clusters.  Now here's a good question.  What is the default cluster size?  Is it 4 KB for any size drive?  Or does it dynamically adjust based on drive size?  Perhaps it chose 64 KB by default?  Otherwise, what would be the space wasted for a single 1 GB file?  Each DVD has 3-6 of those files and I have 300 DVDs...plus 300 GB of FLAC music, each file around 30 MB!  I have very few MP3s...effectively zero MP3s other than podcasts.

[hit_ny]

The default size for NTFS is 4k, unless you use another disk mgmt program like partition magic where you can set this or you specify explicitly using the /a option (allocation size or cluster size) with the default format cmd.

So unless you did this, i'd say its unlikely any other cluster size will be selected.

Windows filesystems are setup for general purpose files, not media ones. The bigger cluster size is better for just media only drives is something that needs to be clarified here. I have not noticed any speed increases using larger clusters as mentioned above.

johnnyboy is saying the space lost by using bigger clusters is compensated by a decrease in the MFT (Master File Table) file which NTFS uses to keep track of what is where on the HD. Does Treesize Pro indicate the size of the MFT file ?

Using chkdsk <drive-letter:>and the details from here on my biggest partition.

 244,187,968 KB total disk space.
 223,066,768 KB in 25238 files.
        13,792 KB in 1885 indexes.            <--- Space used by NTFS indexes
      100,444 KB in use by the system.    <--- Includes MFT and other NTFS metafiles.
        65,536 KB occupied by the log file. <--- NTFS Log file
   21,006,964 KB available on disk.

           4,096 bytes in each allocation unit.
    61,046,992 total allocation units on disk.
      5,251,741 allocation units available on disk.

i gather my MFT+indexes is approx 184MB

Now if a larger cluster size was used, using Partition Magic's Resize Clusters option shows if the cluster size was changed for the current data on that 232GB partition using

8k  cluster size       105.4 MB is wasted
16k  "                    210.7 MB  "  "
32k  "                    421.4 MB  "  "
64k  "                    842.8 MB

So if i used 64k clusters, then i would waste 842.8 MB, now presumably the MFT will come down since there are fewer clusters to manage. But by how much ? the number of clusters to manage has decreased to 1/16, becomes about 12MB for MFT+indexes

but i would still waste 842.MB. The decrease in MFT size is a lot less than what would be wasted going to a larger cluster size.

So No, i don't think more space is wasted using a smaller cluster size. (In my case anyway).

What i'd like  to hear is whether a larger cluster size can increase performance.

In theory it should but i'm waiting to see it shown with figures.

[InflatableMouse]

iozone is a synthetic benchmark. It serves its purpose but I don't think its purpose is real life performance benchmarking.

The increase in read/write speed with larger clusters is not easy to benchmark and I admit that it depends on the situation. It's been a while since I've tested stuff like this but I believe SiSoft Sandra showed an increase of around 10% on a large partition in favor of the 64kb clusters. Having said that, Sandra is also a synthetic benchmark and what I do as well is do a couple of copy/paste actions, zip and unzip a bunch of files and do this with both a lot of small files and 1 or 2 really big files. The last time I remember testing this stuff was with Windows 2000. I don't think I've done it with XP or 2003.

[benn600]

Remember, even if you don't lose a ton of space, there is bound to be some space lost in the 64 KB.  I know it's going to be small, but it could be up to ~63 KB.  At this point, I'm not really worried.  I always wondered where space was taken for the index and file tables.  This makes sense, though, to take it as space is used.  Speed is great, at least from what I can see.  Remember, with 10 drives, even though it's RAID 5, aren't reads able to be sped up quite a bit?  This is probably more sequential reads, though.  There isn't a computer here that even has gigabit access to the server.  We all just have 100 Mb but the top level switch which branches out to different floors or sections is gigabit so each branch gets its own 100 Mb.  This has really helped before.  It especially helps if someone is watching a DVD in the kitchen, which can quickly saturate a 100 Mb connection.  The other branches are still able to get lots of yummy data.

[hit_ny]

iozone is a synthetic benchmark. It serves its purpose but I don't think its purpose is real life performance benchmarking.

what I do as well is do a couple of copy/paste actions, zip and unzip a bunch of files and do this with both a lot of small files and 1 or 2 really big files.

Iozone writes files of sizes that can be configured, so you can tell it to read/write files of 1kB, 10kB, 100kB, 1MB, and so on. I think i went as far as about 10MB as that would be the typical average size of files on the HD. That's not to say you can't keep going till you reach files that are 1GB in size.

I was hoping to see an increase in read speeds, since in theory fewer reads would be required since its pulling much more with a 64kB cluster than a 4kB one, but the speeds were very close. My drives are regularly  defragmented, so time spent to read is minimal even if its 16 times more. Maybe at this point i can say, increasing cluster size does not give a noticeable/measurable result when files are avg size 10MB.  You can get this by running in a cmd prompt

defrag <volume-name> -a -v

-a is analysis only and v gives a verbose output.

Analysis Report

    Volume size                         = 233 GB
    Cluster size                        = 4 KB
    Used space                          = 213 GB
    Free space                          = 19.90 GB
    Percent free space                  = 8 %

File fragmentation
    Total files                             = 25,081
    Average file size                   = 9 MB
    Total fragmented files             = 290
    Total excess fragments           = 1,818
    Average fragments per file       = 1.07
...

it also includes MFT size in the report and must admit i'm bit confused about the difference in MFT size reported here, as compared to chkdsk above

Master File Table (MFT) fragmentation
    Total MFT size                      = 26 MB
    MFT record count                   = 26,980
    Percent MFT in use                 = 99
    Total MFT fragments               = 2

[InflatableMouse]

hit_ny, I suggest you include an Explorer copy/paste action in your benchmark tests. Just time it in seconds. In real life we don't care about tenths or less. It has to be noticable, not only measurable.

[hit_ny]

I think having lots of RAM may favour bigger clusters. You are reading in larger amounts, it gets put in RAM. I only have 256MB on mine so that reduces the difference between small or large clusters. It might actually slow things down as other progs will have to wait.

Reading in various forums, many ppl seem to think its a  given that you use bigger clusters with media only partitions for all the reasons we discussed above.  Proving it is another matter

[benn600]

Yep.  Is there an easy way to check a disk and see what cluster size it's using?  64 KB clusters would've been nice.

[hit_ny]

Read above about chkdsk..and look for allocation unit in the report, its the same as cluster.

[Listener]

I think that several different concepts are being mixed up in this discussion. 

Disk space is allocated to a file in clusters.  If the cluster size is 4K bytes, then a 1 byte file would use 1 cluster of 4K size.  A larger cluster size will result in more wasted space.  The data structures that describe a file's location on disk have to specify each cluster.  A larger cluster size means lewer clusters for a given file size and thus less space for this cluster information.  This description is originally on disk but when a file is opened, Windows reads some or all of it.  If things go smoothly, this descriptive information is kept in the I/O cache while the file is open.  If not, reading data from a file may require several overhead reads.

The cluster size does not directly affect the size of the disk I/O transfers used to read and write a file.  A single sector is usually 512 bytes for regular hard disks but Windows will do various things to allow for much larger transfers whenever possible.  In addition, the I/O device and Windows may be reading ahead and writing behind and cacheing data to produce fewer, larger transfers than you would expect from looking at the I/O requests by an application program.

Bill

[hit_ny]

If things go smoothly, this descriptive information is kept in the I/O cache while the file is open.  If not, reading data from a file may require several overhead reads.

The cluster size does not directly affect the size of the disk I/O transfers used to read and write a file.

 
Agreed, might explain why Iozone showed no significant difference between the 2 cluster sizes.

A single sector is usually 512 bytes for regular hard disks but Windows will do various things to allow for much larger transfers whenever possible.  In addition, the I/O device and Windows may be reading ahead and writing behind and cacheing data to produce fewer, larger transfers than you would expect from looking at the I/O requests by an application program.

All these behind-the-scenes events happening in concert make it quite tricky to say whether going with bigger clusters is necessarily better.

[benn600]

lol.  Yes, they negate the differences between cluster sizes because Windows does extra work with caching, etc.

[Listener]

I really need to upgrade to 2GB.  I have a Core2 Duo and it's quite fast, but I see the hard drive accessing data ALL the time.  I multitask a lot so I definitely need more RAM.  I'd like to buy some more...that will be my next computer components purchase.

Back to your original post.  Adding more RAM is a simple way to improve performance until you get enough.  Windows is heavily dependent on cacheing file contents, its own disk-based data structures and virtual memory.  How much is enough?  For XP, my goal is  2GB now.  For Vista, I'd buy more than that.  Of course, I'm not buying Vista yet.

Each release of Windows expects to use more RAM.  And over time, applications grow in size and expect to use more memory.  Right now, it is fashionable to base applications on garbage collection and application level caches rather than explicitly freed memory and to defer collection until huge amounts of memory have been used.  If you are running several memory hogs at the same time, switching from one application to another can produce a storm of disk I/O as the contents of memory and the cache change.

Bill Hunt

[newsposter]

When 'adding ram' you've got to test out the hardware thoroughly.

Bad ram in a server (both Windows and Linux) is probably the primary cause of corrupted files and filesystems.

Take a day or three and run the very good, self-booting from CD, and FREE tests from http://www.memtest.org/.  Once you are damned sure that your hardware is up to snuff, then add your hard drives, build your mirrors, and rip away......

[benn600]

Good suggestion for everyone else.  I'm already knee deep in the whole array thing.  I don't have anywhere to just temporarily store my 3TB of data.  Holy goodness...I really have close to 3 TB of data.  That seems HUGE.  It's actually a little under that but whatever!

Good point on the RAM issue.  Luckily, I haven't found any corrupted files at all yet.

One big issue I'm having...please help me solve this.  Copying files to the server form my new Vista box is about 1/4 as fast as it was on XP.  I'm not sure what the difference is but the only thing changed here is the OS -- moved from XP Pro to Home Premium Vista.  Are there updates or drivers?  I'm using onboard gigabit network with a new Core2 Duo motherboard...it's a lower end micro ATX one.  I'm thinking it's Gigabyte brand (maybe not).  I can't stand these slow speeds!  Reading, which I haven't really tested too much or noticed slowdowns on, I think is reasonable but I havne't checked.  I'll need to check that.