Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Hilton]

Hi all it's been a while!

@nathan if your around, interested in your thoughts.

Most of the drives in my storage pool are 5+ years old and 2 have started accumulating bad sectors.
As I'm using stablebit drivepool and stablebit scanner with the scanner set to evacuate drives on detection of an error and I always have more capacity spare to cater to my largest 8TB drives if errors are detected. 

I've not lost any data, (fingers crossed) but I was wondering what others do to maintain the long-term health of their storage pools?
Do you recycle and test drives or just replace them?
Do you just replace all drives after 5 years? thats getting pretty expensive eh!
So how do you manage your data when all your drives are getting on a bit? (apart from backups - which i do for my entire music collection and key other data)

This is what I'm currently doing to try get the maximum life out of my drives, though I realise pushing more than 6-7 years on a drive I'm probably asking for problems, but budgets are finite...

Before everyone comments and says you should just replace the drive, yes that's an option, but when you have many 8TB drives, even replacing with 16TB or 20TB drives gets pretty expensive very quickly and drives often have weak areas that fail and can then happily operate for many years more once the areas are remapped.

My thoughts and process...

1. A clean up and a cull.. yep it's hard enough to a clean up and de-clutter in the real world, little lone your digital life.. but I recently did a digital clean-up and deleted a bunch of stuff.. yes old files, old downloads, programs and multimedia stuff, older generations of media that im never likely to use again..  quite liberating actually!  Amazing how much digital junk you can accumulate! A few TB in my case...

2. Keep a drive monitoring program running periodically reading all sectors on every hard drive to force ECC error correction and reallocation of failing sectors...

3. When I get a sector allocation error, the data gets automatically evacuated to other drives in the pool.. 

4. I then remove that drive from the enclosure and plug it into a separate  enclosure for testing on a different computer.  I've seen a failing drive cause weird controller behaviour that can cause other filesystem and drive issues due to excessive time outs and retries, so better to isolate the failing drive ASAP.

5. I check out the SMART logs obviously, and if it's 100s of errors it's probably time to destroy the drive and toss it..

6. If not a high number, I'll run Victoria HDD tool to write to every sector and make sure it forces a sector reallocation.  The reason I like this free tool is that it gives very granular control and detailed logging and tracing of what's happening when every sector is written so you can make a call on what to do with the drive with a little more information than just a SMART log and full format. Just formatting doesn't really tell you much about the drive and if it's really failing or has other controller or mechanical issues.

7. If the drive doesn't have too many long retries and has no errors, I can look at getting it ready to back into the pool. (for me warnings are ok if the retries are under 5 seconds to allow a sector remap and controller reset during an attempted write)

8. So the drive successfully completes the full sector write test? Time to delete the partition and reformat (slow full format)  Check the smart logs and as long a no big increase in sector reallocation, and no pending allocations, and it completes in a reasonable time (24hours for these big 8TB drives) time to then put it back in the pool and put it under some real-world load.

9. Because I'm using stablebit drivepool, it has a bunch of cool balancer plugins and can force duplication and file placement. So I can force some duplication and file placement onto the drive now it's back in the pool.

10. If the drive handles the realworld use for a week or so then I can safely put it back into the pool, If the scanner picks up more additional bad sectors at this stage it's time to replace it.


So what are others doing to manage long term data and drive health?

SMART by Hilton, on Flickr

Victoria-write-test by Hilton, on Flickr

stablebit-fileplacement by Hilton, on Flickr

stablebit by Hilton, on Flickr

[jmone]

Hi Hilton, been awhile!  I've found over the decades that HDD have got much more reliable and that it was particular models that failed vs others that never have issues.  I have two pools.  My MainPC has 8 HDD bays and as this pool files up I buy the latest good value for money HDD at the time (say 18tb for A$500) and swap out the smallest drive.  This drive I then flow down to my backup pool (16 drive rack chassis) and if needed then retire the oldest drive from that with can be up to 10years old by then and now relatively small in size.  I many syc my backing pool using freefilesync.  This keeps the cost of the pools reasonable and give proper backup.  I don't use on pool replication for media files as it just waste space in this scenario.  If  drive was to fail out of the blue I'd just replace it and syc the missing content from the other pool.  If a drive started to throw errors I'd try to remove it and let drivepool do its thing.

[Hilton]

Hi Hilton, been awhile!  I've found over the decades that HDD have got much more reliable and that it was particular models that failed vs others that never have issues.  I have two pools.  My MainPC has 8 HDD bays and as this pool files up I buy the latest good value for money HDD at the time (say 18tb for A$500) and swap out the smallest drive.  This drive I then flow down to my backup pool (16 drive rack chassis) and if needed then retire the oldest drive from that with can be up to 10years old by then and now relatively small in size.  I many syc my backing pool using freefilesync.  This keeps the cost of the pools reasonable and give proper backup.  I don't use on pool replication for media files as it just waste space in this scenario.  If  drive was to fail out of the blue I'd just replace it and syc the missing content from the other pool.  If a drive started to throw errors I'd try to remove it and let drivepool do its thing.

Thanks mate appreciate another perspective.. So how old are your oldest drives and do you worry about them at all?  I guess you're reasonably well covered with your backups...

[Hendrik]

Personally, I replace any drive that starts collecting bad sectors - especially if its more then just one random sector or such. It started failing, and there is no reason to assume it won't continue to do so.

I had a goal of replacing drives every 5~7 years, but since its been running stable I let that slide for a bit. Although I did an upgrade pass in 2021 and now my oldest drive is from 2017, but I have no immediate plans to tackle those. It wasn't so bad when i could rip out 9 old drives and replace them with 3 new ones without capacity loss as I went from 3-4TB drives to 16TB drives (and my smallest ones are now 8TB)

I also use DrivePool, but on top of that I also use SnapRaid to give me redundancy in case of drive failure, because fully restoring data from other sources would take a while (I don't currently have room to keep a full on-site backup, so I prefer to avoid having to do a restore from off-site).
The other advantage of SnapRaid is that it actually can ensure data integrity. Its setup to scan 3% of my data every night (so about the full data once a month) and compare with stored checksums, so if any bitflips happen it can restore those, and it helps to ensure I catch drive failures.

(And SnapRaid works on top of the file system, so other then the disk(s) wasted for parity, it does not otherwise inhibit free use of the data)

[jmone]

I'm away from home so off the top of my head my oldest drives may be 4tb that have to be getting close to 10years old.  As Hendrik said, I'd ditch any that started reporting declinging health in Hard Disk Sentinel.  Also, I have all my data drives formatted as ReFS with Integratory Stream enabled.  Love it.  No more CHKDSK.

[jmone]

The practice of having a scrubber (like with ReFS with integrity steams on) or how Hendrik is doing it is a good idea to prevent bit rot as reading the data every month has the HDD auto rewrite and weak sectors.  I've never had a single validation issue in the 10 odd years with ReFS and if I did I would either delete the file or remove the drive and pull any missing content from the backup.

[Hilton]

The practice of having a scrubber (like with ReFS with integrity steams on) or how Hendrik is doing it is a good idea to prevent bit rot as reading the data every month has the HDD auto rewrite and weak sectors.  I've never had a single validation issue in the 10 odd years with ReFS and if I did I would either delete the file or remove the drive and pull any missing content from the backup.

Yeah ReFS is probably a good idea now. One of my data and partition recovery tools actually supports it now too which is one of the things i was waiting for before changing over.  I guess while im on holidays for a few more days I can take the pool offline and convert and migrate data across my drives... it will take a while though!

[jmone]

Yeah it takes ages!  Also you need a version of Windows that supports it.  I even paid to upgrade to Windows for Worstations just to get ReFS creation support.  All Windows vers will read ReFS but only a few will let you create a ReFS volume.  To get all the right bits enabled you need to format from a command line.

[Hilton]

I have partition master that can format ReFS filesystem.

partition master-refs by Hilton, on Flickr

[jmone]

Nice.  I don't have my batch file on hand but it has the flags for sector size and turning on integrity streams.  Not sure how to do that once formatted.

[jmone]

FWIW found the thread on Drivepool on refs including various format options and discussion.  https://community.covecube.com/index.php?/topic/2944-refs-in-pool/page/2/

[Hilton]

Thanks have started the process.... I had to buy and apply a Workstation key to get ReFS to format.. done now. Didn't even have to reboot though..

In process of removing next drive from pool and then Ill use GS Richcopy 360 to move data across to the new ReFS drivepool and when that's full I'll remove the next drive... this might take a while..

[jmone]

Yup.

[S. Pupp]

I removed my storage from my HTPC and placed the drives in a TrueNAS box (made from my previous, obsolete HTPC).  I have each drive in a RAID 1 setup.  TrueNAS uses the zfs file system. I have it set to scrub once a month, and to run a long SMART test once a month. In 3 years, there has been a single bit flip reported and fixed automatically.  Although it hasn’t happened yet, I plan to remove any drive after the first bad sector.  My experience has been that, after a single bad sector, there is a a rapid acceleration in rate of accumulation of bad sectors.

[newsposter]

new drives are what, US$ 20-25 a terabyte?

Not really all that expensive compared to the time spent recovering from sudden failure *and* the time spent trying to 'unnaturally' extend the life of a hard drive.

Use ReFS or zfs, use raid 10, use a real-world NAS manager like TrueNas, set up your NAS manager with all of the self-tests and alarm/notifications it has (including smtp and/or SMS notifications), use a surge strip or (better) a small ups, config your nas to auto-shutdown if the ups is on battery for more than 5 minutes, and a few times a year dump your nas to "archive quality" hard drives which you then eject and store in a sealed box at the brother in laws house.

[Hilton]

Thanks for everyones input and ideas... Nearly Done! Only 4 hours to go on last drive copies...

Im also going to migrate the drives over to SAS 8088 enclosures and putting a 9200-16E dumb SAS/SATA HBA into my NUC to move away from USB3.1 (10G)

One enclosure will be for my main storage pool and the other for backup.
I dont like RAID any more for this type of data and dont need RAID performance.
Drives are also getting too big to risk multiple failures during RAID rebuilds.
I also prefer to have native OS access to my drives incase of issues with controllers or drives as it makes data recovery easier.

If i put the external USB enclosure under heavy load the drives sometimes dropped off and Ive had a couple drive go RAW with corrupt partition tables - thankfully I've been able to repair the partitions and not lose data but It's happened a couple times now so im paranoid. Native SATA should be more reliable and a little faster.

As cost effective as i could make it..  Just waiting for the bits to arrive.
I looked at going native Thunderbolt 3 but even the cheapest 8 bay TB3 enclosure was nearly double the cost of my 16 bay SATA HBA solution.

For storage bays -
Im using 2 x Silverstone DS380 8 bay cases and 500W PSUs with internal SATA to 8087 cables  - $774 from Scorptec (for both with cables and PSU)
78600_large by Hilton, on Flickr

78599_large by Hilton, on Flickr

2 x dual port 8087 internal to 8088 external PCI adaptors plus
4 x 1M 8088 to 8088 SAS cables.  - $191 from Amazon

61aaLpSUgxL._AC_SX679_ by Hilton, on Flickr


AND

LSI - SAS9200-16E 16-Port 6Gb/s PCI Express SATA+SAS HBA Controller H3-25140-02D for the NUC Extreme. -  $120 from Ebay

s-l1600 by Hilton, on Flickr

[Manfred]

I have 2 x 8 TB + 3 x 10 TB WD Red (all Video)+ 2 TB Samsung EVO SSD (audio, documents, images) + 4 TB, 5TB WD MyPassport (Backup). I use WD Dashboard + Samsung Magican to monitor drive health. WD Red drives are now 5.4 years old and simply work. BD, DVD are ~80 % of my total data volume and are also my local backup. Backup SW is Acronis.
I have a full DR copy of each disk on a different side.

[Manfred]

A question to Hilton:

Do you use your NUC exclusively for Media Center like SW or parallel also for Office like Word, e-mail, Shopping etc.?

And why do you move away from USB?

[Hilton]

Hi Manfred - Its a backup target for network backups and also copies of important files and copies of software installers.

I was using Acronis for many years but switched to something a bit simpler a little while ago.. EaseUS Todo Backup.

I used to run dedupe, incrementals and use lots of the Acronis features but my backups were getting way too big as a percentage of my actual data and I was having performance and reliability issues so just decided to go the simple route.

-----------------

Almost done!

Stablebit-ReFS by Hilton, on Flickr

This is why I like GS 360 Richcopy for data management..

gs360rc-jobs by Hilton, on Flickr

gs360rc-settings by Hilton, on Flickr

gs360rc-log by Hilton, on Flickr
 

[Manfred]

Ha  And I switched  away from EaseUS Todo Backup to Acronis.

[newsposter]

Veeam has several no-cost clients 'community edition' that work well.

And for 'precision' copying involving windows hosts and cifs/samba paths, Teracopy is very nice.

[tzr916]

I've had good luck with Hard Disk Sentinel and Paragon Hard Disk Manager/Backup & Restore.

[marko]

HD Sentinel and Macrium Reflect here, along with Goodsync for weekly copies of media files. Reflect has imaged the C: drive in the wee hours of every morning for years. That one time I needed it, when messing around with an MC beta, that caused, literally, everything, on my C: drive to be attempted to be moved to a different drive, the restore worked flawlessly.

I guess it matters not too much what you use, so long as there is a full backup plan in place. Even if you can't afford an 'off-site' option, everyone should still have something setup, somehow. Windows ships with a backup option, I think? Though I do not know if it's any good.

-marko

[jmone]

HD Sentinel as well.  Freefilesync for previewing adds/moves/changes/deletes before commits.  The backup server is WinServer2016 doing block level incremental backups of PCs main partitions that allows for both bare metal and file based restores.  I find the bare metal restores less than bullet proof however.

[Hilton]

Migration to ReFS completed... And no more drive health issues!

ReFS done by Hilton, on Flickr

No issues by Hilton, on Flickr


Now I just need the DS380 cases and power supplies to arrive with 2 more 8087>Sata cables.... thats a lot of cables compared with USB3.1... well it will be faster and more reliable!
SAS HBA by Hilton, on Flickr

[JimH]

Congratulations!  Thanks for sharing.

[eve]

Veeam has several no-cost clients 'community edition' that work well.

And for 'precision' copying involving windows hosts and cifs/samba paths, Teracopy is very nice.

100% this answer.

Veeam is my go to for Windows & Linux boot / OS backups. I have better tools for files but for imaging systems, Veeam is almost incomparable. It's really, really useful especially once you understand it in depth. The cross platform nature is nice, it's less to learn. Not ideal for non OS / Boot disks, there are far better methods to handle that. For like non massive project files, things like documents and code, I'm big on restic. (I also use it for save games). Borg backup is cool too.

Teracopy works wonders. I put it on all the windows production machines and 'force' it for copy operations. When you're sneakernetting / ingesting massive amounts of footage to and from external disks, the added peace of mind is nice.

I mostly just use Directory Opus's built in copy now on my personal workstation though.

[Hilton]

Finally got around to finishing the DS380 JBOD.  Had to replace a faulty LSI 9200 16E with a LSI 9201 16E before I could migrate across but all loaded up.

The 9200 just barely fit in the NUC but the 9201 had a little room to spare. (shorter card length by 15mm or so)


DS380 JBOD by Hilton, on Flickr


DS380 JBOD by Hilton, on Flickr


DS380 JBOD by Hilton, on Flickr


DS380 JBOD by Hilton, on Flickr


DS380 JBOD by Hilton, on Flickr


DS380 JBOD by Hilton, on Flickr


DS380 JBOD by Hilton, on Flickr


DS380 JBOD by Hilton, on Flickr


DS380 JBOD by Hilton, on Flickr


DS380 JBOD by Hilton, on Flickr


DS380 JBOD by Hilton, on Flickr


DS380 JBOD by Hilton, on Flickr

[zybex]

Thanks for the pics! I like that DS380, I may end up using it too to replace my QNAP NAS. I'd put a mini-ITX board in it though, and it may become the MC Server among other roles.

[Manfred]

How noisy is the DS380?
I have this one (Lian-Li with orginal Fan's replaced by Noctua Cooling) - Ethernet connected - It has Media Center + Server + Media Data + Document Archive. It's build along:
https://audiophilestyle.com/ca/ca-academy/computer-audiophile-pocket-server-c-a-p-s-v4-cortes/

[Hilton]

Dead silent which is why i wanted it instead of rack mount disk trays..

[Manfred]

@Hilton:

1. On the Nuc do you run only Media Center or also Office applications like e-mail, Libre Office etc.?
Background: In principle I could do the same, but I always been sceptical because of Security (DAS is seen by the OS).
2. If you shutdown the NUC is the DAS not powered down?
Thank you very much!

[jmone]

I've never found a better case for a MC Server than the Lian-Li Z70B.  The only downside of this 10 year old case is that it does not have front panel USB-C.  .... but
- 9 x 3.5" HDD backplane with easy remove slots
- 3 x 5.25" ODD bays
- 2 x 2.5" SDD slots
- fits under a desk to probs
- Black Anodized Alum with no bling
- easy to remove font panel to get to the HDD bays

I've got a SAS M1015 flashed to IT mode that drives the DrivePool.  The three ODD are for ripping BD/UHD BD etc.

Wish there was a modern equivalent (or a mod) with USB-C front panel.

[jmone]

My backup server is a bunch of hand me down and 2nd hand e-bay server bits (another M1015 flashed to IT mode and a SAS extender that I think came from a Compaq Server that I had to Zip Tie to the case as there is no mounting option for this board).  It's all in some nasty (aka cheap) 20 bay rack mount case with sharp edges.... but it does the job.

[jmone]

After my server rebuild, one of my (newer) 12TB drives came up with 400 badsectors in one hit.  I've seen this occasionally and my process is:
- Remove the drive from Drive Pool
- Reinitalise the drive in Hard Disk Sentinel (this will take 2 days)
- If it passes adjust the SMART stats (if needed) to ignore the 400 badsector reporting
- Put it back in the pool
- Keep monitoring

As you can see from the HDS process, this drive is so far all "green" and I expect it will finish that way and be fine.  As I've done this before, I've got a funny feeling that HDS is just rereading the SMART stats on the new/clean OS build and it has simply lost the modification to the SMART stats to ignore it from last time and hence reported it again as there is nothing in the Log (so it's not new).

[zybex]

Those temps above 50C are not healthy for the disks.

[jmone]

Yeah - hot day here in Sydney, been doing a the rebuilds / re-initalise, case cover off while this is happening.  Not ideal.  Thinking the high Sydney Humidity is also not great.

Good news is that this backup server normally only comes on at night to do the backups and then powers off.  Been like this for years without a drive failure so far (touch wood) and the oldest drive has to be 5+yo so I'm due for one or more.

[Hilton]

Yeah 50 to 55 pretty standard for me on hot days. normally 40 to 45.
Shouldn't be a problem though, most drives in last 10 years rated for 60 operating temp. not ideal but mine have been in this heat at those temps for 8 years plus. some drives now even rated for 75c..

[Hilton]

@Hilton:

1. On the Nuc do you run only Media Center or also Office applications like e-mail, Libre Office etc.?
Background: In principle I could do the same, but I always been sceptical because of Security (DAS is seen by the OS).
2. If you shutdown the NUC is the DAS not powered down?
Thank you very much!

yes I run some VMs for Home Assistant and dietpi and some other media server apps as well. As well as Radarr, Sonarr, Lidarr and newsbinpro integration.
NUC is on 24/7 and yes needs to be on to serve out DAS network shares.

[jmone]

.... and on checking, one of four hot swap HDD fans in the rack is dead (and just pumping heat) so new fans ordered.... but with the lid closed they drop to between 43-50 across all the drives even with only 3 fans.

[eve]

yes I run some VMs for Home Asistant and dietpi and some other media server apps as well. As well as Radarr, Sonarr, Lidarr and newsbinpro integration.
NUC is on 24/7 and yes needs to be on to serve out DAS network shares.

There might be a bit of confusion here with what he's asking.

If you wanted to run an actual desktop OS without access to the DAS, but on the same NUC, that's possible! Something like Proxmox (or unRAID) can let you spin up full VMs with tighter control over what resources they can access. Outside of highly targeted attacks, you're very safe in terms of a virus on the Desktop OS VM causing data loss outside of the resources it's given.

Furthermore, certain office / word processing programs, email interfaces, tons of stuff (even things without proper Web UIs) can run in Docker containers (different security considerations but for typical users, probably more than enough).

[eve]

I've never found a better case for a MC Server than the Lian-Li Z70B.  The only downside of this 10 year old case is that it does not have front panel USB-C.  .... but
- 9 x 3.5" HDD backplane with easy remove slots
- 3 x 5.25" ODD bays
- 2 x 2.5" SDD slots
- fits under a desk to probs
- Black Anodized Alum with no bling
- easy to remove font panel to get to the HDD bays

I've got a SAS M1015 flashed to IT mode that drives the DrivePool.  The three ODD are for ripping BD/UHD BD etc.

Wish there was a modern equivalent (or a mod) with USB-C front panel.

Closest thing you can get your hands on nowadays are going to be Fractal XLs. Essentially all of my builds use Fractal R5s R6s or the newer 7s. They come in a few sizes, are super unobtrusive, and they're relatively attractive. Plain black, no windows, no nonsense.

I haven't found a 4U chassis I've fallen in love with Frankly, I don't mind slapping 2 Fractal cases next to each other in a rack sitting vertically. I'm not in a huge rush to move all my systems over from the "desktop" form factor, despite the systems living in a rack.

On the smaller size of rack units, there's a small company that makes absolutely gorgeous fanless rackmount chassis for intel NUCs that are super tempting

[lepa]

I also have used Fractal Design cases, currently on 7 XL. 12 data disks and 2 for parity. I think there are be 2 easy to install places still left. Mostly satisfied but there are some metal vibration noises here and now which might go away if you twist or press the side panels. It was worse with previous revisions though.

[jmone]

Looks like the XL 7 would be the closest if I need a replacment.  For now I'm just running a USB-C extension cable from the back of the unit.  The other option is a bit of dremel work to get the Mobo USB-C cable to some location on the front.

[Hilton]

I also have the xl 7 as my main pc case. Great case!

13900K / RTX 4090 by Hilton, on Flickr

[eve]

I also have used Fractal Design cases, currently on 7 XL. 12 data disks and 2 for parity. I think there are be 2 easy to install places still left. Mostly satisfied but there are some metal vibration noises here and now which might go away if you twist or press the side panels. It was worse with previous revisions though.

Interesting. I've actually never heard that and I'm *super* sensitive to this stuff. My most 'loaded' fractal chassis has 12 3.5s in it and doesn't vibrate. You can definitely cram way more in, like I've seen ones with 18 drives.

Funny that seemingly everyone is using them. I guess Fractal pretty much has the 'I don't want a "gamer" case' market cornered.

I'm pretty sure the 4U option I want is finding a high quality amp chassis and taking a dremel to it. I'd love something close to the faceplate + handles that Bryston amps have.

[jmone]

Don't you hate rebuilds at which point older components start acting up!  On my Main PC, the LSI 9240-8i started populating Event Veiwer with all sorts of errors and warnings.  Thankfully, I had another card (not sure of the difference) but it is a LSI 9223-8i and so far it appears to be stable. 

I'm thinking that both this and the card in my Backup Server (with SAS Expander) are really long in the tooth and thinking about a "newer" card.  Maybe the LSI 9300?

[MartinM]

For media I use 2 PCs, one HTPC I built back in 2012 which has Ubuntu 22.04 LTS on a SSD, and one 2TB and one 4TB HDD (I think both are WD Greens....., not the best choice I know now) but they are still alive! And a few years back to fight bitrot and to have some redundancy I built a little server with TrueNas SCALE and two 6TB and openZFS. It only comes alive to make back ups and scrubs. And I use freefilesync to move files around. Still plenty of space for my needs Oh and I check SMART readings here and there hehe.