INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Media Backup Methods  (Read 7429 times)

zxsix

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1753
Media Backup Methods
« on: May 18, 2010, 04:16:58 pm »

I've recently changed/enhanced my backup strategy and spent some time evaluating and comparing options.  Thought I'd share my methods here in case it may help someone else.  Hopefully others will share their personal methods.

First, I must define my scenario:
MC is installed on various laptops and workstations around the house and at my office.
These all connect in to the Library Server feature of MC which runs on a Windows Server 2008 currently.
All media files reside on a 4 terabyte NAS box which contains mostly flac audio, a few mp3, a ton of images, and some video files.

So there are 2 general locations of files that need to be backed up:
All files contained on the NAS storage server.
MC's library files contained on the server's hard drive at C:\Users\username\AppData\Roaming\J River\Media Center 15\Library

OLD METHOD:
For the past few years, I've only had an onsite backup, using external hard drives.  I used a standard full/incremental backup strategy.
On the first day of the month, a complete backup was made of all files.
On Sunday nights I did a weekly incremental backup, which covered all files changed since the prior monthly full backup.

PROBLEMS WITH OLD METHOD
I decided that my old backup strategy protected me well from inadvertent data modifications and deletions.  It also protected against hard drive failure.
But it didn't protect me against theft, or loss from fire.  I also couldn't keep more than about 2 months worth of revisions due to available disk space on the external drives.

RESEARCHING ALTERNATIVES
So, I set out to streamline my backup process, preferably with an offsite storage option.  I began to research internet services that offer backup storage "in the cloud".
There are many of these, and most have some sort of maximum space limitation that my collection exceeds, or a charge on a per-MB basis, making the option too expensive.
I found 3 fairly well-known services that advertise UNLIMITED space.  Hmmm...yes, that'll do!

MOZY   Mozy gets pretty good reviews from users, and truly has unlimited disk space for a flat fee of around $50/yr.
I declined to use Mozy for 3 reasons:
1) It wouldn't let me install it's software on my server due to it running the server operating system.  It only supports desktop operating systems.
In order to use it on a server, you have to upgrade to a business class subscription, which has a per GB charge in addition to the monthly fee.
2) Reviews I read from customers indicated that while the disk space is unlimited, the bandwidth wasn't.  Backup speeds were throttled, causing the initial backup to take several weeks to complete!
At the frequency that files change, I could see the initial backup never completing to 100%.  I'd have to start the backup, then never touch the rating or any other tag stored in the file until it completed.  Not to mention any new albums added in the meantime.
3) Since I couldn't install it, I looked at screenshots, and read what was available on their website.  It seems like a decent product, but no feature really stood out from the crowd.

CARBONITE
I used Carbonite during the trial period.  I checked at 6:00pm every evening how many flac files it backed up each 24-hour period (about 150 songs per day).
The software has almost no options other than what files to backup.  Honestly the only thing you have control over is a bandwidth throttle between high and low.
Items of note about Carbonite:
1) selecting files to backup is done in explorer instead of in the application.  Simply right click a folder and choose carbonite > back up these files.
2) in explorer, each file had a green or orange dot overlay on it's icon.  Green indicates the backup contains the latest copy of the file.  Orange indicates the file hasn't been backed up yet, or the backup doesn't contain the latest revision.
3) The backups contain only one revision of the file.  It does not retain any older copies.  Mess up a file's tags, don't notice it for 24 hours?  Bang, backup now has the bad copy also :(
4) Many reviews of bandwidth throttling online and I experienced this myself.  I have 10Mb upstream available and the fastest Carbonite ran was 1.2Mb.
5) Carbonite's own website acknowledges the bandwidth cap, but it gets worse!  After you backup around 35GB, they throttle you even slower!
6) Based on the 150 song per day completion, and the bandwidth cap, I estimated it would take about 280 days to make my first complete backup!  Obviously once that's done, the files don't change much, so it would only need to do a handful of changed files and a handful of newly imported files each week.  But come one, almost a year to complete one full backup!?!
Overall, I liked Carbonite, especially how it indicated which files were backed up and which weren't.  However, I opted to not subscribe to Carbonite due to the bandwidth limitations.

CRASHPLAN
Crashplan doesn't have the name recognition that Carbonite does, but I ultimately subscribed to their service.
Items of note about Crashplan:
1) NO BANDWIDTH CAPS!  Crashplan is quite adamant that they do not throttle your upload speed.  Obviously I don't expect to run wide open at 10Mb the whole time, but I did experience much faster speeds than with Carbonite.
I was able to average around 5Mb with some peaks at 9Mb.  Looks like general internet bandwidth fluctuations and not a cap to me.
2) Files are selected using the Crashplan application using a tree folder structure with checkboxes.  Management isn't done using explorer like with carbonite.

3) I did the same timing test, how many flac files are backed up in a 24-hour period?  Answer=950!  A stark contrast to Carbonite's 150 files per day!
4) Mozy and Carbonite are for online backups to the cloud only.  Crashplan has a different approach.  Crashplan can backup to a different computer on your network.  This is free!  Crashplan can backup to an external hard drive.  This is free!  Crashplan can backup to your friend's harddrive at his house and he can backup to yours.  This is free!
Crashplan can also backup to the cloud, just like Mozy and Carbonite, and also for a roughly $50/yr fee.
5) Crashplan has file revisions!  Unlike it's competitors, Crashplan can keep multiple revisions of a file for over a year!  For the same $50!
6) Now for the negatives, and they are minor:
   a) No indicator of which files specifically are backed up and which are not, only a count of pending files and estimate of how long until the backup is 100% complete.
   b) No cool colored icons like Carbonite has so you can tell on a per-file basis that the latest revision is backed up or not.  Honestly that's the only feature Carbonite has going for it over the competitors.

THE NEW WAY
So, now having chosen Crashplan for my offsite backups, I'm covered against theft and fire at last.
Since Crashplan can backup to both the cloud, and to local hard drives, I won't need to run my old backup software anymore, freeing up some system resources.
I still get a local backup copy to my external drives that can be quickly and easily restored, I get an offsite copy for extra protection, and I get to keep revisions for much longer than I'd been keeping them before.



  
Logged

gvanbrunt

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1232
  • MC Nerd
Re: Media Backup Methods
« Reply #1 on: May 18, 2010, 05:41:21 pm »

Thanks for the great info! I've been using Carbonite for about 6 months and bandwidth cap drives me nuts. I have about 160GB of data and it never really is finished. I'm boing to give crashplan a whirl and see how they are. May be able to get away with free version backing up to an external drive at another location...
Logged

dcwebman

  • Citizen of the Universe
  • *****
  • Posts: 2154
Re: Media Backup Methods
« Reply #2 on: May 19, 2010, 07:48:20 am »

Thanks for the info. I've been doing back ups to an external hard drive every week and storing that offsite at work to protect against fire/theft. Well I guess someone could get into my locked desk if they wanted to and steal it. But I've been meaning to do a second backup to the cloud and haven't been able to decide on a service. Will need to check out Crashplan.
Logged
Jeff

zxsix

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1753
Re: Media Backup Methods
« Reply #3 on: May 19, 2010, 09:27:32 am »

Carbonite's integration with explorer is a compelling feature that crashplan lacks.  To select which files to backup, you must use the checkboxes in the tree.  In my case, it's not a big deal.  I only have to check 4 boxes:  1 for MC's library folder, 1 each for the root of my music, photo, and camcorder created movies folder.
One other limitation of crashplan that is supposedly going to be enhanced in a future upgrade, but may apply to some of us here:
If you backup to a local disk, that single disk must be large enough to hold the entire contents of the backup.
So if you have 3TB of files you want to backup, you can't use two 1TB disks and have the backup spanned.  You'd need to RAID those drives together to create a larger single drive.
Tied into that, in the current version, you can't have 2 separate selection lists to backup.
Meaning if I select all of my music, photos, and videos to be backed up to the cloud, I can't choose to backup only music to the local drive.  The same selection list applies to both places.
Separate selection lists is a highly requested feature on their forum and they say will likely be available in a future upgrade.  But for now, that has to fit into your plans.  It works out for me, since my local backup disk is a 4TB raid array and it fits all of my content.  Their compression is pretty darn good.  I haven't checked mine yet, but on their forum, there are reports of 900GB taking up 450GB of actual disk space.
They also have a feature called de-duplication that is important.
You can read the technical details on their site but here's an example that would apply to us:

45MB flac file gets backed up on Monday.
On Tuesday, you play the song and change the rating on it.
On Wednesday, it gets backed up again since the file has changed, another 45MB added to the backup drive.

Now, using their de-duplication, it's backing up only the bits that have changed, not the whole 45MB file.
So it would backup 45MB flac (and possibly compress it a bit smaller).
Then you change the rating.
Then the next time it backs up, it would only consume those few bytes of the file that are different.
That's how they are able to save so many incremental versions back of each file, even on a local disk.
Logged

Vocalpoint

  • Citizen of the Universe
  • *****
  • Posts: 2007
Re: Media Backup Methods
« Reply #4 on: May 19, 2010, 09:35:13 am »

I have almost an identical setup to the OP...4TB Windows Home Server with 400GB of FLAC audio, photos...the usual drill. I do it the fast and easy way...my server case has a built in drive caddy slot (Antec) and each week (24x7) - a fresh 1TB drive collects automated ongoing backups from all shares (I use SyncBack SE on the WHS). Each Sunday morning I simply remove the drive, pop in a new one and take the backups to the bank. Another copy is stored at work...I just rotate them in and out.

All these cloud services are way too slow for the kind of file movement I do. Adding files, tagging files and so on...each change to any file  - triggers a requirement for it to be backed up...and with the awesome mirroring and automation capabilities with Syncback - I can move thousands of files in a few hours to these drives and be assured that all copies are sound and I have triplicates of everything.

With the price of drives plummeting each minute...I couldn't imagine every using a cloud service for anything over a few hundred MB....and then there's the pain of a restore....jeez :)

Cheers!

VP
Logged

tunetyme

  • Galactic Citizen
  • ****
  • Posts: 410
  • Have tunes will travel
Re: Media Backup Methods
« Reply #5 on: May 23, 2010, 05:34:49 pm »

I like the idea of having off site storage.  But I agree that it is too slow.  I downloaded a program from Microsoft called "synctoy".  It compares every file to see if there are any changes to the file including tag updates.  It is quick and accurate.  You can sync in both directions and see what files it plans to delete or update.  I do monthly updates.  I have over 22,000 flac files so I really appreciate the speed.  I keep a backup at home and one off site.  After I do an update I defrag the disc.  Since I have experienced a crashed hard disc and a motherboard I make sure I also keep a backup of my playlists.  I use both MC and M3u formats. 

Thanks for sharing, I will look into crashplan as additional insurance.

Tunetyme
Logged

DarkPenguin

  • Citizen of the Universe
  • *****
  • Posts: 1921
Re: Media Backup Methods
« Reply #6 on: May 23, 2010, 08:35:22 pm »

I like the idea of having off site storage.  But I agree that it is too slow.  I downloaded a program from Microsoft called "synctoy".  It compares every file to see if there are any changes to the file including tag updates.  It is quick and accurate.  You can sync in both directions and see what files it plans to delete or update.  I do monthly updates.  I have over 22,000 flac files so I really appreciate the speed.  I keep a backup at home and one off site.  After I do an update I defrag the disc.  Since I have experienced a crashed hard disc and a motherboard I make sure I also keep a backup of my playlists.  I use both MC and M3u formats. 

Thanks for sharing, I will look into crashplan as additional insurance.

Tunetyme


Synctoy synced about 6 months worth of my data into the ether.  I much perfer syncback to it.

My main backup is via Mozy.  I like the new version 2.0.
Logged

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14463
  • I won! I won!
Re: Media Backup Methods
« Reply #7 on: May 23, 2010, 09:17:16 pm »

Synctoy synced about 6 months worth of my data into the ether.  I much perfer syncback to it.

Interesting, I'm a big SyncToy fan and have not had any issues (well I did open a ticket with a problem between SyncToy and WHS but they have now fixed that).  I especially like the ability to "preview" what I'm about do do.
Logged
JRiver CEO Elect

Gl3nn

  • Galactic Citizen
  • ****
  • Posts: 384
Re: Media Backup Methods
« Reply #8 on: May 24, 2010, 10:53:48 am »

I've been using ViceVersa (http://www.tgrmn.com/) for a couple of years now and have been very happy.  Count me as another who simply files away a copy on a 1.5gb HD and puts it in a safe deposit box a couple of blocks away.  Easy and fast for me.  Cheap too.  I also have a double backup system, networked, at home.
Logged

Vocalpoint

  • Citizen of the Universe
  • *****
  • Posts: 2007
Re: Media Backup Methods
« Reply #9 on: May 24, 2010, 01:32:21 pm »

Interesting, I'm a big SyncToy fan and have not had any issues (well I did open a ticket with a problem between SyncToy and WHS but they have now fixed that). 

I was a big SyncToy fan too...until it vaporized some data. It's only a matter of time with that utility. That's why SyncBackSE was (and continues to be) the best 29 bucks I ever spent...except for Media Center of course :)
Logged

tunetyme

  • Galactic Citizen
  • ****
  • Posts: 410
  • Have tunes will travel
Re: Media Backup Methods
« Reply #10 on: May 25, 2010, 08:46:52 am »

Synctoy can be a problem if you set it up to sync in both directions.  Since this is a backup, I simply use "echo" which copies new and updated files from the main disc to the backup disc and it renames and deletes on the main disc are repeated on the backup.

The key is to always check the preview of all the delete file and folders.  If you have a problem on your main disc this is where it will show up. 

The other thing I do is turn off the automatic imports in MC and set "fix broken links" to no.  If a file is damaged or deleted then MC tells me and I can restore it from my backup.

Tunetyme
Logged

HTPC4ME

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 2760
  • LIFE IS A RADIO... CRANK IT UP TO 11!
Re: Media Backup Methods
« Reply #11 on: May 25, 2010, 09:58:38 am »

i've been using superflexible file synchronizer for years.. and have had no complaints.

for those of you doing online backups.. do you think these companies will let you know when they become bankrupt? and give everyone a few days to grab there backups if need be before closing the doors? just a scenario i'm throwing out there.

ive never used them.

Logged

Vocalpoint

  • Citizen of the Universe
  • *****
  • Posts: 2007
Re: Media Backup Methods
« Reply #12 on: May 25, 2010, 11:08:17 am »

For those of you doing online backups.. do you think these companies will let you know when they become bankrupt? and give everyone a few days to grab there backups if need be before closing the doors? just a scenario i'm throwing out there.

If one of these guys goes bankrupt...usually the customer will be the last to know...plus - if you have 75GB of data out there...given the slow transfer rates - it could take weeks to get it all back....not a few days....
Logged

DarkPenguin

  • Citizen of the Universe
  • *****
  • Posts: 1921
Re: Media Backup Methods
« Reply #13 on: May 25, 2010, 11:37:37 am »

If Mozy is spun out of EMC I'll worry.
Logged

zxsix

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1753
Re: Media Backup Methods
« Reply #14 on: May 25, 2010, 03:02:09 pm »

I'm not worried if they go out of business.
I still have the original files on a RAID NAS box.
I then have a onsite hard drive backup that's immediately restorable.

I only count on the online storage in case of fire of theft.
The rottweiler and german shepherd will keep the chances of theft at a minimum.
Logged

Magic_Randy

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2367
  • I used to be indecisive, but now I'm not so sure..
Re: Media Backup Methods
« Reply #15 on: October 09, 2010, 03:39:24 pm »

Has anyone had success with any cloud backups where the source data is ~600gb?

It looks to me like my backup would never finish.

Logged

gvanbrunt

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1232
  • MC Nerd
Re: Media Backup Methods
« Reply #16 on: October 19, 2010, 10:42:06 pm »

With CrashPlan you can "seed" a backup. Backup to external hard drive, move to off site computer and the start backup again. If you don't have a friend etc that you can backup to you would have to do it to CrashPlan's servers and there is a charge for the seed and the yearly fee as well.

I use the free method mentioned. It works good.
Logged
Pages: [1]   Go Up