INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Indexing Offline Backups  (Read 3630 times)

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Indexing Offline Backups
« on: January 16, 2008, 08:34:14 pm »

Does anyone else have a good system for indexing offline backups (CDs, DVDs, Tape, & unplugged HDDs)?  I have 50+ backup discs that aren't very well organized.  As part of my ongoing projects, I would like to go through my backups and decide a few things:

1. What really needs to be kept.
2. Remove duplicate backup items.
3. Essentially recombine my backup data and re-burn to new media.
4. This further protects the data and it will give me the opportunity to make two copies.  As of now I don't really have offsite backups for archived stuff.  Generally speaking, archives are not very likely to be needed but when I go through the stuff I'll be able to better make that call.

I'm looking for a good index solution that will keep a rock solid database of each piece of media (named) and all files contained + details of the contents of files.  Ideally it could get the text from documents and maybe store some small thumbnail or something.

Or does anyone have a better solution or system?
Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Indexing Offline Backups
« Reply #1 on: February 02, 2008, 04:45:13 pm »

Still need this.  Anyone have any ideas?  I just want something to create a savable index of backups so I know exactly what is saved (and on what disc).
Logged

AoXoMoXoA

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 1551
  • I am a kangaroo . . . . no, really!
Re: Indexing Offline Backups
« Reply #2 on: February 02, 2008, 05:56:47 pm »

Have a look at CD Bank Manager: http://www.qunomsoft.com/

It looks like it may help you get the discs more organized
Logged
. . . the game is rigged

Alex B

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 10121
  • The Cosmic Bird
Re: Indexing Offline Backups
« Reply #3 on: February 02, 2008, 06:04:36 pm »

Why not use MC?

How-to: add All Media & CD database to the tree

EDIT

You can create a new library for the offline backups. Also, if the backup media has enough room you may want to install MC using the new "portable" switch on the same media.

If you use tapes I would recommend using the tape backup program's tape list/log and e.g. an Excel sheet or similar for detailed info.
Logged
The Cosmic Bird - a triple merger of galaxies: http://eso.org/public/news/eso0755

AoXoMoXoA

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 1551
  • I am a kangaroo . . . . no, really!
Re: Indexing Offline Backups
« Reply #4 on: February 02, 2008, 06:22:12 pm »

Why not use MC?

How-to: add All Media & CD database to the tree

When the CD's arenot in the drives, how does MC handle the 'broken links'?   ?
Logged
. . . the game is rigged

Alex B

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 10121
  • The Cosmic Bird
Re: Indexing Offline Backups
« Reply #5 on: February 02, 2008, 06:38:21 pm »

Offline files have never been a problem for MC. You can browse them similarly like online files.

I am surprised that you have never tried MC's CD library. If you have ripped your CDs with MC it probably has a library of them all.

If the CDs or DVDs are burned outside MC you can add them to the library just by browsing them once. The CD/DVD database is not limited to the audio or video files. It can be used for documents as well.

Similarly MC can be used for indexing offline document files on external hard drives.
Logged
The Cosmic Bird - a triple merger of galaxies: http://eso.org/public/news/eso0755

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Indexing Offline Backups
« Reply #6 on: February 02, 2008, 07:39:00 pm »

The only issue is that I want a very indepth index of the data.  For example: I would like something like Copernic search for all the files.  That means backed up text from documents (perhaps a limited amount) and thumbnails for images, etc.  I want all this stuff to be conveniently kept in a set location (including thumbnails).  This database will essentially become the lifeblood of my backup procedure.  I want a really advanced program designed specifically for this.  I'd rather not try to get MC to do this.  Believe me, MC is my most used program as it is now (for entertainment).
Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Indexing Offline Backups
« Reply #7 on: February 02, 2008, 07:45:57 pm »

I am trying the suggested program now (trial).  So far I'm very impressed.  I can have groups and disks.  I have two groups right now:
1 Copy
2 Copy

My intent is that some data does not need a second backup.  This means mainly stuff I'm keeping for little purpose.  It's tough to decide but there are some things that I am borderline deciding to even keep/backup.  In general, if it's not heavily used on the server, I like to archive it to optical media so the server is neat and clean.  One item that pops into my mind is a certain podcast that I watch.  I really enjoy the educational nature of the show and decided that it was worth archiving because downloading the episodes can be a big hassle to download again (various reasons).  I decided to burn them to DVD--therefore allowing me to delete them--but I don't feel the need to burn two copies like I would with digital photos.  Any other suggestions?

Help me develop a strong backup procedure now!  I have tons of CD/DVD discs all over in an unorganized pattern.  I want to create a nice system for cataloging and storing them--some including offsite and others not.

Any other program suggestions while I'm considering them?  Any free ones?!
Logged

mesue

  • Galactic Citizen
  • ****
  • Posts: 395
Re: Indexing Offline Backups
« Reply #8 on: February 02, 2008, 10:29:39 pm »

WhereIsIt? does a good job. It's not free, tho.
http://www.whereisit-soft.com/index.html
Logged
Sue

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Indexing Offline Backups
« Reply #9 on: February 02, 2008, 10:32:14 pm »

I have indexed about 20 discs so far.  It takes longer than I was expecting...obviously taking more time for DVDs and lots of file discs + ones that it can grab some contents (like MP3s and TXT files).  I probably have another 40 or so.

I'm finding several copies of old DVDs that I burnt too many copies for.  Trying to decide what to do with them.  Should I keep them as the ultimate backup (many copies) or get rid of them?  I have a disc destroyer that I run all discs I get rid of through (several times) so if I do get rid of them, I wouldn't ever be able to retrieve the contents.  Part of me thinks keeping them is the best choice but I really want to down size and take my current backups and essentially archive archive them and almost start over.  I want to go through the past years of backups and find that really is important and needs backup offsite.  I take lots of web site work I do and archive it but that kind of stuff (that I've spent hundreds of hours creating) I never want to get rid of or lose.  So in that case I want two copies (one being offsite).

Essentially right now if the worst were to happen, I have basically all data backed up to hard drive (outside of gigantic movie files that I own the originals to).  The biggest complaint is my optical backups.  Restoring from my backup hard drives would likely be a piece of cake.  Just copy all the files over and I have everything.  Of course it's good to have a few copies.  So if the worst case happened twice and I lost both my main and backup drives, well, I have the most important stuff backed up.  But I would have to spend hours figuring out what is the most recent backup data.  I don't even have the discs in chronological order!  Somehow they got mixed around.  I usually wrote the date, though.  I need a system to follow.

Of course with DVDs, I can't back too much up, either!  Maybe digital photos and documents if I don't include large media files.  Music couldn't be backed up.

When I decide to backup pictures, I usually go through and back them all up.  This is a huge mistake, though, because as you can see, I can quickly start wasting tens of discs because a complete backup now takes about 11 DVDs.  Oh how all that data would fit nicely on a single Blu-Ray!  Of course it's more than 3 times the price for media + $400+ for the drive.  I could backup my music on 7 Blu-Ray (DL of course).  50GB is sort of the minimum for a mass backup medium.  It's the point where you really start getting some stuff--a couple hours of SD or compressed HD miniDV, tens of thousands of pictures, and even a lot of FLAC!  150 CDs per Blu-Ray.  So if not FLAC, about 300 times the storage capacity (of course CDs are rarely full).
Logged

mesue

  • Galactic Citizen
  • ****
  • Posts: 395
Re: Indexing Offline Backups
« Reply #10 on: February 03, 2008, 12:26:55 am »

I'm done with optical storage. Hard drives have gotten so cheap and they are so convenient. I rip just about everything to hard discs now. And now I have the MediaSmart Server doing duplication. The discs I do keep... they go in those 2-disc sleeves and are stacked in a drawer. A few of the more important ones, I toss in my hubby's fire-safe, but soon I will replace those with an external drive.

I don't know how much BlueRay disks hold, but for that kind of money I could buy 2 TB worth of hard disks.
Logged
Sue

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Indexing Offline Backups
« Reply #11 on: February 03, 2008, 12:52:54 am »

You're telling me!  I have 7TB of online, usable storage in a single drive letter (RAID6).  I'm a bit paranoid and will not settle for just magnetic media.  Optical media seems like a good additional backup procedure.  I'm set in doing both (no questions).  But like I said, I only backup the vital data to optical (digital photos & documents).  Plus, it's easy to make archival copies at various points...oh and I have Shadow Copy on my server so I have 64 previous versions over the past month or more so that can really help as well.  Of course I also keep all of my CDs and DVDs in their complete form.  I don't get rid of the cases or cover art.  No way when you look at what they all cost!  May be moving soon and I'm thinking about checking into building an extra room in the basement as sort of a big fire-*resistant* room to store our original photos/slides/CDs/DVDs/Tapes, etc.  All of which I hope to finish digitizing soon but I wouldn't throw the originals away just yet.  Still have a few tapes and lots of slides/photos :(
Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Indexing Offline Backups
« Reply #12 on: February 03, 2008, 03:16:22 am »

I would possibly like it if the program I finally select would be very intelligent about everything--essentially store thumbnails of the images, sections of text from various parts of the document (300 characters every 1,000 characters, for instance, up to X characters and then lower frequency), and in video, why not store low res, re-encoded clips in the same scattered format?  10 seconds at the start and a few seconds later on all over.

Right now the database is at about 124MB with ~35 discs loaded.  That is about 3.5MB/disc.  I'd be fine with devoting a DVD to this index!  Now I do understand that it would literally have to read about the entire disc because it would need lots of data from the video clips and most if not all of the images for thumbnails and I bet the database would grow huge very quickly but it could be handy!  And then it should calculate CRC values for all the files and compare cross files between discs and link identical files.  That would be very helpful for me.

It's already taken me several (maybe 10) hours to get this far.  I'm probably 70% finished.  I want to get this project finished asap so I can move onto other projects...unfortunately I just typed up a list of 28 decent sized things to do tomorrow :(
Logged

benn600

  • Citizen of the Universe
  • *****
  • Posts: 3849
  • Living: Santa Monica CA Hometown: Cedar Rapids IA
Re: Indexing Offline Backups
« Reply #13 on: February 03, 2008, 09:11:46 pm »

CD Bank Cataloger does store JPG thumbnails.  I just didn't notice the extra tab showing a preview.  So now I know more about why it takes so long to examine each disc--especially ones with lots of images.
Logged

marko

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 9143
Re: Indexing Offline Backups
« Reply #14 on: February 04, 2008, 09:33:46 am »

I use MC to manage audio and image discs, for everything else, I use WhereIsIt. For $10 more I'd highly recommend it.

"Using MC as a total music index"

-marko

dcwebman

  • Citizen of the Universe
  • *****
  • Posts: 2154
Re: Indexing Offline Backups
« Reply #15 on: February 05, 2008, 07:42:12 am »

Ben, instead of backing up to DVDs which just takes longer, I would personally find some old hard drives, IDE or SATA, and get one of these cables:
http://yabb.jriver.com/interact/index.php?topic=41690.msg284688#msg284688

You're going to be able to store a lot more than a DVD can store, even if it's a simple 10GB hard drive, and read/write data a lot faster.
Logged
Jeff
Pages: [1]   Go Up