For background, I have dual QNAP NAS units on a gigE network with each box using two bonded gigE ports. Backup between the two NAS boxes averages 50 MB/sec which translates to about 5.5 hours per terabyte (interestingly, bonded Ethernet does not buy anything for long term transfers.) I have more than 3 TB of music in the database. File system transfer rate is the obvious problem.
I am experimenting with extensive metadata enhancements to the music files in an effort to create more effective access to a complex library. Sometimes there are wholesale tag changes to a large portion of the files which can result in 15 hour back ups. I often work 12 hour days, so this is the problem (cue Monty Python: "luxury! 11 of us lived in a cardboard box at the bottom of a lake!)
A safe tag backup regime could possibly be implemented by just reading the tag header, which is usually of a fixed length, and over-writing just the header of the backup file when changes are detected (assuming the matching backup file contained a header of the same length). The metadata block is clearly delineated and the entire header including space for album art is measured in no more than hundreds of kilobytes whether flac or mp3 or any other popular format. Hence, a 3TB backup of metadata changes which would have taken more than 15 hours when whole files are copied could be done in about 15 minutes. That is a huge savings in time. So, why on earth would one wish to do it the long way? It is not about network loads (sorry if I implied that in an earlier post), it is about time. And, for safety, if there were any difference in the header size, then the entire file would be copied but that would be an extremely rare event. What could possibly go wrong...go wrong...go wron...wo gron....
But seriously folks, I do not think that sync programs that look for differences across the entire file and only updates the differences would be reliable and actually could introduce damage. If one is only working on metadata changes and anything outside of the tag header has changed then there is a bigger problem which should probably not be propagated to a backup file copy. At the very least, the entire file should be copied to backup if any portion outside of the header has changed.
Another reason for not copying an entire large file when only a small portion of its header has changed is because it badly fragments the backup file system. This is because the updated file copied to a backup volume would be written to free space then the old file on the backup volume deleted. If only a fixed length header on the backup volume were over-written (block-level update) then this unnecessary file system thrashing would not occur.
And finally for an even simpler example: if, for instance, only the 'Last Played' tag were updated, a flac file of a track from a CD would require about a 50 MB transfer for a 4 byte change. The newer HD audio files would require 250 MB or more to copy. This is extremely inefficient.
What is needed for current large music collections is a block-level application for inline (in file) metadata backups that only acts on the tag header and is smart about any other changes in the file outside of the header. If anything out of the ordinary occurs, the program either copies the entire file or logs an error and copies nothing. Such a program should be relatively straight-forward to write (though I am not necessary volunteering to do so;)
