More > JRiver Media Center 21 for Windows
Backups and Tagging
ErikN:
I do periodic, incremental backups of our storage including all of our media. This was the first backup since using MC. I had expected the back to be about 10GB since there were only a couple of new videos and, size wise, everything else is inconsequential.
For our music, I did audio analysis and some other tag adjustments. Of course that meant that every audio file changed. Not ideal but, in the grand scheme still not too bad. My 10GB backup became 20GB.
The real problem was videos. Luckily MC didn't directly tag mp4 and m2ts/ps. However, I have a small percentage of wmv. Every one of these changed. This caused my 20GB incremental backup to swell to 200GB.
Is this a problem for anyone else or did I miss a setting somewhere?
Would it be possible to add a settings like:
- Use sidecars for all video files
or
- Use sidecars for all files > 100 MB (size adjustable by the user)
Arindelle:
Hi,
I take it you want to reduce the time spent on incremental backups?(I presume space is not the issue as 200gb is not a real lot .. ) Are you backing up from or to a slow NAS or via a slow network connection? I have 4tb of audio without counting any videos, so I'm always backing-up incrementally a lot more than that (when you say all files were analyzed and that added 10Gbs).
I'm not sure if this will help or that you even want to do this, but you can choose which tags you want to be written to disk. The remaining will still be kept only in the library file. To do this you can choose via Options=>Library&Folders=>and uncheck the box "Save in file tags when possible". Make sure you are disciplined about keeping JRiver library backups and archiving multiple copies.
I'd try and find what is changing and causing these larger video files to be chosen for an incremental set ... I hardly ever have video files, once initially tagged, changed to trigger an additional back-up. Some playback stats you don't need, somebody adding a rating - whatever ... if the main tags aren't modified, the files shouldn't be added to your "incremental" set right?
Although in theory you can avoid this entirely by unchecking all the fields as I indicated above, but the result would be tha no major modifications would be written to the files ... I wouldn't want to do that .. Anyway, maybe you created a custom field that changes all the time? or maybe there is just one or two fields that are the culprits.
Might want to look at what trigger choices you have for your back-up software too. You might want to separate your other "storage" from your media as you don't need to do "full backups" in the cycle normally. A backup set of MC library files and all media kept current daily more or less + an archive of separate drives, set aside in a safe place updated basically 2 to 3 times per month is all I do, personally.
As for sidecar files, I don't believe you can force a particular file format to use them. I suppose you could convert the files to a format that do use them -- is this wise? Doubtful :)
PS- If you didn't know this, you can always write tags retained just in the JR library to be embedded in the file itself (see library tools) or vice-versa.
ErikN:
The issue is the time it takes to perform a backup. My main library sits on a 12 disk hardware RAID. My backups push new/changed files over esata to an encrypted store + read back w/ hash check. The net-net is that a backup averages about 100MB/sec. What should have been a 2 min backup became a 30 min backup.
The vast majority of my library is video. My fear is taking some innocuous action in MC (audio analysis of video files, hand tagging, etc.) triggering a multi-TB incremental backup. It seems better to keep tags adjacent to large files instead of inside them.
I guess I'm asking if I am violating some kind of 'best practices' with MC? Or, is it normal for a small tag adjustment to cause a n-GB file to change? Are most peoples' tags effectively invariant?
ps. For the 'wmv' files that got caught up in the backup, it appears MC set a field called 'beats-per-minute' into the wmv itself.
Arindelle:
--- Quote from: ErikN on September 08, 2015, 01:57:13 am ---I guess I'm asking if I am violating some kind of 'best practices' with MC? Or, is it normal for a small tag adjustment to cause a n-GB file to change?
--- End quote ---
No, its normal. if you have, made changes to tags and the fields are marked to write to disk. I gave you a solution to this above. Just remember to backup and archive the JRiver Library backup-zip files!
--- Quote --- Are most peoples' tags effectively invariant?
--- End quote ---
I'd say so ...
I mess with my audio files all the time and add information etc. But for the video files, I import em, tag them, back them up once and archive them and thats it ... no playback stats are written to disk for me.
I'd say what is common is to import, verify the tagging and re-tag if needed, back-it up, archive it. However it is also normal to analyse your audio, rebuild thumbnails and things as part of the import process, if you do it later logical that it will had the info to the file container right?
--- Quote ---ps. For the 'wmv' files that got caught up in the backup, it appears MC set a field called 'beats-per-minute' into the wmv itself.
--- End quote ---
normal, its part of audio analysis. You probably haven't run this on these files yet.
mwillems:
--- Quote from: ErikN on September 08, 2015, 01:57:13 am ---The vast majority of my library is video. My fear is taking some innocuous action in MC (audio analysis of video files, hand tagging, etc.) triggering a multi-TB incremental backup. It seems better to keep tags adjacent to large files instead of inside them.
--- End quote ---
For almost every kind of video file I've encountered MC already uses sidecars instead of writing to the file. Certainly true for .mkv, .ts, .avi files etc. So what you're asking for is kind of already the situation for most video files (IME anyway).
I don't have any .wmv files, but I would expect any issue you're experiencing would be limited to files that can store tags inside them (which is pretty atypical for video files). Audio is different, as most formats support embedded tagging.
You can control whether files get tags written to them as Arindelle mentioned if this is a dealbreaker for you, but be sure to backup your library regularly.
--- Quote ---I guess I'm asking if I am violating some kind of 'best practices' with MC? Or, is it normal for a small tag adjustment to cause a n-GB file to change?
--- End quote ---
Any change in tags that gets written to a file will obviously change the date stamp. Whether that triggers a backup of the entire file or just the changed section really depends on your backup solution, not on MC.
For example, I use an rsync-based backup solution,* and if I edit embedded tags rsync just sends the changed parts of the file, not the whole file. Modern de-duplicating backup systems send even less information than rsync. However, some backup systems send the whole file everytime, which is obviously suboptimal (but it sounds like that's what your solution is doing).
So you're not violating MC best practices, but you might want to look into a more granular backup solution if your solution sends the whole file anytime something changes (which is kind of surprising behavior with modern backup software TBH given that rsync is FOSS and solved this problem 20 years ago).
*NB: On the off chance you're already using an rsync-based solution, it may be defaulting to whole file transmission if both locations are perceived as "local," in which case you need to make sure to pass the --no-whole-file flag to force it to only do a delta backup (this will mostly help speed things up if bandwidth is your bottleneck, rather than disk i/o, but will transmit less data for sure).
--- Quote --- Are most peoples' tags effectively invariant?
--- End quote ---
After initial setup and analysis, my tags rarely change, but when there are significant tag changes to files with embedded tags (i.e. mostly audio files), I expect to see a somewhat larger backup load. Again, the load will depend on your backup solution, my load increases based on the number of files changed, but definitely does not require re-transmitting the full files (or anything close to it).
--- Quote ---ps. For the 'wmv' files that got caught up in the backup, it appears MC set a field called 'beats-per-minute' into the wmv itself.
--- End quote ---
That's an audio analysis field, and shouldn't ever be "reset" once it's set, so I wouldn't expect this specific issue to recur.
Navigation
[0] Message Index
[#] Next page
Go to full version