Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[blgentry]

I've been experimenting with making partial library clones.  That is, making a sub-set of my library and turning it into a new library.  There are a variety of reasons you might want to do this.  I think one of the most popular reasons would be to have a full standalone copy of *part* of your library on a second machine.  In my case, I wanted to make a version of a library that could fit on an external portable drive.  Again, just a sub-set of the Library.

I found a really good thread where Glynor discusses how he uses the Handheld Sync to build a copy of his library to be used by itunes.  He does conversion to MP3 at the same time.  His setup is a bit complicated, but I'm pretty sure he uses an MC library on a laptop to "receive" the converted files, and then MCiS to move the media files over to Itunes.  The thread is here:

https://yabb.jriver.com/interact/index.php?topic=97242.0

There's something rather important in this thread that I didn't know about:  Handheld Sync can use the MPL format for it's Playlist when it does a sync.  This means that you can copy part of your library to some other location, and along with it, an MPL file that contains EVERY BIT of the metadata about those media files, including custom fields, and anything else that might not be saved in the media file.

This makes building a partial clone pretty simple:

1.  In the Handheld Sync Tool, set up a directory on a drive you want to use for the partial clone.
2.  Make a playlist or smartlist on the Source library that contains the media you want to clone (a subset of your media).
3.  Configure the directory structure of the Handheld Sync.  Set any conversion options you might want.  Set the Playlist type to MPL.
4.  Perform the Handheld Sync  (Sync Now).
5.  Create a new blank library. On the machine you are going to put it on.
6.  File > Import Playlist ... and select the MPL file you created above.

Now that subset is imported into your new library.  This could be done on the same machine, or on a second machine.  As long as the paths are the same, you're good to go.  If that paths *aren't* the same, you can use Find and Replace in the Rename, Move, and Copy tool to fix the paths.

Of course you can get slightly more sophisticated and do a real clone of your Library first, then delete all of the media from it.  This helps if you want to retain any Views or anything else you have created.  Then you can import the MPL you created and all of the exported media is now in the new library.

This is pretty advanced stuff, but I think there are people here that might want to do this, but didn't know it was possible.  You could really use this for quite a few applications.  How about making a partial clone of your Library to be used on an ID?  There are some details you might need to work out (like converting path separators from \ to /), but I think it would work really well.

I just felt like sharing.  Hope someone enjoys this.

Brian.

[Arindelle]

I can see where that could be pretty cool for some people, Brian.

Question though, you mention custom field meta data.  Would importing this sync MPL file create the fields locally in the "cloned" library? I thought these fields would have to exist already to be able to import the metadata ... but maybe I'm wrong on this as I've never tried this method you're proposing. Also, would this copy over calculation fields or only custom data fields?

[blgentry]

I'm 99% certain that custom fields need to be created before they can be populated by an MPL import.  Good point actually!

So you'd need to use the more sophisticated method which replaces step 5 with:

A.  Back up Source Library.
B.  Create new Library.
C.  Import Backup of Source library into New library.
D.  In New library, find all media.  Highlight and delete from library only (not from disk).

That way, you'd have all Custom fields created, all calculated fields, all views, playlists, etc.  So kind of like a Library template, just with no media.  Then when you do the Import Playlist command on the MPL file, it will get all of the copied media *and* the metadata about that media, all in one shot.  Pretty cool stuff. 

Brian.

[bblue]

That's a nifty approach.  So how would you handle additions to the main library to propagate them to the other libraries? It looks like you would have to do all the same steps (above) after adds to the main library, or perhaps do the adds for each library?

My total Audio library is about 4TB, and the steps making a partial library would take quite a bit of time, even if everything was SATA 3 and/or USB 3.

I would love to have MC in charge of backing up my entire library when needed, or handle updates to a portable library drive (containing everything, and doubling as a backup), but it and regular backup refreshes would *have* to be one way sync.  No way would you want the portable or backup to send changes that may have been inadvertent back to the master.

Is a one-way sync addition possible?  That would really be nice.  As it is I have to use Goodsync in one-way mode for all refreshes from the Master, and then separately manually copy over the most recent active Playlists from the master to portable/backup, to import them as needed.  Perhaps there is a better way to manage that.

--Bill

[blgentry]

That's a nifty approach.  So how would you handle additions to the main library to propagate them to the other libraries?

That's one of the GREAT things about this approach.  It's just about automatic.  Your Sync controls what the Partial library gets.  If you add something new to the Main library and it goes into one of the Playlists you are syncing to the Partial library, then running Sync will just copy it over to the Partial library location.  It *also* updates the MPL file, which contains all of the metadata. 

Something wonderful I discovered about this:  Once you've done the sync to add new media to the Partial library, the next time you open it (the Partial), it will see that the MPL file has been updated and it will add all of the new media and metadata!  I was quite surprised and happy to see that happen.

My total Audio library is about 4TB, and the steps making a partial library would take quite a bit of time, even if everything was SATA 3 and/or USB 3.

Yeah, that's a lot of data.  The handheld Sync system won't copy the same thing twice if it's already there.  So each new sync should just be an incremental copy of the new stuff only.   So, the first time will take a really long time.  The updates should be fast after that.

This would seem to accomplish your desire for a "one way sync".  It's not totally automatic, but it's close. 

Brian.

[bblue]

Thanks.

As usual, I am stumped almost at the beginning of the process.

I can define the remote parameters, no problem, but under "Files, Paths, & More" for "Audio Path" it is not functioning as expected.  I want to use the Filename database field (a proper full path to the song), but no matter what I use there, [Filename], field(filename,x) (0 or 1), never returns the path, even though the Filename (path) is shown properly in the tags window for each title.  It always returns something like:

C__Users_Bill Blue_AppData_Roaming_J River_Media Center 21_Temp_01 - How I Am Different.flac

Instead of:

E:\FLAC\flacHD\Aimee Mann  HD\Bachelor No. 2  HD 88k-24b\

with the title "01 - How I Am Different.flac" in that directory.  This would create a hierarchy for each title starting at drive E:\FLAC\...

What magic does it take to grab the correct field data?

I don't want the disk storage title built from [artist]\[album]\[title], etc. because it does not necessarily represent the original hierarchy.

--Bill

[blgentry]

You've discovered one of the limitations of the Handheld Sync tool.  I need to do more testing with it, but what I saw when I was doing a bit of testing was this:  Any field that contains slashes (I'm on a Mac) or backslashes (I'm assuming) converts those path separator characters to underscores.

I too was wanting to make the directory structure in the Partial Clone a parallel structure to the original Library.  I wanted the same intermediate paths and all of that.  I tried, but didn't find a way to do it using the Handheld sync.

Eventually I decided that it really didn't matter that much.  Because I *could* organize it with the fields that were available to me (Album, Artist, etc), even if I couldn't replicate my original paths.  More importantly, MC *knows* where the files are and I never, never, never browse MC's library by path name when I'm actually trying to play movies or music.  I only browse using path names when I'm trying to do file maintenance.

You might have some sort of different application that makes having the same path names a more serious requirement.  For me, it was just something I wanted to do for consistency and I decided it wasn't important and let it go.

Brian.

[bblue]

Thanks Brian.

You've discovered one of the limitations of the Handheld Sync tool.  I need to do more testing with it, but what I saw when I was doing a bit of testing was this:  Any field that contains slashes (I'm on a Mac) or backslashes (I'm assuming) converts those path separator characters to underscores.

That alone might be something that could be managed in real time with a substitution statement.  But if you look at the two paths I showed, they are completely different and have no particular bearing on each other.  The one created not only has underscores, but doesn't really exist as part of the database.  THAT's the biggest problem.  Where does it come from?  Why should that be displayed instead of the real path (slashes aside)?

I too was wanting to make the directory structure in the Partial Clone a parallel structure to the original Library.  I wanted the same intermediate paths and all of that.  I tried, but didn't find a way to do it using the Handheld sync.

Eventually I decided that it really didn't matter that much.  Because I *could* organize it with the fields that were available to me (Album, Artist, etc), even if I couldn't replicate my original paths.  More importantly, MC *knows* where the files are and I never, never, never browse MC's library by path name when I'm actually trying to play movies or music.  I only browse using path names when I'm trying to do file maintenance.

You might have some sort of different application that makes having the same path names a more serious requirement.  For me, it was just something I wanted to do for consistency and I decided it wasn't important and let it go.

Right.  Well, the problem is that the actual path information contains additional information about the recording that is in a different format than that derived from tag fields.  As I have multiple versions of many recordings (sample rate, HD or not, DSD or not and media type (vinyl, studio tape, etc), which is used to manually be able to identify sources when not using MC, and for general organizing when not being able to see tags.  Using just tag fields, many recordings would seem like duplicates to the handheld processor, some would be dropped as such, or deleted later.  It's like maintaining two different 'standards' depending on which method of access is used.

I could normalize them to each other by rearranging the tags to emulate the full Artist name, album name, etc that is used in the hierarchy, but that makes for a more messy database view from within MC.  There I want just the relevant information for MC to display, and other information can be derived from the display of tags.

But the real question, is why can't I get the true file pathname from tags in the handheld config if it is correctly displayed in the tags windows?  Seems like a bug.

--Bill

[blgentry]

But the real question, is why can't I get the true file pathname from tags in the handheld config if it is correctly displayed in the tags windows?  Seems like a bug.

Yeah, that's a mystery to me.  I tried using [filename (path)] in a handheld and it "worked" in that it gave me the real path elements and not anything like what you showed. But it also converted all the / to _ ,which breaks trying to use it as a path element.

I sort of understand your requirements for tracking your different versions of albums.  I makes sense on a system that would be used with several different music programs.  I sort of don't understand why you'd use anything other than MC though!  (Only partially kidding, MC really is THAT good.)

I guess I'm really unclear on what you're trying to do with your partial (or full?) library clone.  Maybe you'd be better served by doing a real copy (using rsync or robocopy or something) and then doing a database update on the new library you create to change the top level drive and/or directory names.  Something like:

1.  Back up Source Library to zip file.  (MC's library, not the media)
2.  Copy media from Source drive to Destination drive.
3.  Create new library on Destination machine.
4.  Restore Source Library to Destination Library.
5.  Use Rename, Move, and Copy tool to change drive letter and/or top level path for all media in Destination library.

Brian.

[bblue]

Yeah, that's a mystery to me.  I tried using [filename (path)] in a handheld and it "worked" in that it gave me the real path elements and not anything like what you showed. But it also converted all the / to _ ,which breaks trying to use it as a path element.

I found the reason for that was that I was operating from one MC removed from my MC Server machine.  If I try the same thing on the Server itself, it doesn't produce the bogus path, it produces the unusable, but 'correct' path (modified, below).  A syntax that seems to work:

Code: [Select]

\replace(removeleft([filename],3),/#_#/,\\)\
It's interesting to note that I discovered accidentally on MC19 that this:

Code: [Select]

\removeleft(filepath(),3)\
worked.  While I was testing this, MC 21 on the server updated itself to 21.0.11 and both versions work the same now.  The difference is that the first fix takes care of the _ characters in place of \, but the second one doesn't need to.  They're already in the correct format.  I don't know if it's a real change or errors in the documentation.

On the 'one-removed' machine, if you examine any exported library file .MPL or backup XML file, they show the full and correct path to each song.  Since that information is clearly available, I'm at a loss to understand why the path is sabotaged to the $APP temp path name.  CAN ANYONE FROM JR COMMENT ON THIS?  If that is done for some downstream reason if originating from a non-server, could a variable be introduced for use in Files, Paths, & more?  Something like filepath(realpath) or similar?  That would be stellar if possible.

I sort of understand your requirements for tracking your different versions of albums.  I makes sense on a system that would be used with several different music programs.  I sort of don't understand why you'd use anything other than MC though!  (Only partially kidding, MC really is THAT good.)

Yeah, I know it is.  But it isn't perfect, and sometimes it's just easier to use a simpler program for one-off files.  Otherwise they change whatever Now Playing is active.  Plus the original reason was to be able to find what you were looking for in the original hierarchy for import into DAW software, or access quickly in another location where MC isn't installed.  It usually is by the time I leave, but different strokes and all.

I guess I'm really unclear on what you're trying to do with your partial (or full?) library clone.  Maybe you'd be better served by doing a real copy (using rsync or robocopy or something) and then doing a database update on the new library you create to change the top level drive and/or directory names.  Something like:

1.  Back up Source Library to zip file.  (MC's library, not the media)
2.  Copy media from Source drive to Destination drive.
3.  Create new library on Destination machine.
4.  Restore Source Library to Destination Library.
5.  Use Rename, Move, and Copy tool to change drive letter and/or top level path for all media in Destination library.

That's basically what I've been doing, but it's a hassle if you need to do it frequently and for minor database updates.  The top level drive letter can be ignored, pretty much.  I put the first directory below top level.  That way the portable drive can be mounted in any position and have the same hierarchy. 

--Bill