INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1] 2   Go Down

Author Topic: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)  (Read 52051 times)

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!

Last Updated: 21 Aug 2016

Background:  MC has all the power you need to maintain and modify data within it's GUI.  A few members however want to be able to import/export their library to tools like excel, some want to the modify and update this information, and others would like to be able to use a spreadsheet to add new items (for existing meta data holdings, dummy files etc)

This utility has 4 Options (see pic)
- Export the Library From MC: Exports your entire Library (or just the items Selected in the MC GUI) to a MPL and optionally converts to a Unicode/Tab Delimited TXT file which you edit in Excel etc
- Convert a MC MPL Playlist File: Converts an existing MC MPL (playlist) File to a Unicode/Tab Delimited TXT file which you can edit in Excel etc
- Update MC Library: Updates existing Items in your MC Library from a Unicode/Tab Delimited TXT file
- Create/Import a MC Playlist File: Creates a MC MPL (playlist) File from a Tab Delimited TXT file (optionally asks it you want import) that lets you add new items to the MC Library

Examples: Read the posts below for examples of what it can do.
- Adding Items to MC from a Spreadsheet - useful if you want to add items with no associated file such as managed a physical disk collection
- Updating Items in MC from a Spreadsheet - useful if you want bulk update vales for existing items.
- add your "off line" media library to MC by 8139david
- How to update a non editable System Field like Date Imported
- others???

Notes, Limitations, Future Stuff: 
- It is an AutoHotKey script so it only runs under Windows
- It takes me about 2mins to export my 30K library to a Unicode/Tab Delimited Text file and will use a bunch of Memory in the process
- It will create a spreadsheet of all used Library Fields (eg if you have not populated a field it will not appear), many of these fields are system fields
- It does not handle some extended characters sets well, so keep an eye on your files - It should now work with Unicode Chars
- *BEWARE* it WILL change or add stuff to your Library and there is no "Undo".... so backup your Library before you modify it.
- To increase the compatibility with various TXT editors, Excel etc, any field in MC that has Tabs and Carriage Returns Line Feeds embedded in the data will be shown as {TAB}, {CR}{LF} in the exported Text File.  So in an Excel you may have a Description that looks like "This Movie version is:{CR}{LF}{TAB}- The Original Theatrical Release{CR}{LF}{TAB}- Filmed in Sydney Australia" all in one cell, and in MC it will appear as:
     This Movie version is:
          - The Original Theatrical Release
          - Filmed in Sydney Australia

"Key" Vs "File Key":  In the MC GUI, the primary key for all entries is shown as "FileKey", yet during the import and export this field is oddly called just "Key".  So if you are using the "Update MC Library" the Column Header in your TXT file should be called "Key" (not "FileKey") or nothing will match and be updated

Setting Up Media Server Access:  This script uses MC's Web Services (MCWS) to Import, Export, and Update your Library.  For this to work, you will need to ensure that:
- Media Server is running
- In "MC --> Tools--> Options--> Media Network" check both "Use Media Network" and "Authentication" with "Username/Password"
- When you first Run Data Fiddler, it will ask you to enter your Connection Details, and allow you to test it works.  If you ever need to change these just delete the "MC Data Fiddler.ini" file in the same directory.

Options: 
- I've attached both a 32 and 64 bit compiled Version and the original AHK Script.  The 64 Bit will handle much larger libraries but will only run on 64Bit versions of Windows
- The first time you run it, it will ask you for your MC Login Details.  The second time the menu will be presented

Let me know of any bugs!

Thanks
Nathan
Logged
JRiver CEO Elect

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #1 on: January 29, 2016, 07:10:29 am »

Thanks! I'll try it soon, probably tonight :)
Logged

CountryBumkin

  • Citizen of the Universe
  • *****
  • Posts: 3352
Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #2 on: January 29, 2016, 07:38:33 am »

 How does your program differ from the built-in tool "File>Export Playlist (where you export "all files" to a .cvs  and select either "visible fields" or "all fields")?

The JRiver export tool seems to provide a nice spreadsheet file with each File listed on a separate row with its tag properties, that can then be opened/edited in Excel.

I think it is the "import process" that is missing.

Thanks for your work!
Logged

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #3 on: January 29, 2016, 02:46:42 pm »

Nothing yet, I was playing with how to import dummy files to MC and then update them, so ended up looking for how to convert MPLs as they contain the data differently to the Native CSV Export from MC.  One thing I could not get was an export that included the FileKey data when creating a CSV using "Included Fields = All Fields" (you have to use Only Visible Fields), which makes it harder to then later do an update.  MPL export has this info, but I should do a comparison of what other fields different between the two methods.

Anyway, in summary this is actually a MPL Converter (it asks MC for a MPL of everything and parses that)
Logged
JRiver CEO Elect

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #4 on: January 29, 2016, 09:30:59 pm »

Update: 29JAN16
- Update: Cleaned up the output a bit
- New: Added the ability to a Convert a MPL to a Unicode/Tab Delimited TXT file
- New: Added the ability to Convert a Unicode/Tab Delimited TXT File to a MPL

Adding Items to MC from a Spreadsheet:  This is where it now gets a bit more useful.  You can now use any editor that can create a "Unicode/Tab Delimited Text File" (just about anything but in particular Excel for most) and convert it to a MPL that when Imported into MC will add the items to the Library.  

1. Create Your Spreadsheet (see pic) - Make sure in each cell the First Row (the Header Row) contains a valid "MC Library Field" name that you want to populate (non-MC Library Field names will be ignored by MC).  Add you data for each item you want to create in the rows below (note: now all lines needs to have data for all fields, as blank data cells will be ignored when creating the MPL).

2. Save Your Spreadsheet (see pic) as a Unicode/Tab Delimited Txt File

3. Run MC Data Fiddler and select "Create/Import a MC Playlist File (from Tab TXT File)" and when prompted select the file you created in Point 2.  

4. MC Data Fiddler will then create a "MC_Import.mpl" file (and optionally offer to Import it for you).  I'd recommend that you Open it up in Notepad and have a look that all seems well then manually import this file by Dragging it onto the MC Menu Bar --> Import.  Or Right Click on the newly created MPL file -->  Media Center --> Import.  Either way, your Data should be in the MC Library (see pic)

You will also be able to see what was imported into MC each time by looking at the MC--> Playlists--> Imported Playlists--> "MC Fiddler Import YYYMMDDHHMMSS".  You can delete these playlist files without deleting the items that they imported.

*** WARNING *** - Be careful, backup your library first as there is no Undo if something goes wrong.  Try a small import first.  

Logged
JRiver CEO Elect

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #5 on: January 30, 2016, 03:04:05 am »

Thanks! It works.

Also, I just realized how to import by yourself a mpl playlist for several movies within JRiver :)
See this post: http://yabb.jriver.com/interact/index.php?topic=102843.msg714506#msg714506
The key is that Media Server has to be running.

By the way, in Data Fiddler, you have some fields wrong, eg "MediaType" should be "Media Type", and "SubMediaType" should be "Media Sub Type", see attached field for a correct mpl playlist.
Logged

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #6 on: January 30, 2016, 03:35:47 am »

Glad it works.  Where do you see "MediaType" and "SubMediaType" (I only see "Media Type" and "Media Sub Type)?

The only oddity on the naming convention I've found so far is that in MC, the primary key is shown as "File Key" but in the MPL it is marked as "Key"
Logged
JRiver CEO Elect

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #7 on: January 30, 2016, 03:48:43 am »

My bad. I must have made the mystake myself in the txt file when I first tried your program.
The fourth option does create a mpl playlist with the same fields as the txt file. So as you write above, at least some of those have to be labelled exactly like in JRiver.
Logged

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #8 on: January 30, 2016, 04:03:47 am »

Update: 29JAN16b
- New: Added the ability to Update MC from a Tab Delimited TXT File

Updating Items in MC from a Spreadsheet:  You can now use any editor that can create a "Tab Delimited Text File" (just about anything but in particular Excel for most) and use it to update a MC Library directly.  

1. Create Your Spreadsheet (see pic) - Make sure in each cell the First Row (the Header Row) contains valid "MC Library Field" name that you want to CHANGE (non-MC Library Field names will be ignored by MC).  One of the Fields MUST be labeled "Key" and have the "FileKey" reference shown in MC.  This is an oddity, MC shows it in it's GUI as "FileKey", but Media Center Web Services (MCWS) use "Key" and hence so does the MPL and TXT files.

Add you data for each item you want to modify in the rows below.

2. Save Your Spreadsheet as a Tab Delimited Txt File

3. Run MC Data Fiddler and select "WriteInfo : Update MC Library from Tab Delimited TXT File" and when prompted select the file you created in Point 2.  Your Data should then be updated in the MC Library (see pic)

*** WARNING *** - Be careful, backup your library first as there is no Undo if something goes wrong.  Try a small update first.
Logged
JRiver CEO Elect

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #9 on: January 30, 2016, 04:09:57 am »

My bad. I must have made the mystake myself in the txt file when I first tried your program.
The fourth option does create a mpl playlist with the same fields as the txt file. So as you write above, at least some of those have to be labelled exactly like in JRiver.

No Probs - you should now be able to mass import your physical Disk Library in MC withe all the data.

Let me know of any oddities!

Thanks
Nathan
Logged
JRiver CEO Elect

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #10 on: January 30, 2016, 04:21:53 am »

Apart from any bugs, the only other thing I can think of that may make sense is instead of extracting all items from MC, I can make it a Search String (eg Audio only etc).  That said, MC is way more powerful as you can always display what you want in the MC GUI and the export that as a MPL which Data Fiddler can then process into a TXT file for modification and later update. 
Logged
JRiver CEO Elect

Arindelle

  • Citizen of the Universe
  • *****
  • Posts: 2772
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #11 on: January 30, 2016, 04:31:25 am »

Wow! powerful stuff, thank you Nathan. Hope to have time to try this this weekend!
Logged

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #12 on: January 30, 2016, 04:33:31 am »

I'm trying to use your program (latest version) to import dummy movies whose information is specified by a tabbed text file.
What keys should I give them?

But there is a simple solution: from the txt file, use your program to create the mpl playlist. Then manually do a "Import playlist" within JRiver. This imports the corresponding movies, in this case dummy files Movie1.vob, etc
Logged

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #13 on: January 30, 2016, 04:35:36 am »

So you don't actually need any dummy files at all.  It will create the entries in MC without anything needing to be populated in the FilePath unless you want some file to be actually played.  Does this make sense?
Logged
JRiver CEO Elect

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #14 on: January 30, 2016, 04:38:21 am »

Once I have my "dummy" movies imported, I want to do a "Get Movie and TV information" and store the information, including the Cover art.
Don't I need a dummy file for that?
Logged

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #15 on: January 30, 2016, 04:44:12 am »

Nope, no need for an actual dummy file at all.

...in other words, create a Tab Delimited file (from say Excel) with all the fields (and corresponding data) you want populated, and then use Data Fiddler to create the MPL that you can then check and import.

Once it is in MC, you can then do all the "normal" stuff like Get TV and Movie Info but play the file and it will add the extra Meta Data.

The only think I'm not sure off is how the Coverart will be saved (as we have had a storm and my main Internet is down so hard to test!)
Logged
JRiver CEO Elect

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #16 on: January 30, 2016, 04:47:30 am »

That's what I was checking at the same time.
You're right!
No problem with the cover art: it is saved where the options say it should be saved.
In my case: C:\Users\David\AppData\Roaming\J River\Media Center 21\Cover Art\Videos
Logged

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #17 on: January 30, 2016, 04:50:29 am »

Beyond playing the inexistent movie, the other thing it cannot do, of course, is "update tag from library".
The latter is useful if you want to have a sidecar for each movie.
But then of course you need a dummy movie file.

I feel more secured with a sidecar: easy to reimport if the library gets corrupted.
But I suppose the solution is to do proper library backups?
Logged

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #18 on: January 30, 2016, 04:55:13 am »

Excellent!  On the "backup" you can always just export it as an MPL as it is the same thing.
Logged
JRiver CEO Elect

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #19 on: January 30, 2016, 04:55:14 am »

In the description of the program (first post), you should mention this:
- Media Server has to be running
- with Authentification selected ; the username and password asked by Data Fiddler are those specified in the Authentification of the Media Server
Logged

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #20 on: January 30, 2016, 04:57:07 am »

Excellent!  On the "backup" you can always just export it as an MPL as it is the same thing.
You're right! Thanks!
Logged

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #21 on: January 30, 2016, 05:05:00 am »

Thanks - Updated the OP re: Library Server

Also, can you test (as with my net down I can not).  If you populate the "Filename" field in a TXT file with unique data like "C:\MyDummyMovies\Movie1.MKV" (even though it does not exist) and import via a MPL with Data Fiddler, can you then create a side car file in that folder (the folder will need to exist)?

PS - I could also make Data Fiddler to auto import the MPL file (so you don't have to do it manually) but I'm wary without a user eye balling it first.  What do you think?
Logged
JRiver CEO Elect

Arindelle

  • Citizen of the Universe
  • *****
  • Posts: 2772
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #22 on: January 30, 2016, 05:14:44 am »

Nope, no need for an actual dummy file at all.

...in other words, create a Tab Delimited file (from say Excel) with all the fields (and corresponding data) you want populated, and then use Data Fiddler to create the MPL that you can then check and import.

Once it is in MC, you can then do all the "normal" stuff like Get TV and Movie Info but play the file and it will add the extra Meta Data.

The only think I'm not sure off is how the Coverart will be saved (as we have had a storm and my main Internet is down so hard to test!)
hmm just thinking out loud, as I haven't had the chance to try this, lets say I want to ONLY update certain fields for existing media .... are you saying that if I export a spreadsheet, and only want to bulk overwrite two fields, I'm assuming that I would NOT remove any of the other fields data and just update the two fields correct? If the other fields were removed, I assume it will nuke whats in them? or will it just update what is on the spreadsheet.

I'm asking that because
Quote
One of the Fields MUST be labeled "Key" and have the "FileKey" reference shown in MC.  This is an oddity, MC shows it in it's GUI as "FileKey", but the MPL and TXT files use "Key".
If the filekey is present in the spreadsheet could for example could it be possible to create say a 10 field template to share update/metadata? Is this "filekey" acting as a Unique record ID? Thinking if this could be a tool to sync metadata between non networked libraries/media files

Also, I was thinking of creating non-system Unique IDs by "track/album") -- there are posts on this in the forum, already, but using excel to populate this then reimport could be a pretty neat solution.

oh BTW -- what happens with system fields -- is this a way to overwrite ones that you normally can't (like date imported?)
Logged

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #23 on: January 30, 2016, 05:23:33 am »

...so it will update only the fields that are present in the spreadsheet and will leave the rest as they are :)

The issue with updating another library is that the file key will need to also exist and match as it will update the data for a file key on whatever library you are connected to. If the filekey on this other library is not the same you will push data into it's fields that will not correspond.  The update function will not create an entry if it does not exist, it just updates one that does... Does that make sense?
Logged
JRiver CEO Elect

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #24 on: January 30, 2016, 05:32:17 am »

...and just tested on the System Field "Date Imported" and I could update it just fine!  No idea on the data format however (I just cut and past from other file... the value I used in the spreadsheet was "139493357" and MC now shows it as 16/03/2014).
Logged
JRiver CEO Elect

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #25 on: January 30, 2016, 05:42:39 am »

FYI - the date format of Value 1 = 1/1/1970 so seems to be using Unix/POSIX/Epoch time in MC incremented by seconds

Edit - I notice if you stuff up a field value like "Date Imported" MC then shows in this field "not imported"
Logged
JRiver CEO Elect

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #26 on: January 30, 2016, 06:00:34 am »

Thanks - Updated the OP re: Library Server

Also, can you test (as with my net down I can not).  If you populate the "Filename" field in a TXT file with unique data like "C:\MyDummyMovies\Movie1.MKV" (even though it does not exist) and import via a MPL with Data Fiddler, can you then create a side car file in that folder (the folder will need to exist)?

PS - I could also make Data Fiddler to auto import the MPL file (so you don't have to do it manually) but I'm wary without a user eye balling it first.  What do you think?

I tried. Then I can see only a few tags, I suppose it categorize it as Data by default. So it seems problematic.
Even if I add a Media Type before hand, it just imports a playlist, without accompanying movie.

I tried all the options.
You get a sidecar only by Update Tags from Library, and this works only if you have an existing, non-empty dummy file.

Concerning your PS:
I tend to think like you.
But maybe you could add this as a 5th option?
Logged

Arindelle

  • Citizen of the Universe
  • *****
  • Posts: 2772
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #27 on: January 30, 2016, 06:25:43 am »

...so it will update only the fields that are present in the spreadsheet and will leave the rest as they are :)

The issue with updating another library is that the file key will need to also exist and match as it will update the data for a file key on whatever library you are connected to. If the filekey on this other library is not the same you will push data into it's fields that will not correspond.  The update function will not create an entry if it does not exist, it just updates one that does... Does that make sense?
thats terrific !... of course you'd need the same key, but if you create an external library from a restore originally, this would be the same .. I'l try to test later today

thanks this could be very cool for me
Logged

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #28 on: January 30, 2016, 06:49:24 am »

Concerning edit first post:
Try it yourself, but I think you must be authenticated for update library from txt file to work.

Also, it would be nice if the name of the mpl file was more varied.
Thus, if toto.txt is taken as input, the output would be toto.mpl.
And if JRLibrary is exported, then it's given a unique name, eg JRLibrary-2016-01-39-13-49 (including date with hours, perhaps even seconds).
Logged

ferday

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1732
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #29 on: January 30, 2016, 02:07:26 pm »

Thanks for the effort jmone!

Logged

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #30 on: January 30, 2016, 06:18:45 pm »

Update: 31JAN16
- Changed: When Files are Created the FileName will be appended with a DateStamp in the format of YYYYMMDDHHSS (24 hour) as well use the input file name where appropriate.
- Changed: The MPL playlist header is "MC Fiddler Import YYYYMMDDHHSS" so MC will see each as a unique import (else it would ignore the import) and you can now see what was imported each time in the MC --> Playlist --> Imported Playlists.

Also, Updated the OP with David's observations regarding Authentication must be on for the update library feature to work.
Logged
JRiver CEO Elect

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #31 on: January 30, 2016, 06:41:53 pm »

Many thanks again, jmone!!!
I'll try the new version tomorrow.
Logged

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #32 on: January 30, 2016, 09:38:39 pm »

Update: 31JAN16a
- Changed: Updated the OP Instructions and the two example posts
- Changed: Updated the layout and working of the Main Menu
- Added: Added the option to import the MPL directly into MC once created from a TXT file
- Fixed: Updated the parser to handle blank lines in data sets  better (normally at the end of files)

Logged
JRiver CEO Elect

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #33 on: January 31, 2016, 02:30:55 am »

Very nice work!

Here are two pics for the function 'Create/Import a MC playlist (from text file)'.
I first ran it with essai.txt (with dummy vob files in the folder), then with essai2.txt (without dummy files).
As discussed, dummy files are not needed, unless one insists on creating later corresponding sidecars.

essai.txt
Name   Filename   Director   Media Type   Media Sub Type   toto   tutu
Movie61   Movie61.vob   A B   Video   Movie   a   b
Movie71   Movie71.vob   C D   Video   Movie   x   y

essai2.txt
Name   Director   Media Type   Media Sub Type   toto   tutu
Movie161   A B   Video   Movie   a   b
Movie171   C D   Video   Movie   x   y

The MC playlist obtained from essai2 is essai_2_20160131092341.mpl:
Code: [Select]
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<MPL Version="2.0" Title="MC Fiddler Import 20160131092341" PathSeparator="\">
<Item>
<Field Name="Name">Movie161</Field>
<Field Name="Director">A B</Field>
<Field Name="Media Type">Video</Field>
<Field Name="Media Sub Type">Movie</Field>
<Field Name="toto">a</Field>
<Field Name="tutu">b</Field>
</Item>
<Item>
<Field Name="Name">Movie171</Field>
<Field Name="Director">C D</Field>
<Field Name="Media Type">Video</Field>
<Field Name="Media Sub Type">Movie</Field>
<Field Name="toto">x</Field>
<Field Name="tutu">y</Field>
</Item>
</MPL>
The fields "toto" and "totu" are created in the playlist BUT they are not imported in JRiver's library, since they do not correspond to existing fields.

Since Movie 161 and Movie 171 are already tagged as Video and Movie, one can do immediately a "Get Movie & TV Show information".
Otherwise, one would have to specify manually in JRiver first their Media Type "Video", then their Media Sub Type "Movie", before being able to Get Movie Information.
Logged

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #34 on: January 31, 2016, 02:59:14 am »

Here is some basic information about AutoHotkey, which was used to make the script and the Windows exe program:
https://autohotkey.com/
'AutoHotkey (AHK) is a free, open-source macro-creation and automation software for Windows that allows users to automate repetitive tasks. It is driven by a scripting language that was initially aimed at providing keyboard shortcuts, otherwise known as hotkeys, that over time evolved into a full-fledged scripting language.'
See also https://autohotkey.com/docs/Tutorial.htm and https://en.wikipedia.org/wiki/AutoHotkey
Logged

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #35 on: January 31, 2016, 03:27:06 am »

Thanks for the testing and I'm glad it works!  Given that the topic of how to manage a Off Line Media Collection in MC comes up from time to time, could I trouble you to write up a little Guide in this thread for future users?

Using Dummy Files:  Before I ripped all my disks to HDD, I too used dummy files to get entries into MC... but now we can create entries without actual dummy files I've been pondering if it is worth the effort and complication at all, and it has some downsides.  I've been just playing with dummy files and one of the issues is that MC will try to parse them to populate data fields (which will all be the same and not reflective of your actual library should you want to use these tags).  Important things like Duration as well as technical details like Dimensions, FPS, Compression, Audio Tracks etc.  These will all be wrong if you use Dummy Files.

If the only issue of NOT using dummy files is that "Update Tags from Library" will not work, I have a couple of suggestions:
- Instead of using Sidecar files, just select all the entries and use MC to create a MPL for them all
- I could make a MC Menu Item called "Update Tags from Library (with no file)" and it would create a sidecar for each of the selected (or all) items without a file path (or some other logic)

Also please let me know of any oddities or enhancements.

Thanks
Nathan
Logged
JRiver CEO Elect

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #36 on: January 31, 2016, 03:39:54 am »

Thanks, I will write this little guide, reusing some of my posts.

The added functionality you suggest would be nice.
Currently, JRiver allows you to export only your entire libraray as a mpl playlist.
So it would be great to export any selection, if desired with only some fields selected, as a mpl playlist.
And why not as a tabbed txt file too.
But maybe this is better integrated directly in JRiver?

Well, if you can partially add the functionality to Data Fiddler, it's great!
So that it can update the original mpl and txt files from the Library.
Logged

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #37 on: January 31, 2016, 03:59:48 am »

Actually, you can already create a single MPL for just selected Items (but it is all fields minus the Key for some reason).  Just highlight the ones you want then File--> Export Playlist then select:
- MPL
- Selection

If you want just the fields in you view (including FileKey), then use File--> Export Playlist then select:
- Text File (CSV)
- Only Visible Fields
- Selection

I'll think about the "Update Tags from Library (with no file)" option from within MC some more.... but it won't be for a couple of weeks as I'll be travelling soon.  Also if the aim is to have a backup of the meta data, the better option is probably just to back up the library itself!
Logged
JRiver CEO Elect

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #38 on: January 31, 2016, 04:04:44 am »

Here is some basic information about AutoHotkey, which was used to make the script and the Windows exe program:
https://autohotkey.com/
'AutoHotkey (AHK) is a free, open-source macro-creation and automation software for Windows that allows users to automate repetitive tasks. It is driven by a scripting language that was initially aimed at providing keyboard shortcuts, otherwise known as hotkeys, that over time evolved into a full-fledged scripting language.'
See also https://autohotkey.com/docs/Tutorial.htm and https://en.wikipedia.org/wiki/AutoHotkey

...and as with all my Utils, I post both the compiled versions and the script itself.  Feel free to edit the script as you please but be warned there is little (if any) notes in the scripts as I'm way to slack (though it makes it hard for me to remember why I'm doing stuff a couple of months later!)
Logged
JRiver CEO Elect

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #39 on: January 31, 2016, 04:05:28 am »

You're right. I had missed the option of exporting a mpl playlist for a selection.
I had seen the various options for csv. (Why not offer all of them for mpl export, in JRiver?)
Good travels!
Logged

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #40 on: January 31, 2016, 04:11:50 am »

Thanks!, also you have a day for any serious bugs for me to look at then I'm off to China and the Harbin Ice Festival (among other things)... at -25C it should be "brisk"!

Edit - Opps: "January is the coldest with the lowest temperature dropping to -38 C (-36 F)" very Brisk!
Logged
JRiver CEO Elect

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #41 on: January 31, 2016, 04:15:37 am »

Using Dummy Files:  Before I ripped all my disks to HDD, I too used dummy files to get entries into MC... but now we can create entries without actual dummy files I've been pondering if it is worth the effort and complication at all, and it has some downsides.  I've been just playing with dummy files and one of the issues is that MC will try to parse them to populate data fields (which will all be the same and not reflective of your actual library should you want to use these tags).  Important things like Duration as well as technical details like Dimensions, FPS, Compression, Audio Tracks etc.  These will all be wrong if you use Dummy Files.
That's true, this information is incorrectly populated with dummy files.

When I have time, maybe I'll use your tool to clean up my library of "dummy" movies:
export mpl from JRiver for this selection
suppress the corresponding movies from the library (after a backup)
convert it to tabbed txt fil using your tool
open in excell and suppress columns containing false information
convert back to mpl and import

The only thing is that currently, the cover art of each movie is in a special folder with its dummy file (ifo or bdmv), eg:
Inception > VIDEO_TS > VIDEO_TS.jpg
I would need to rename the covers and move them around in some way or other (all cover art back to JRiver's default folder for covert art).
Logged

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #42 on: January 31, 2016, 04:28:36 am »

I found that MC will store the Coverart in it's Coverart Folder if the filename does not exist (or next to the file if it does), so no probs there (plus you can can always get it again with the "Get Info" tool). 

The more I think about it the less I'm liking using Dummy Files, it all seems hard, gives wrong info, and it's not used by MC anyway.  The only "downside" is not being able to create a sidecar file using "Update Tags from Library", and what does that give you anyway (well just an individual copy of the meta data from the library) and in reality what are you every going to use it for Vs a Library Backup or a full MPL pull?
Logged
JRiver CEO Elect

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #43 on: January 31, 2016, 04:40:26 am »

...another thought, the "Export the Library from MC" option dumps the entire library in MPL format to memory that I then use to create the TXT file.  I could also write this entire file out in it original form as "MC_Library_YYYYMMDDHHSS.MPL" and then you would have all data for all fields (including the File Key) in MPL format.
Logged
JRiver CEO Elect

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #44 on: January 31, 2016, 05:13:16 am »

Update: 31JAN16b
- Changed: The "Export the Library from MC" option will save the full library as "MC_Library_YYYYMMDDHHSS.mpl" and optionally ask if you want to convert it to a Tab Delimited TXT file for easier editing in Excel etc

Note: Unlike the internal MC GUI option to export as an MPL this will also output the "Key" information for all items.
Logged
JRiver CEO Elect

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #45 on: January 31, 2016, 06:51:08 am »

Thanks.
Exporting the key would be a nice feature for JRiver itself.
This way you can select some files, and export playlist as you want, while still having the key.
Logged

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #46 on: January 31, 2016, 07:46:49 am »

I'm trying to recreate my library of dummy movies, like I said earlier.
I have a problem when opening the txt file in Libre Office Calc: some fields are too long.
Indeed, for some reason, in some cases, all the fields are recognized as one field, and several movies are put into that single field.
Cf. pic.
Any idea to solve this problem?
Logged

8139david

  • Galactic Citizen
  • ****
  • Posts: 345
Data Fiddler: Use For Off Line Media Collection
« Reply #47 on: January 31, 2016, 09:48:59 am »

I manage all my digital media on JRiver: films and albums.
They are on some hard disc from which they can be played from JRiver.
In that sense, they are "on line" media.

Now, I also have concrete, physical media: dvds, blurays, cds, lps...
Those are not present on any hard disc accessible by JRiver.
In that sense, they are "off line" media.

I would like all my media, whether on line or off line, catalogued by JRiver.
How can I incorporate the off line media?
Answer:
- create a tabbed text file containing the relevant information about each media;
- then use Data Fiddler to update JR's library with the corresponding information.
Cf. the more detailed explanations concerning Data Fiddler given by jmone in this post:
http://yabb.jriver.com/interact/index.php?topic=102906.msg714492#msg714492

For movies in particular
JRiver's "Get Movie & TV Show Information" works really very well, using only the title of the movie.
So suppose I want to catalog my own dvds for the first time.
Then I simply create a tabbed text file like:
 
My dvds.txt
Code: [Select]
Name
Star Wars
Startrek

or

My dvds (with Type and Sub Type).txt
Code: [Select]
Name Media Type Media Sub Type
Star Wars Video Movie
Startrek Video TV Show

Then I use Data Fiddler: "Create/Import a MC Playlist File (from Tab TXT File)".

The MC playlist obtained from My dvds (with Type and Sub Type).txt is this:
My dvds (with Type and Sub Type)_20160131165203.mpl:
Code: [Select]
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<MPL Version="2.0" Title="MC Fiddler Import 20160131165203" PathSeparator="\">
<Item>
<Field Name="Name">Star Wars</Field>
<Field Name="Media Type">Video</Field>
<Field Name="Media Sub Type">Movie</Field>
</Item>
<Item>
<Field Name="Name">Startrek</Field>
<Field Name="Media Type">Video</Field>
<Field Name="Media Sub Type">TV Show</Field>
</Item>
</MPL>

When imported in JRiver, it looks like the first attached pic of JRiver.

Remark: since Star Wars and Startrek are already tagged as Video and each given an appropriate Media Sub Type (Movie or TV Show), I can immediately do a "Get Movie & TV Show information". Otherwise, I would have to specify manually in JRiver first their Media Type "Video", then their Media Sub Type, before being able to Get Movie & TV Show Information.

Of course, this procedure also works if I have more information in my text file.
Suppose I have already collected a lot of information about my blurays (cf attached pic of excell spreasheet).
Then I have this text file:
blurays.txt
Code: [Select]
Name Commentaire Genre Media Type Media Sub Type Producer Critic Rating Director Actors IMDb ID
Solaris Vu au cinéma en 1992 Drama; Science Fiction; Adventure; Mystery Video Movie Viacheslav Tarasov Tmdb 7,5 / 10 Andrei Tarkovsky Donatas Banionis; Natalya Bondarchuk Tt0069293
Jules Et Jim Vu au cinéma en 1985 Drama; Romance Video Movie François Truffaut Tmdb 7,3 / 10 François Truffaut Jeanne Moreau; Oskar Werner; Henri Serre; Marie Dubois; Sabine Haudepin; Vanna Urbino; Serge Rezvani; Anny Nelsen; Michel Subor Tt0055032

This yields this mpl file -- see attached pic for imported playlist in JRiver:
blurays.mpl
Code: [Select]
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<MPL Version="2.0" Title="MC Fiddler Import 20160131171835" PathSeparator="\">
<Item>
<Field Name="Name">Solaris</Field>
<Field Name="Commentaire">Vu au cinéma en 1992</Field>
<Field Name="Genre">Drama; Science Fiction; Adventure; Mystery</Field>
<Field Name="Media Type">Video</Field>
<Field Name="Media Sub Type">Movie</Field>
<Field Name="Producer">Viacheslav Tarasov</Field>
<Field Name="Critic Rating">Tmdb 7,5 / 10</Field>
<Field Name="Director">Andrei Tarkovsky</Field>
<Field Name="Actors">Donatas Banionis; Natalya Bondarchuk</Field>
<Field Name="IMDb ID">Tt0069293</Field>
</Item>
<Item>
<Field Name="Name">Jules Et Jim</Field>
<Field Name="Commentaire">Vu au cinéma en 1985</Field>
<Field Name="Genre">Drama; Romance</Field>
<Field Name="Media Type">Video</Field>
<Field Name="Media Sub Type">Movie</Field>
<Field Name="Producer">François Truffaut</Field>
<Field Name="Critic Rating">Tmdb 7,3 / 10</Field>
<Field Name="Director">François Truffaut</Field>
<Field Name="Actors">Jeanne Moreau; Oskar Werner; Henri Serre; Marie Dubois; Sabine Haudepin; Vanna Urbino; Serge Rezvani; Anny Nelsen; Michel Subor</Field>
<Field Name="IMDb ID">Tt0055032</Field>
</Item>
</MPL>

The field "Commentaire" has been ignored since it does not correspond to a Library field of JRiver.
(If I wanted to avoid that, I should either have renamed that field "Comment" in the text file, or created a new field "Commentaire" in JRiver.)

And for albums?
I haven't tried it myself. But clearly more information is needed, at least: Artist, Album, and Name for each song or piece.

PS: If the files you're cataloging are offline, then JRiver will give them some unchangeable attributes, such as zero (or empty) Duration, Dimensions, File Size. This cannot be avoided, since these attributes are read directly from available files --and in the case of off line media, there is no real file with the proper properties that JRiver can access.
Logged

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #48 on: January 31, 2016, 03:00:40 pm »

Thanks David.  I had a quick look at the TXT and MPL files you sent.  The TXT file opens fine in Excel so it may be just Libre Office Calc...

When I look at the files it is this section where Libre seems to be thrown:

MPL
<Field Name="Description">"The Three Ages," Buster Keaton's First Feature-Length Film After A Number Of Comedy Shorts, Is His Parody Of Griffith's "Intolerance." Keaton Tells Three Parallel Stories About The Perils Of Romance, One Set In The Stone Age, One During The Roman Empire, And One During The 20th Century.</Field>

TXT
"The Three Ages," Buster Keaton's First FeatureLength Film After A Number Of Comedy Shorts, Is His Parody Of Griffith's "Intolerance." Keaton Tells Three Parallel Stories About The Perils Of Romance, One Set In The Stone Age, One During The Roman Empire, And One During The 20th Century.

I'm wondering if it is the double Quote Marks (") around "The Three Ages," as these don't appear for me when imported into Excel (maybe Libre is seeing them as a Delimiter or an Escape Char then reading everything from there on into one cell?).  Try deleting the Quotes one at a time from the start of "The Three Ages" in the txt file and importing into Libre to see what causes it?

I'm hoping to look at the multi line parsing this arvo with some luck....
Logged
JRiver CEO Elect

jmone

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 14267
  • I won! I won!
Re: Guide: Data Fiddler (MC Library <-> Spreadsheet Import/Export/Update)
« Reply #49 on: January 31, 2016, 11:47:02 pm »

Update: 01FEB16
- Changed: Updated how the Import/Export code works to work in reading and writing multiple lines with a data field and handle embedded Tab, and Carriage Returns/LineFeeds.  This should increase the compatibility with various TXT editors, Excel etc, as any field that has Tabs and Carriage Returns Line Feeds embedded in the data will be shown as {TAB}, {CR}{LF} in the exported Text File.  So in Excel you may have a Description that looks like "This Movie version is:{CR}{LF}{TAB}- The Original Theatrical Release{CR}{LF}{TAB}- Filmed in Sydney Australia" all in one cell, and in MC it will appear as:
     This Movie version is:
          - The Original Theatrical Release
          - Filmed in Sydney Australia

If editing the TXT file, you can added/remove {CR}{LF}{TAB} as needed

Let me know of any other "weird" characters that cause issues.
Logged
JRiver CEO Elect
Pages: [1] 2   Go Up