Topic: amg.pl: an A M G helper and dummy file creator (Read 88431 times)

SkGe · « **Reply #50 on:** November 05, 2013, 12:44:07 pm »

One question about the new similar albums.
If i want to create a playlist with this field and to be associated to the specific name of albums it can be possible?

MrC · « **Reply #51 on:** November 05, 2013, 02:32:35 pm »

I don't see any easy way to relate the various entries in this list field with the entries on your library. Playlists are lists of files (tracks, in the case of Audio), and the entries in Similar Albums are Artist\Album pairs, where Album may or may not match the entries in your library. I can't think of away to expand each of those Albums into the possibly corresponding tracks in your library.

SkGe · « **Reply #52 on:** November 20, 2013, 03:05:51 am »

Hey MrC, i didn't come here in the last time, because this piece of ART work very well, but something is rather annoying.
The diacritics when are pulled in the mpl is messed up. Can be done something with this problem?!

MrC · « **Reply #53 on:** November 20, 2013, 04:00:59 am »

Sample URL please.

SkGe · « **Reply #54 on:** November 20, 2013, 06:40:51 am »

This is a few site witch where i meet that problem:
http://www.allmusic.com/album/in-search-of-sunrise-vol-3-panama-mw0000228136
http://www.allmusic.com/album/in-search-of-sunrise-vol-11-las-vegas-mw0002549913
http://www.allmusic.com/album/in-search-of-sunrise-vol-6-ibiza-mw0000488883

MrC · « **Reply #55 on:** November 20, 2013, 12:55:21 pm »

Please give this pre-release version a try:

Changes in 1.14:
   - Fix: function to capture Recording Date was not being called. Key 'rdate' maps to
   MC field "Recording Dates".
   - New: new field Similar Albums: key 'similar'.
   - Fix: the Track Pick value is now set to 1 (instead of 'pick') for track picks.
   - New: better diagnostics when config file cannot be opened.
   - Fix: AMG entries with no tracks were failing to produce output even with -A.
   - Fix: output MPL was not correctly UTF-8 encoded.

Edit: pre-release removed in favor of version 1.14 (see first post in this thread).

SkGe · « **Reply #56 on:** November 20, 2013, 02:24:25 pm »

I tested the new file and it seems to work fine.
As always MrC, thank you.

MrC · « **Reply #57 on:** January 16, 2014, 04:48:34 pm »

I've updated the script to version 1.14. The most notable change since the pre-release posted above is that when using --mcws, you no longer have to manually Copy the track data. The selected track's metadata will automatically be placed on the clipboard.

Version 1.14
- New: manual Copy of clipboard not required when using --mcws
- Fix: function to capture Recording Date was not being called. Key 'rdate' maps to
MC field "Recording Dates".
- New: new field Similar Albums: key 'similar'.
- Fix: the Track Pick value is now set to 1 (instead of 'pick') for track picks.
- New: better diagnostics when config file cannot be opened.
- Fix: AMG entries with no tracks were failing to produce output even with -A.
- Fix: output MPL was not correctly UTF-8 encoded.
- Fix: remove extra comma in options argument list (might have caused problems setting mcws password
from the command line).

stevem87 · « **Reply #58 on:** February 04, 2014, 12:34:55 pm »

Trouble with the last track being left out of the script.

Here is the URL: http://www.allmusic.com/album/to-the-sea-mw0001985381

Command line: perl amg.pl -u http://www.allmusic.com/album/to-the-sea-mw0001985381

MrC · « **Reply #59 on:** February 04, 2014, 12:47:29 pm »

Thanks. I just tested the same and received all 13 tracks, but via MPL import and MCWS.

Your clipboard does indeed have 13 tracks on it.

Let's work offline, I'll send you my email via PM. I'd like to get a copy of the debug output and the created mpl file.

Add to your command line:

-d -v -o - --mpl > debug.txt

and email me the debug.txt file if you could.

MrC · « **Reply #60 on:** February 04, 2014, 02:41:29 pm »

Ok, we figured out the problem.

stevem87's view had an empty column at the end of the view, and this confounded the detection of old-style clipboard contents vs. the newer contents available in more recent MCs. For now, don't have an empty column at the end of the view's file list.

I found another bug too - if the Filename (name) or Filename (path) field was in the view, but not the Filename field, the script didn't notice this, and would add new (non-file backed) entries into MC rather output the error indicating that Filename was missing.

I'll fix these for the next release.

stevem87 · « **Reply #61 on:** February 04, 2014, 03:55:52 pm »

Sorry Mike, one more thing...

Where can I put the credit mapping commands in the config file? I want to map the 'Producer' credit to the Producer tag field. Just don't know where to put it.

Thanks

MrC · « **Reply #62 on:** February 04, 2014, 04:43:16 pm »

First you'll need the update I posted above (v1.15) which corrects the issues we worked out today, plus fixes an issue mapping remapped credits when they were not specifically asked for with --include.

Next, you'll need to use the config file, so copy and edit it. You'll find a line that is commented out in the Credits section that shows some mappings:

   #Credits: Technical As.*:ignore, Photo.*:ignore, Saxophone:ignore, Producer:Producer, Composer:Composer

You'll want an uncommented version specifying your mapping(s):

   Credits: Producer:Producer

This says to map the pattern "Producer" (left side) to the field named Producer (right side) in MC. See the comments in the config file.

Finally, you'll want to add the config file to your command line:

   -c name_of_your_config_file

MrC · « **Reply #63 on:** February 04, 2014, 04:45:47 pm »

I've updated the script to version 1.15:

Changes:
- Fix: remove extraneous debug output when getting similar artists.
- Fix: a missing Filename column was sometimes not detected.
- Fix: ensure inclusion of mapped credits when no exclusion is present.
- Fix: problem mis-detecting old clipboard data when last column in view was empty.

stevem87 · « **Reply #64 on:** February 04, 2014, 07:41:35 pm »

I got another one..or two. Is there a way to stop the script from changing my cover art. I have the artwork set and embedded inside the file. After running the script it changes it to external location.

Also, I tried running it in mcws and it gave me an error and only populated the first few tracks. It did populate some, so the authentication stuff must be ok.

MrC · « **Reply #65 on:** February 04, 2014, 07:46:24 pm »

The script doesn't touch the Image File field (and hence your cover art).

When we ran it on your system, where you didn't have a Filename field, you were getting new entries. Are you still doing the same, and is this the cause?

I suppose when you import fields, MC might be re-calculating where artwork comes from, as based on your preferences. I've not seen this happen. What is your preference setting for cover art location in Options > File Locations for Cover Art?

I'd need to see the error you get w/MCWS. Can you post it? Was it MC that gave the error, or the script?

MrC · « **Reply #66 on:** February 04, 2014, 08:54:37 pm »

I posted 1.16 which fixes a problem introduced in 1.15.

MrC · « **Reply #67 on:** February 04, 2014, 09:10:25 pm »

Quote from: stevem87 on February 04, 2014, 07:41:35 pm

I got another one..or two. Is there a way to stop the script from changing my cover art. I have the artwork set and embedded inside the file. After running the script it changes it to external location.

I have reproduced this here. This is an MC bug. It appears that when an MPL is imported, MC is re-calculating where cover art is located. And it seems to prefer external cover art over embedded cover art, so immediately after doing the import, the Image File field is changed from "Inside File" to the name of the external cover art location/file. You can correct this by running a Cover Art > Quick Find In File / Cover Art Directory.

Reported: http://yabb.jriver.com/interact/index.php?topic=87247.msg597343#msg597343

stevem87 · « **Reply #68 on:** February 04, 2014, 09:18:42 pm »

I was just going to say that is how I fixed it, but it would be nice if it didn't happen. I attached a text file of the error from the script.

Thanks

MrC · « **Reply #69 on:** February 04, 2014, 09:30:28 pm »

OK, there are several problems.

The first was correctly in the 1.16 release I just posted. Please download and use that release. Sorry about that.

The second is that amg is timing out, but this shouldn't be catastrophic - just no lyrics.

The third you can fix. Its this:

*** can't open config file amg-config.txt--mcws

Add a space between the two arguments, before the --mcws.

The fourth is that MC is balking about the data being presented in the URL for doing the updates. I'll have to decompose the data to see what's going on, and what MC finds offensive. Back ASAP...

stevem87 · « **Reply #70 on:** February 04, 2014, 09:34:22 pm »

Yeah, I had fixed the others already. It's the last one that was the real problem.

MrC · « **Reply #71 on:** February 04, 2014, 10:39:37 pm »

I tested the URL you were using (10,000 Maniacs, MTV Unplugged), and it worked for me here with MCWS.

I don't see anything wrong with the URL being sent, so we'll have to dig deeper.

To your command line, add:

-d -v -o - --mpl > debug.txt

so we can capture the entire session. Send the debug.txt and your config file. I'm probably going to have to reproduce everything exactly.

stevem87 · « **Reply #72 on:** February 05, 2014, 10:10:00 am »

No need, I used the update that you posted and it works fine now. I tried it on a few albums and works great. I have been playing around with the credits field and mapping different credits to different field, man that is awesome! This script does everything I wanted it too and more. Thanks so much!

BTW, Using mcsw fixed the cover art issue.

Also, the AMG rating field always comes in as '5' regardless of what they rated it. I tried it as an integer and decimal, same result.

MrC · « **Reply #73 on:** February 05, 2014, 11:14:29 am »

Excellent news, thanks for the follow-up.

There are two rating-related keywords: rating and ratingmc. The keyword 'rating' is the 0 to 9 point number amg uses to encode the 0 to 5 rating with .5 increments. The 'ratingmc' keyword is mapped to fit MC's 0 to 5 integer. The --rating <format> option affects the former, not the latter.

With my test last night on the 10,000 Maniacs album, Rating was correctly set to 2 (keyword 'ratingmc', by default mapped to MC's Rating field), and the AMG Rating field (keyword 'rating') was set to 3 (which is shown as 2-stars on their site).

stevem87 · « **Reply #74 on:** February 05, 2014, 04:30:55 pm »

OK, that same error keeps popping up. When I run it .mpl it imports fine. Happens with certain albums consistently, but other albums are fine.

Thanks

MrC · « **Reply #75 on:** February 05, 2014, 05:08:32 pm »

I'll try to reproduce it here. If you can narrow down the problem a bit, to certain tracks, and certain fields, that would help. I'm suspecting that the MCWS service's new support for packaged field/value pairs is having trouble with something.

I don't see a server error in the debug.txt file, so I'm not sure what I'm looking for.

In your config file, you have an error:

Optins: -o image file

Options is misspelled, and you currently can't use whitespace inside an option argument such as the one you specified with -o ("image file"), so use image_file, or imagefile,or something with no whitespace instead. I'll have to add some quoting mechanisms here to support whitespace.

stevem87 · « **Reply #76 on:** February 05, 2014, 05:22:04 pm »

Depends on the album, with that one it did pull in anything. With Aerosmith's Greatest Hits it pulled in everything but the last two tracks. With 10,000 Maniacs it was pulled in the first two. I thought it might be a timeout thing or something, but I tried the same album a few times and it stopped at the same places. I moved on to the next album, and it worked fine, went back again and it stopped in the same place.

As for the typo, I'll just take it out, I don't think it helps anything anyway.

Thanks

MrC · « **Reply #77 on:** February 05, 2014, 05:44:36 pm »

Do these tracks already have track or disc numbers which may be incorrect? If so, use the sequence option -s or --sequence. This tells the script to ignore your disc and track numbers, and just use the sequence of files in the view.

Whenever you get an errors like this, feel free to email me the debug.txt output for that particular run (you may have to re-run the command to capture the output). That will give me the most data regarding what might be going wrong.

Timeouts should be seen in the output.

Please re-download version 1.16. Somehow I goofed in creating the zip archive, and your version is the old 1.15 version. Sorry.

And thanks for your help and patience.

stevem87 · « **Reply #78 on:** February 05, 2014, 06:08:59 pm »

Double checked the track numbers on this, no problems here.

BTW, the option of having the mpl spit out the separate files (when the 'filename' field was absent did have some merit. It worked on things like box sets, where all of the tracks or discs are not present/renamed or whatever. That way you could just copy the tags track by track and not write over the names and track numbers. I used this for the Genesis box sets that have like 4 studio albums 5 DVDs and a bonus disc. I just needed the bonus disc because the studio albums were named accordingly. The file that came in the mpl had all of the tracks including the DVDs.

stevem87 · « **Reply #79 on:** February 05, 2014, 06:36:15 pm »

This one got up to track 5 on disc two... All fields populated on track 4, none on track 5. Tried it twice.

MrC · « **Reply #80 on:** February 05, 2014, 07:05:36 pm »

I've been able to reproduce there error here; it was your credits mapping that triggered the problem.

My current thinking is that the URL being generated to post the updates is too long (the most recent one is 6685 chars), so I'll try breaking it up into smaller chunks to see if that helps.

By the way, your server's password is in the config file. I'll remove the file from your post above, but you should probably strip it before posting. If you care, you might want to change it in MC.

[Edit: Nope, not length. Keep trying...]

[Edit: Got it! Grab the new 1.17 version.]

MrC · « **Reply #81 on:** February 05, 2014, 09:23:12 pm »

I just posted version 1.17 which should fix your issues (and maybe those reported by others in the past, that went undiagnosed).

Changes:
- Fix: content being returned from web pages was not being decoded from external UTF8 to internal form.

I was failing to decode the UTF8 from the web pages, and when it was later being re-encoded for output to MC, some characters became invalid (and this is why things failed sometimes, but always on the same tracks). Unfortunately, MC does not indicate this to be an invalid UTF8 problem, but instead claimed it was a CSV formatting problem with the CSV-encoded content passed to MCWS. Now we know.

stevem87 · « **Reply #82 on:** February 05, 2014, 09:37:03 pm »

Brilliant!!!! I think that's got it.

MrC · « **Reply #83 on:** February 05, 2014, 10:26:10 pm »

Great, thanks for the confirmation.

I have a overarching question for all...

Currently the script does not populate fields when there are no values reported via amg. This means stale, perhaps incorrect values can remain for your tracks. This might happen if you've used the wrong URL, or accidentally placed the wrong value in some field. So if you have a bunch of tracks, that have a field, say Track Pick, set to 1, and run the script, but only Tracks 2, 3, and 10 are Track Picks, the script will only set a value of 1 for those tracks, leaving the others untouched. But the tracks already had bad data; so the Track Pick field remains clouded.

Alternatively, certain fields can be assumed to always have values for each track, and for Boolean (True/False) types, they are set to 1 for TRUE, and could be set to 0 for FALSE when there is no data present. When the field types are Decimal and Integer fields, setting a 0 value seems the right thing to do (but I don't know your field types), and a 0 value is tantamount to an empty value. But for String and Large value fields, a 0 is a 0 and there's no way to clear a field via MPL or the method I'm using via MCWS. So if I write a 0, and your Track Pick field is a String field, you'll get a 0 in your columns (this could be a good indication to you to change your data type).

However, for a field like Lyrics, amg may not have any lyrics for some tracks, but you may. So clearing your Lyrics fields for these tracks would not be good.

But if you had the wrong lyrics for some reason, and you run the script, there's way to tell the script to clear the field, and no way currently to force a clearing of such a field anyway (the value could be set to "N/A" or some other string to indicate emptiness).

My feeling is that if you've asked for the data to be pulled, it should be correctly reflected in your tracks. And in some cases, missing data means FALSE and in other cases missing data means DUNNO (and you might want LEAVE ALONE or OVERWRITE).

What to do?

stevem87 · « **Reply #84 on:** February 06, 2014, 06:05:56 am »

I would opt for leaving it the way it is. Let the user decide that the data is wrong. I'm still more concerned about it overwriting data that I don't want it to, like names and track numbers. Or getting the box set info like I discussed earlier.

SkGe · « **Reply #85 on:** February 06, 2014, 07:21:54 am »

Just some question.
Sometimes when grab info from amg to my album, some album are not in the same line as the track listed on that page so if i don't check first the tracks to be as the link it will write on my album the track listed on that link, so that it will broke my album and need to redo back to the previous and to look and fix the track list and to grab again. Now the problem is after i grab info the script already write the info on my track list and i can't redo with ctrl+z so i need to look manually in the track list and change to be exact as the link where i grab the info, is somehow a better solution to redo the info back to that state or i do by hand. Not always i look at the track list but just when i see they change after i import the script. Sometimes is pretty annoying, to see that and need to check every single track of change and redo.
Thx!

stevem87 · « **Reply #86 on:** February 06, 2014, 12:30:52 pm »

Got the error again...

MrC · « **Reply #87 on:** February 06, 2014, 12:33:00 pm »

@sirgeo -Some tips:

   - first check that the AMG URL you are using is the correct one for your album release. Sometimes the primary listing is correct, other times you'll want to use another one listed under the Releases tab. Compare your number of tracks with their listing. The status window at the bottom of the MC window will show you how many items you have selected.

   - be sure your tracks either have valid and correct Track #/Disc # entries OR use the -s or --fileregex options. The script needs some way to correlate your (possibly ordered) tracks in the view with the always ordered tracks on the web page. The script tries to use your disc/track numbers, and if those don't exist, will use the order in the view. Or you can override that behavior by using the --sequence (i.e. -s) option (which means use the order of the items in the view), or --fileregex option to grab a track number from your file names.

   - you can make a backup of your track data using File > Export Playlist, and use the MPL format, selecting All Fields and The Selection. Do this before you use the script on an album, and if you don't like how it turned out, you can just File > Import Playlist that MPL file to restore the entries (mostly, it won't CLEAR data to empty).

MrC · « **Reply #88 on:** February 06, 2014, 02:04:53 pm »

Quote from: stevem87 on February 06, 2014, 12:30:52 pm

Got the error again...

Drats. I'll have a look, thanks for the report.

stevem87 · « **Reply #89 on:** February 06, 2014, 02:18:09 pm »

Thanks, here's another. Happened a few time now.

MrC · « **Reply #90 on:** February 06, 2014, 04:04:20 pm »

Edit: This is resolved in MC 19.0.116.

I'm now pretty sure this is a bug with MC. It appears that MC is getting confused during the deconstruction of the URL being sent to the MCWS service into its constituent components (decode UTF8 URL encoding, and then decode CSV data).

The failure is occurring on the character é, which is Unicode: U+00E9 and UTF-8: C3 A9.

The URL we're passing is using UTF8, and that character encoded as %C3%A9 occurs in two places. This causes the MCWS write to fail. If I change one or both them to Unicode representation of %E9, then the write succeeds.

I'm going to have to get some JRiver help on this.

Edit: if this isn't addressed within a couple of days, I may implement a workaround; after an error is encountered, the script would fall-back to single-field writes for the track. This will make the updates slower, but at least should work reliably until the problem is resolved.

stevem87 · « **Reply #91 on:** February 06, 2014, 06:08:20 pm »

FYI, I tried a different release page on both of those albums and it worked fine.

MrC · « **Reply #92 on:** February 06, 2014, 08:55:32 pm »

Quote from: stevem87 on February 05, 2014, 06:08:59 pm

BTW, the option of having the mpl spit out the separate files (when the 'filename' field was absent did have some merit. It worked on things like box sets, where all of the tracks or discs are not present/renamed or whatever. That way you could just copy the tags track by track and not write over the names and track numbers. I used this for the Genesis box sets that have like 4 studio albums 5 DVDs and a bonus disc. I just needed the bonus disc because the studio albums were named accordingly. The file that came in the mpl had all of the tracks including the DVDs.

I'd like to understand your usage scenario here a bit more:

a) which AMG URL are you referring to?
b) what discs/tracks do you have?

The goal of the script is to match your selection when the correct entries, so I'd like to continue to refine ways to do this if necessary, so that you don't have to copy / paste.

Also, if you only want specific fields, you should use --include (or -i) to specify the list of desired fields. This is not only much faster, because it avoids unnecessary page look-ups (there can be MANY per album), but also can help you avoid overwriting fields you want unmodified.

MrC · « **Reply #93 on:** February 06, 2014, 09:49:34 pm »

stevem87 · « **Reply #94 on:** February 07, 2014, 07:55:51 am »

Quote from: MrC on February 06, 2014, 09:49:34 pm

@stevem87
Note: in the next release, I've reversed the MC field/regular expression, so lines will look like this:

Thanks for showing me that simpler way of writing the Credit mapping, but does this mean I have to rewrite my config file?

stevem87 · « **Reply #95 on:** February 07, 2014, 07:59:47 am »

BTW, there is some issue with character mapping going on. When certain characters are on the page, they get imorted like this:

Vanda &amp
Young

and this:

Robert John &quot
Mutt&quot
Lange

stevem87 · « **Reply #96 on:** February 07, 2014, 08:25:45 am »

In regards to my copy and paste thing, here are some examples. I emailed you mpl files of what I have in my library. With Genesis, the set contains 5 studio albums that are in essence the individual albums. When you rip them they come in as the regular individual release. There is a bonus disc of unreleased tracks that still needs to be accounted for as you see in my mpl file. I guess there is a way to get your script to map to those, but I don't know that is really easier than the copy and paste method.

The Police release is a little different, It too contains all of the full studio albums and bonus tracks, but they are thrown on four CDs rather than separate disc that would retain the proper track numbers. For this I manually changed the track numbers and album names, so in my collection I have the individual albums and an extra disc of bonus tracks which were originally spread out on the set.

http://www.allmusic.com/album/genesis-1976-1982-mw0000779142

http://www.allmusic.com/album/message-in-a-box-the-complete-recordings-mw0000102060

stevem87 · « **Reply #97 on:** February 07, 2014, 11:52:23 am »

http://www.allmusic.com/album/i-never-loved-a-man-the-way-i-love-you-mw0000651820

On this one it loaded tracks 1-8 and I got the same error. I then selected tracks 10-14 in the library and re-ran the script. It pulled in those except for the last track, which is the same name as track 9.

MrC · « **Reply #98 on:** February 07, 2014, 12:37:28 pm »

Replying to your last 4 posts...

1. about the & and " characters (these are HTML entities, used because & and " are reserved characters in HTML) - which fields are you seeing these in, and which AMG URL. I'll check to see where all the conversions might be going wrong. I thought these were all addressed, but obviously not.

2. I replied to your email regarding your MPL samples, and offered some suggestions for dealing with box sets. I see the URLs in the post above too, so that's great, thanks.

3. I'll test out Aretha Franklin album, but I'm pretty sure its the same problem. Track 9 has an track review, and it contains some UTF-8 characters which may be triggering the same issue. Today, I'll see how difficult it is to try the fall-back workaround I suggested above so at least you might proceed w/out having to wait JRiver's and my conversations.

Edit: added number 4...

4. Yes, you'll need to change how your Credits entry is formatted starting with the next release. Sorry, its just not worth trying to have to code fix it for you, and I think the benefit will outweigh the pain. If you've have not updated your Credits mapping since your last posting of the config file, the samples above are easily reformatted, or send me your current one and I'll return a valid one for you.

stevem87 · « **Reply #99 on:** February 07, 2014, 12:56:06 pm »

1. http://www.allmusic.com/album/back-in-black-mw0000188862 Producer credit (mapped from Credits).

4. No worries, I can do it myself, just wanted to be sure before I did it.

Thanks,
Steve

INTERACT FORUM

Author Topic: amg.pl: an A M G helper and dummy file creator (Read 88431 times)