Topic: amg.pl: an A M G helper and dummy file creator (Read 85091 times)

MrC · « **Reply #100 on:** February 07, 2014, 02:17:13 pm »

Quote from: MrC on February 07, 2014, 12:37:28 pm

1. about the & and " characters (these are HTML entities, used because & and " are reserved characters in HTML) - which fields are you seeing these in, and which AMG URL. I'll check to see where all the conversions might be going wrong. I thought these were all addressed, but obviously not.

There were two bugs here. One, I've addressed. In MCWS mode, I was not decoding the HTML entities before sending them off to MCWS. The MPL mode didn't have this problem, because the HTML entities are required.

[Edit: this bug is now resolved in MC 19.0.116]

The second bug is a bummer. It may be the same, or another, MC bug. MC is not correctly handling the deconstruction of the URL into CSV, and then into text for the field. So even after I've applied my fix above and convert the HTML entities, in the case of double quotes, MC ends up removing them entirely. So an entry:

Robert John "Mutt" Lange

gets placed into the field as:

Robert John Mutt Lange

I've posted in the beta thread, but am not sure it will get addressed for a while. The workaround I've proposed will solve this issue, but will slow down the MCWS writes substantially since I'll have to check every value for double quotes and then fall back.

I'd say, for now, use MPL...

stevem87 · « **Reply #101 on:** February 07, 2014, 04:30:18 pm »

Thanks again, it would seem that the per track pulls (lyrics and song reviews) are going to add the most mileage. I have about 3,000 CDs in my library, should I be concerned? Would spreading them out over time help?

MrC · « **Reply #102 on:** February 07, 2014, 04:36:19 pm »

So others can follow, your comment is in reply to our email exchange about disabling fields you don't need, reducing the number of pages pulled.

You're correct - lyrics and track reviews are expensive. Pull them if you want, and just be reasonable and I think everything will be fine.

I hope you're importing the URL so in the future you can more easily get to the release's page.

MrC · « **Reply #103 on:** February 07, 2014, 08:02:41 pm »

For the next release, I've implemented page caching:

- New: pulled pages are now cached to avoid excessive server hits. The cached directory named '_lwp_cache',
created in the current working director, may be safely deleted at your convenience. One sample run for 10 tracks,
requiring 20 pages which initially took 17.6 seconds, is reduced to 1/5 second for subsequent runs. Page cache
expires as per AMG page cache settings, which is about 8 hours.

SkGe · « **Reply #104 on:** February 08, 2014, 02:01:48 pm »

Question?
If an allmusic link don't have any track listed and i want to grab info from it the scrip will ignore all the track and don't do nothing. Do i need to have track there or what?!

MrC · « **Reply #105 on:** February 08, 2014, 02:08:45 pm »

URL please...

SkGe · « **Reply #106 on:** February 08, 2014, 02:29:42 pm »

Here is an example
http://www.allmusic.com/album/guilty-mw0001632154

MrC · « **Reply #107 on:** February 08, 2014, 02:49:19 pm »

Include the -all option on your command line to force non-track specific (i.e. album-specific) items to be output.

MrC · « **Reply #108 on:** February 08, 2014, 06:48:39 pm »

I've posted version 1.18. There is one incompatible change if you are using Credits mappings. Be sure to see the Credits section in the amg-config.txt file.

I think you're going to like the new page caching and --range options. Try running the script with the same URL a second time.

Version 1.18
   - Change: Credits mappings are swapped: the old pattern:field is reversed to field:pattern. This makes it much
   easier to see your field mappings up front, espeically when using multiple Credits lines, and patterns are long.
   - New: pulled pages are now cached to avoid excessive server hits. The cached directory named '_lwp_cache',
   created in the current working director, may be safely deleted at your convenience. One sample run for 10 tracks,
   requiring 20 pages which initially took 17.6 seconds, is reduced to 1/5 second for subsequent runs. Page cache
   expires as per AMG page cache settings, which is about 8 hours.
   - New: option --range <range list> allows selecting the AMG discs and tracks to be used. This is useful for
   box sets, etc., to select only those items that map to your collection. The --range option implies the --sequence
   option. The <range list> is one or more comma-separated <disc spec><track spec>, where either one may be omitted,
   and <disc spec> is a 'D' followed by a <range>, and <track spec> is a 'T' followed by a <range>. A <range> is
   a number followed optionally by a dash an another higher number (e.g. 3, or 4-9). An omitted <disc spec> or
   <track spec> implies ALL. Examples:

   --range d2 all tracks from disc 2
   --range d2-4 all tracks from discs 2 through 4, inclusive
   --range d2T1-9 tracks 1 through 9 from disc 2
   --range T1-4 tracks 1 through 4 from all discs
   --range D1,D3,D5T1-10 all tracks from discs 1, 3, but only tracks 1 through 10 for disc 5

   - Fix: in MCWS mode, HTML entities were not being unencoded (note: this revealed in MC bug, which is unresolved
   currently).

I've updated the instructions on the main page to indicate that you should change directories into the amg folder you created when you unzip the archive.

stevem87 · « **Reply #109 on:** February 09, 2014, 09:25:56 am »

MrC., Getting an error with the new script. "can't locate LWP/useragent/Cached.pm" ....

MrC · « **Reply #110 on:** February 09, 2014, 01:33:24 pm »

Did you move the amg.pl script out of the folder that contains the LWP folder? The page caching code needs that, so keep the amg.pl, config file, and LWP folder all in the same folder, and execute by CD'ing into the folder (you're probably on the Desktop). In the command shell, instead of:

cd Desktop

do

cd Desktop\amg_1.18

and run as normal.

stevem87 · « **Reply #111 on:** February 09, 2014, 04:25:29 pm »

Thanks, that sounds like it is it.

stevem87 · « **Reply #112 on:** February 09, 2014, 06:37:51 pm »

On this URL http://www.allmusic.com/album/genesis-1983-1998-mw0000490100

I am trying to pull tracks 1, 3, 5, 7, 9, 11-13 off of disc 9. I tried just pulling track 1 from disc 9 (comes in as track 2)

I wrote it like this --R D9T1, D9T3, D9T5, D9T7, D9T9, D9T11-13

What came back was disc 9 tracks 2-9.

tried it lots of different ways and I keep getting the same result.

MrC · « **Reply #113 on:** February 09, 2014, 06:47:09 pm »

Two problems I see...

1. Use -R (single dash for letter options) or --range (double dash with long option names).

2. If you do this:

   --R D9T1, D9T3, D9T5, D9T7, D9T9, D9T11-13

then you need quotes around the argument to protect the white space from the command shell:

   -R "D9T1, D9T3, D9T5, D9T7, D9T9, D9T11-13"

Instead, do this...

  -R D9T1,D9T3,D9T5,D9T7,D9T9,D9T11-13

stevem87 · « **Reply #114 on:** February 09, 2014, 09:46:58 pm »

Thanks, but it pulls in 1-8 now, not the track specified.

MrC · « **Reply #115 on:** February 09, 2014, 09:57:06 pm »

Bah, my algorithm didn't consider these individual tracks in a single disc (I assumed a range). I'll fix it tonight. Thanks.

MrC · « **Reply #116 on:** February 10, 2014, 12:07:28 am »

Version 1.19 is now posted to fix the --range problems you had.

If it will help, I can modify the --range parsing code to allow you to more easily express:

-R D9T1,D9T3,D9T5,D9T7,D9T9,D9T11-13

as this:

-R D9T1,3,5,7,9,11-13

So you'll use comma to separate items within a disc or track range, and semicolon to separate multiple ranges, as in:

-R "D9T1,3,5,7,9,11-13; D10T1-3,5-7"

stevem87 · « **Reply #117 on:** February 10, 2014, 05:46:35 am »

Thank you. Sorry for being such a pain. I know I have been blowing up this board. The comma looks good, less key strokes.

stevem87 · « **Reply #118 on:** February 10, 2014, 03:12:31 pm »

It appears that the -e option is not working now

MrC · « **Reply #119 on:** February 10, 2014, 05:03:00 pm »

I'm not seeing a problem w/-e (aka --exclude), and that code hasn't changed in a while. If you have problems, I'll almost always need to see your command line, and config (strip out MCWS_username and MCWS_password lines) to reproduce the problem, or see where there might be a problem with the command line itself.

I have the new range code done and working.

stevem87 · « **Reply #120 on:** February 12, 2014, 06:34:52 am »

Sorry, I figured out the -e thing. Did you do another release with the range update. I don't see it.

MrC · « **Reply #121 on:** February 12, 2014, 11:15:50 am »

I had not posted the update, as I was waiting for an MC update that fixes the two issues we discussed earlier (MCWS failures and problems with Unicode characters). The beta release that came out last night appears to have resolved the issue. Once that's made available, you should be able to test the cases where you were getting failures.

MrC · « **Reply #122 on:** February 13, 2014, 11:28:42 pm »

Version 1.20 is now posted.

Version 1.20 Changes:
   - Change: --range specs are now a semicolon separated list of individual disc/track pairs, where within a disc track pair,
   disc or tracks may be specified as a comma-separated list. Examples:

   --range "D9T1,3,5,7,9,11-13; D10T1-3,5-7" tracks 1,3,5,7,9, 11-13 for disc 9, tracks 1-3 and 5-7 for disc 10*
   --range D1-3T1-9,11-13,15;T1-3,5-7;D6T1-5 tracks 1-9, 11-13, and 15 for discs 1-3, tracks 1-5 for disc 6,
   and tracks 1-3 and 5-7 for other discs

   * note the double quotes surrounding the entire --range argument, due to use of (optional) white space

   - New: enable connection caching for better performance.

Note: Be sure to update to at least MC 19.0.116, as it fixes the two issues mentioned earlier in this thread (MCWS failures, and loss of double quotes).

stevem87 · « **Reply #123 on:** February 16, 2014, 06:48:51 am »

Thanks MrC, but I can't find 19.0.116, only 19.0.108.

JimH · « **Reply #124 on:** February 16, 2014, 08:11:02 am »

The latest build is always at the top of the MC19 board.

stevem87 · « **Reply #125 on:** February 16, 2014, 02:22:10 pm »

Got it thanks. BTW, I can't seem to get the -R to work anymore. I am also getting some extra lines coming through the script in the command shell. Check the attached file.

Thanks
Steve

MrC · « **Reply #126 on:** February 16, 2014, 04:29:44 pm »

Sorry about that. I'll send you via email a pre-release 1.21 version to see if it resolves the issue.

stevem87 · « **Reply #127 on:** February 16, 2014, 06:44:02 pm »

That did it, thanks again!

MrC · « **Reply #128 on:** February 16, 2014, 06:47:08 pm »

Thanks for testing and the feedback. Version 1.21 posted.

Version 1.21 Changes:
- Fix: fix some error messages when disc number was empty (single-disc albums).
- Fix: set Disc # / Track Pick values to 0 to reflect no disc value (i.e. single disc) / non-track pick. These
will be shown as empty by default in MC for Integer fields.
- Fix: broken logic testing --range values when no disc was specified.

stevem87 · « **Reply #129 on:** February 21, 2014, 03:47:02 pm »

Hey MrC, I keep getting a error on this URL: http://www.allmusic.com/album/try-this-mw0000325851
The script say that there is no main HTML page available.

MrC · « **Reply #130 on:** February 21, 2014, 04:03:43 pm »

I'm not having any problems with that link. I wonder if your cached page is corrupt? You could delete the cached pages, which are under _lwp_cache.

MrC · « **Reply #131 on:** February 21, 2014, 10:42:07 pm »

I've just posted a large update to version 1.22. Most of the code has changed, to make it as much the same as possible with discogs.pl script. There are also some nifty new features here. Enjoy.

Version 1.22
- New: option --create will create dummy files (useful for tracking vinyl).
- New: amg.pl now iterates over all urls on the clipboard so multiple albums can be updated, or on the command line when
creating dummy tracks.
- Change: remove obsolete debug option --clipfile.
- Change: fieldname 'url' changed to 'rid' (to align with discogs.pl). Option --rid, --url, and -u are equivalent
(but not -r, which is reserved for --rating).
- Fix: a disc-only --range spec (e.g. D2) not working.
- Internal: amg.pl, discogs.pl, and pscriptor.pl now share common modules (under directory MCUtils).
- New: be tolerant of uppcase letters in key names used on the command line

stevem87 · « **Reply #132 on:** February 22, 2014, 03:03:22 pm »

Clearing the cache worked, thanks!

MrC · « **Reply #133 on:** February 22, 2014, 03:14:40 pm »

I saw an occurrence of this last night too, and clearing the cached fixed the issue. I'll dig deeper next time it happens.

By the way, I just implemented:

- New: key 'totaltracks' mapped by default to Total Tracks returns the total number of tracks across
all discs.

MrC · « **Reply #134 on:** February 22, 2014, 10:33:56 pm »

I've posted version 1.23:

Version 1.23
- New: key 'totaltracks' mapped by default to Total Tracks returns the total number of tracks across
all discs.
- New: warn when an --include or --exclude key does not exist.
- Fix: do lazy pull-parsing of per-track info.
- New: debug and verbose messages are easier to read.
- Internal: more cleanups and bug fixes.

SkGe · « **Reply #135 on:** February 23, 2014, 09:45:54 am »

This script become better with each release thx you for spending the time doing this for us.

SkGe · « **Reply #136 on:** February 24, 2014, 03:41:09 am »

Failed to get page: http://www.allmusic.com/album/razorblade-suitcase-mw0000080628/releases
500 read timeout
Use of uninitialized value $_ in pattern match (m//) at amg.pl line 876 (#1)
   (W uninitialized) An undefined value was used as if it were already
   defined. It was interpreted as a "" or a 0, but maybe it was a mistake.
   To suppress this warning assign a defined value to your variables.

   To help you figure out what was undefined, perl will try to tell you
   the name of the variable (if any) that was undefined. In some cases
   it cannot do this, so it also tells you what operation you used the
   undefined value in. Note, however, that perl optimizes your program
   anid the operation displayed in the warning may not necessarily appear
   literally in your program. For example, "that $foo" is usually
   optimized into "that " . $foo, and the warning will refer to the
   concatenation (.) operator, even though there is no . in
   your program.
This error appears on some albums after i try to get info, but if i try to retry the information again, maybe the site isn't responde and reach that 500 read timeout it show me the same info. I suppose that info what he got from that link is stored somewhere and wont do again a check, like before.
I checked that link with release and seems to have info there but the script don't want to recheck again and instead it pop up this message. How to deal with that?! If i close the cmd and open again the same message pop up appear.

SkGe · « **Reply #137 on:** February 24, 2014, 03:59:08 am »

I recheck and read the readme so i deleted the cache file from inside of folder and seems that have solve the problem.
But that mean every time when the site doesn't respond in that 500 read timeout i will need to delete the cache? or can be done in another way?

MrC · « **Reply #138 on:** February 24, 2014, 11:48:11 am »

I think the module I'm using for caching has a bug, and I'll dig into that later. For now and until I find it, clearing the cache works.

Edit: I may have a fix for this. Tonight I tagged almost 2400 files before I finally hit the cache error. I applied my fix, and the the problem did not reappear.

Coincidentally, when the cache issue hit, I was tagging a Buckwheat Zydeco album 100% Fortified Zydeco, and this triggered an MC bug with a filename containing a % character. I've posted the problem in the beta thread.

SkGe · « **Reply #139 on:** February 25, 2014, 07:18:32 am »

Another issue, on some albums some tracks have [cd-rom track] as name, and if you didn't check your tracks with the link you get info, that track is rewrite as [cd-rom track] can this be avoided?

MrC · « **Reply #140 on:** February 25, 2014, 11:45:26 am »

As always, I need a URL to work out the details.

SkGe · « **Reply #141 on:** February 25, 2014, 01:38:30 pm »

Here is the link with the thing i told you and did something change on date type because it only get as year and nothing more.
I use the same cmd as i did before and with the new changes.
http://www.allmusic.com/album/from-the-roots-up-mw0002377235

MrC · « **Reply #142 on:** February 25, 2014, 01:56:20 pm »

I've posted version 1.24:

Version 1.24
- Fix: don't cache any 5xx responses. This may help the problem users were seeing with 500 replies to
good page requests.
- Fix: the Date field was returning year only (cause by an incomplete debugging cleanup from 1.23).

Regarding the [CD-ROM Track] track name, I think that's pretty rare, and esoteric, so unless we find a bunch of these, I'd prefer not to special case this one this single name. What you should do is use the --range option, to indicate you want tracks 1-12:

--range T1-12

SkGe · « **Reply #143 on:** February 25, 2014, 02:29:03 pm »

Thx for the response, i suppose is pretty rare, it occur to me 2 or 3 times all this time so i will do a check of tags before i grab the data. (Did you do something with my request on the other thread?).

MrC · « **Reply #144 on:** February 25, 2014, 03:02:28 pm »

Quote from: sirgeo on February 25, 2014, 02:29:03 pm

(Did you do something with my request on the other thread?).

Which one? You leave me guessing. :-)

- The cache error might be resolved.
- The date problem is fixed.

The release notes above for 1.24 say this.

SkGe · « **Reply #145 on:** February 25, 2014, 03:07:36 pm »

Seems we have a miss communication

I did ask you if you can help me with the request in the other thread about albums playcount, you didn't answer so i presume or you don't read or don't understand my thoughts

The last version is checked and works ok so far. The date is ok.

stevem87 · « **Reply #146 on:** February 27, 2014, 08:39:26 am »

I have a question, why does the 'Date' field only populate with the year. Even though the full date is shown under 'Release Date' on the web page. I seem to remember that the script used to populate the full date, but recently it is not.

MrC · « **Reply #147 on:** February 27, 2014, 12:06:55 pm »

See reply #142.

MrC · « **Reply #148 on:** February 27, 2014, 06:40:56 pm »

I want to give you folks a heads up. I've added a couple of new fields, as I've learned a little bit more about AMG track, artist, and album IDs.

Currently amg.pl wants an entire AMG URL, which is a lengthy string that includes the base URL, the album's name, and an album identifier:

   http://www.allmusic.com/album/morning-phase-mw0002607867

However, the last part, the mw0002607867, is sufficient to generate a URL to the same page. By combining the base with the Album ID:

   http://www.allmusic.com/album/ + mw0002607867

you get a simple, unique Album URL. As such, I've added the capability to save the Album ID portion alone. In addition, I'm also providing the Artist ID and the Track ID (so you can create Links to go directly to the Artist or Track page directly). The new fields that will be available are:

   url => AMG Album URL
   artistid => AMG Artist ID
   artisturl => AMG Artist URL
   tid => AMG Track ID

The existing fields will still be available:

   rid => AMG Album ID
   turl => AMG Track URL

However, and here's the Heads Up. The script will only read it's 'rid' value from the 'rid' field, which currently contains a full URL. And from now on, it will always store the shorter album ID into the 'rid' field and the full URL into the 'url' field. So if you want to save your full URLs, you'll need to create a new field and copy the URLs to that field. The script won't care if the value in the 'rid' field is a full URL or an ID - either will work.

Example: My 'rid' field is currently named AMG URL. Since the script will now store an ID into that field, and I want to keep the URLs (for the time being anyway), I'll create a new MC field called AMG ID. I'll copy the AMG URL values for all my files into AMG ID (using MC's Move/Copy fields tool). And I'll edit my amg-config.txt file to assign 'rid' to the field name AMG ID, and the new 'url' key will be assigned to AMG URL. New album IDs will be stored in 'rid' and the full url will be stored into the 'url' field.

Got it?

Side note: I've personally processed over 14k files.

Edit: I've now processed all my files. Today I added composer id/url to the script. With all these ID or URL fields available, you can create some nifty direct album, track, artist, and composer Links in MC to take you directly to the relevant page. I have, and am routinely using, MC Links for AMG Album, Track, Artist, and Composer data, which use the stored IDs, and if they don't exist, do a search on AMG.

MrC · « **Reply #149 on:** March 04, 2014, 11:10:25 pm »

Quote from: stevem87 on February 04, 2014, 07:41:35 pm

I got another one..or two. Is there a way to stop the script from changing my cover art. I have the artwork set and embedded inside the file. After running the script it changes it to external location.

Just to follow-up on this. This thumbnail deletion problem is an MC bug which is fixed in an upcoming MC19 beta release. Stay tuned.

Edit: MC version 19.0.120 is posted. It solves the cover art problem.

INTERACT FORUM

Author Topic: amg.pl: an A M G helper and dummy file creator (Read 85091 times)