INTERACT FORUM

Windows => Third Party Plug-ins, Programs, and Skins => Topic started by: MrC on January 27, 2014, 12:17:35 am

Title: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on January 27, 2014, 12:17:35 am
Description

The following command script pulls metadata from Discogs to tag your Media Center tracks.  In addition, it can also create dummy files that can be imported and tagged to simulate collections for which physical media does not exist (e.g Vinyl, desired CDs, etc.).

The script is written in Perl, and can be used on Windows, Mac OS X, or Linux.  It basically works like this:

    1. You supply an album's Discogs release ID
    2. You select the album's tracks in a view in MC
    3. You run the script

The script will do one or both of:

  - update MC fields directly via MCWS (version 19.0.54+, Media Server needs to be running with authentication enabled)
  - produce an MPL file which will update the MC fields when imported into MC via File > Import Playlist

Available Metadata

The following metadata can be retrieved (when available):

    Per-Album Values
        Album Artist
        Album
        Genre
        Styles
        Country
        Date
        Date Released
        Label (combined label and catalog #)
        Disc #
        Discogs Release ID
        Discogs Release URL
        Media Format (Vinyl, CD)
        Credits
        Notes  (coming soon)
        Barcode
    Per-Track Values
        Artist
        Track #
        Name
        Credits
        Duration (currently disabled)

Download the attached zip file, and unzip the items to a folder (right-click > Extract All).  See the README.txt file inside the folder.

Here are several videos showing the script in action.
- See the Demo movies at the download site in the Instructions

For best results, download the files and view in your preferred player.

See also this thread:

   http://yabb.jriver.com/interact/index.php?topic=86465.msg594479#msg594479 (http://yabb.jriver.com/interact/index.php?topic=86465.msg594479#msg594479)
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on January 27, 2014, 12:23:18 am
Notes

Implemented, will be in the next release:

Future:
   - Support other Discogs searching mechanisms if possible.
   - Complete the merge with amg.pl (much of the code is shared now).
   - Download cover art from Discogs (this is implemented, but won't work as discogs changed their API terms to require authentication).
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on January 27, 2014, 12:23:24 am
Usage Tips

Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: cinjin on January 28, 2014, 01:22:55 am
MR C Thanks for all the work you put into this. I have been using musicbrainz then jaikoz then mp3tag to completely tag my files. This is gonna greatly reduce the time it took. A great time saver Thanks !!!
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on January 28, 2014, 06:52:10 pm
Thanks.  I'm interested in getting feedback.

You may have also noticed I completed the cover art download work (noted above in post #2), and was hoping for a few bug reports that I could fix before releasing 1.01.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: broncodan on January 28, 2014, 11:14:09 pm
I am very interested in this but not sure where to start - I guess I need to get the discog's album id?  Is there anyway to get this data automatically?  I have quite a large library and doing this manually would take weeks?

I saw on another post about this discogs info that you were extracting this data from the discogs.xml.  One thought for me would be able to get that XML data into an excel file (for me I would only be looking to get the genre/styles/original release date (song and/or album) and the album/artist information to search from.  With that information in a spreadsheet I could export my library to excel with the blank fields created, run a macro in excel to compare album/artist between the 2 sheets, if there is a match - copy the information to the blank fields.  Use the track info lookup excel spreadsheet created by Just Some Guy - which can export excel data to MPL - then reimport the mpl and have the information added to the library.

This also prevents hitting the discogs server with a lot of pings (which in the past has gotten my ip banned).... :)

Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on January 28, 2014, 11:40:25 pm
There's no automatic way to get your Release ID for an album.  Rather, there are many releases for any given album, sometimes more than a hundred.  For example, check out Pink Floyd's Dark Side of the Moon:

   http://www.discogs.com/Pink-Floyd-The-Dark-Side-Of-The-Moon/master/10362

Each of the items there is a Release, and one release may be different from another.  While many have essentially the same track lists, each has a different quantity and quality of credits and extra meta-information (release date, label, remastering, etc.), and of course the Catalog #.  And some releases have extra tracks, extended versions, and different packaging, and obviously different media formats (CD, SACD, LP, 8-Track, Cassette).

There is the concept of a Master Release, which is a bit of an overarching entry with a track list, year, genre, but little other metadata.

Now, if you wanted a way to, say, return the first 'CD' release, and didn't really care about the other metadata beyond track info, I could have the script query by Artist/Album Name/Format, and just accept the first release found that matches the criteria (and number of tracks probably), and ignore assigning unverified information (Catalog #, Country, Label, Release Date, etc.).  Or perhaps the list can be narrowed and more refined with your own good metadata (such as Date, Country, Label, or whatever).

And there can be more problems, for albums where exact title matches are difficult, especially with Greatest Hits albums, where there might be many different "Greatest Hits" masters with entirely different tracks.

So, in summary, with only two pieces of metadata (Artist/Album), results will be less than spectacular.   With better metadata, results should theoretically be better.  But at some point, if you have perfect metadata, you don't need the script, and I'm not sure where that balance is yet (and we probably don't know until user requests come in).

My approach to this type of stuff is to get the data when I play an album (cover art, extra metadata, lyrics), as it is few second process, for each 60 minutes of playback, roughly.  But I'm open to suggestions.

I would suggest you start by trying it out with a test album or two, and see what you need and want, and how well it works for you.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: lepa on February 03, 2014, 12:37:13 pm
Thanks MrC. I still haven't tested the script but was wondering could this be used through MC's send to external program? If the (album) files have discogs id already filled could I use that to fill all the other metadata? That would be convenient.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on February 03, 2014, 01:00:35 pm
Thanks MrC. I still haven't tested the script but was wondering could this be used through MC's send to external program? If the (album) files have discogs id already filled could I use that to fill all the other metadata? That would be convenient.

That's OK, at your leisure.

You could use Send To, but it isn't designed for this, and here's why.  Send To operates one file at a time, and will invoked the external command for every file you have selected.  And it will leave a command window open for every file you've selected, which you'll have to close.  Send To's paradigm is rudimentary and simple; it wasn't designed for batch or multi-file operations.

But discogs.pl operates on all the files at once, and needs them all to determine certain sequencing and minimize calls to the Interweb.

So, bad idea.

You'll have a much better experience if you just create a batch file (or chose your own scripting language) which launches a command window and invokes discogs.pl with the proper arguments and leaves the command window open for you to review any diagnostics.  Then, all you need to do is double click the batch file when you are ready to do some lookups on the selection in MC.  Ask if you need help here.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: lepa on February 19, 2014, 02:53:42 pm
Just tested with one album. Few humble suggestions reflecting my needs.

Could Label and Catalog # be separated. Currently both are written to Catalog # field.
I've been using Label = Publisher in MC and Catalog # = Catalog #

The other one would be Composer field in MC.
Could discogs 'Written By' / 'Composed By' written to MC's Composer field (; as a separator).
This data is now written to Credits field it seems. I guess I could use the scripting thing you released but it would be nice if this script could fill the composer field automatically.

Thanks again for the script.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on February 19, 2014, 02:58:07 pm
Great, thanks.

Re: Catalog # and Label.  I thought about this.  If I recall correctly, there are some entries which have more than a single Label/Catalog listing, and I wasn't sure how users wanted those mapped.  Suggestions?

Re: Composer, etc.  I haven't migrated the Credits mappings yet that the amg.pl script has.  I'll do that.  I'm in the process of a grand merger, so discogs.pl, amg.pl, and pscriptor.pl will share all the common code, and amg.pl will pick up capabilities that discogs has, and vice versa.  Hold tight on this...
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: lepa on February 19, 2014, 03:22:44 pm
Re: Catalog # and Label.  I thought about this.  If I recall correctly, there are some entries which have more than a single Label/Catalog listing, and I wasn't sure how users wanted those mapped.  Suggestions?
This is tough one. I've got no killer suggestions. I guess I have handled this case by case when there has been multiple label information.

IMO for re-releases you should use the re-release company. e.g. Led Zeppelin Classic Records vinyl releases should use that as a publisher and not Atlantic (just my opinion of course)

That said I'm not sure if discogs have some kind code of conduct for that. Should the actual releaser be added as top of label list or is there any ruling to how list these....?

It looks like it could be like that:
http://www.discogs.com/Led-Zeppelin-Led-Zeppelin/release/2154266
http://www.discogs.com/Led-Zeppelin-Led-Zeppelin/release/1125921

Other solution could be taking them all and separate with ;
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on February 19, 2014, 03:28:22 pm
Where does the re-release company. "e.g. Led Zeppelin Classic Records vinyl" come from on the page?
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: lepa on February 19, 2014, 03:38:26 pm
It is just one of the Label fields in xml.

Judging with that one example it would suggest that first label field is the releaser of the release in concern.

Like in http://www.discogs.com/Led-Zeppelin-Led-Zeppelin/release/2154266
First label - catalog # pair says "EastWest Japan ‎– AMCY-2431" which is the company behind the reissue.

and in http://www.discogs.com/Led-Zeppelin-Led-Zeppelin/release/1125921
First label - catalog # pair says "Classic Records ‎– SD 8216" which is the company behind this reissue.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on February 19, 2014, 04:08:15 pm
Oh, that you already have.  I see, your "Led Zeppelin Classic Records vinyl" was literally just an example; I was confused, because the pages you reference are specific.

So you just want the first one, and you want the first one split as Label and Catalog #.  Right?
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: lepa on February 20, 2014, 08:21:22 am
Right. Might go wrong sometimes depending of discogs editor but maybe it would be too messy to add all of them and i suppose most of times that first entry identifies the release quite well. Of course one can always check release from the discogs also if id is saved to the tags.
I've been using publisher tag (because it was already there) in mc for discogs labels but user defined label field would be fine also.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on February 21, 2014, 12:36:33 pm
FYI: Yesterday I finished the album credits mapping, and the change to return the first catalog and label, storing them into their own fields (Catalog # and Publisher).  I'm finishing up a few more items, and then I'll publish the update.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on February 21, 2014, 10:43:32 pm
I've just posted a large update to version 1.02.  Most of the code has changed, to make it as much the same as possible with the amg.pl script.  There are also many nifty new features here.  Enjoy.

Version 1.02
    - Fix: forgot to bump version number for 1.01
    - Fix: empty release years, or those that contained month and day values were not handled correctly.
    - Internal: amg.pl, discogs.pl, and pscriptor.pl now share common modules (under directory MCUtils).
    - Fix: problem mis-detecting old clipboard data when last column in view was empty.
    - Change: field key 'label' deprectated and split into two new field keys 'catalog' and 'publisher'.
    - Change: only the first (primary) Catalog # / Label is returned via the new field keys 'catalog' and
      'publisher'.
    - Change: key 'credits' mapped to 'Album Credits' now by default, since MC's Credits field is Video-Only.
    - New: credits mapping to selectively map credits to other MC fields, or ignore certain credits.  See the
      Credits section in the config file.
    - New: the Discogs Notes section (key: 'notes') is now available and mapped to MC's Description field.
    - Fix: a missing Filename column was sometimes not detected.
    - New: be tolerant of uppcase letters in key names used on the command line.
    - New: option --fileregex added, which allows pulling track numbers from filenames.
    - New: option --all will add album info for tracks not in Discogs (useful for extra-track disks, etc.).
    - New: option --range <range list> allows selecting which discs and tracks are to be used from the release.
      This is useful for box sets, etc., to select only those items that map to your collection.  The --range
      option implies the --sequence option.  The <range list> is one or more semicolon separated lists of
      individual disc/track pairs, where within a disc track pair, disc or tracks may be specified as a comma-
      separated list.  Either the Disc or Track portion may be omitted to indicate ALL items.  Examples:

        --range d2                                                       all tracks from disc 2
        --range d2-4                                                    all tracks from discs 2 through 4, inclusive
        --range d2T1-9                                                tracks 1 through 9 from disc 2
        --range T1-4                                                    tracks 1 through 4 from all discs
        --range "D9T1,3,5,7,9,11-13; D10T1-3,5-7"       tracks 1,3,5,7,9, 11-13 for disc 9, tracks 1-3 and 5-7 for disc 10*
        --range D1-3T1-9,11-13,15;T1-3,5-7;D6T1-5     tracks 1-9, 11-13, and 15 for discs 1-3, tracks 1-5 for disc 6,
                                                                               and tracks 1-3 and 5-7 for other discs

        * note the double quotes surrounding the entire --range argument, due to use of (optional) white space
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: lepa on February 22, 2014, 02:08:52 am
Nice! Thanks MrC.
So I can use --all parameter to add album info for example to the album art files? Of course I could just test it...  ;D
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on February 22, 2014, 02:12:09 am
Yes, and that's an obvious usage that didn't occur to me.  Just make sure the art is last in the album grouping.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on February 22, 2014, 10:32:40 pm
I've posted a new version 1.03.

Version 1.03:
    - New: key 'totaltracks' mapped by default to Total Tracks returns the total number of tracks across
      all discs.
    - New: warn when an --include or --exclude key does not exist.
    - New: debug and verbose messages are easier to read.
    - Internal: more cleanups and bug fixes.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: lepa on February 23, 2014, 01:41:03 pm
Working very well and is extreme time saver :)
Dare I i still ask for fetching of the Barcode and other identifiers...
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on February 23, 2014, 02:28:53 pm
Sure.  I have to modify the WWW::Discogs module.

There can be many different identifiers:

    Matrix / Runout: 10 C 268-098434
    Matrix / Runout: 10 C 268-098.434
    Other (Depósito Legal): B. 5695-1977
    Rights Society: S.G.A.E.

    Matrix / Runout (Side A Label): MX178307
    Matrix / Runout (Side B Label): MX178308
    Matrix / Runout (Side A Runout Groove): ' A SBP234948 A 1359-700
    Matrix / Runout (Side B Runout Groove): ' A SBP234948 B 1360-700

    Barcode: 4 988009 604220
    Matrix / Runout: DPA-19 2
    Rights Society: JASRAC

    Barcode: 7 24382 97482 6
    Other (Distribution Code): F: PM 520
    Other (Distribution Code): I: 070
    Other: CDEMD 1060
    Matrix / Runout: teltron 7030-2 8297482 INDUSTRIA ARGENTINA

Maybe you just want Barcode?

[Edit: I have the barcode being returned:

    - New: key 'barcode' mapped by default to MC field Barcode returns the release's Barcode, if it exists.
]
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: lepa on February 26, 2014, 04:02:42 pm
Yes,I think that barcode is enough.
Other info may or may not be correct for the album I've got and it is impossible to automatically choose the correct deadwax info from the discogs when there are variants.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on February 26, 2014, 05:29:51 pm
Version 1.04 is posted.

Version 1.04 Changes:
    - New: key 'barcode' mapped by default to MC field Barcode returns the release's Barcode, if it exists.
    - Fix: some more debugging message cleanups.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jctcom on March 01, 2014, 05:56:05 pm
2 questions (For now).

1st.  I do no know anything about Perl (or other programming languages) but I am very interested in this.  I have downloaded the package and edited the discogs-config.txt file to match and / or add the fields missing or with a different name in MC.

I have read through the thread and can't find anything that actually tells me how to run this package and load the Discogs Catalog # and basically us it?

2nd.  Having gone through the thread and the discogs-config.txt file I didn't see any mention of "Original Release Date".  Is this pulled from discogs and implemented somewhere?

Thanks and sorry for being such a noob in this regard.

Carl.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 01, 2014, 07:05:53 pm
1st.  I do no know anything about Perl (or other programming languages) but I am very interested in this.  I have downloaded the package and edited the discogs-config.txt file to match and / or add the fields missing or with a different name in MC.

I have read through the thread and can't find anything that actually tells me how to run this package and load the Discogs Catalog # and basically us it?

No need to know about perl.

The README.txt file gives you the steps.  Have you looked through that file?

2nd.  Having gone through the thread and the discogs-config.txt file I didn't see any mention of "Original Release Date".  Is this pulled from discogs and implemented somewhere?

There are two dates that can be pulled - the release date and a year-only value of the date.

On the top of a discogs release page, you'll see a "Released" field, and the date there may be a year, year/month, or year/month/day value, as a string.   This will be stored in the MC field Date Issued, by default (this would be a user-defined field).  You would see this as the 'rdate' key in the config file.

The year-only value, seen on a page as "Year", gets assigned to MCs Date field.  The "Year" value is typically seen at the top of a discogs Master page.  This is the 'date' key you'll find in the config file.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jctcom on March 01, 2014, 07:34:55 pm
No need to know about perl.

The README.txt file gives you the steps.  Have you looked through that file?
I did read the file.  I guess there is either something I missed or something I don't understand.

What I believe are the relevant parts of the TXT file:
The script is written in Perl, and can be used on Windows, Mac OS X, or Linux.  It basically works like this:

    1. You supply an album's Discogs release ID   
 (If "Run The Script" isn't the first step then I am not sure where to "supply" the Discog's Info to? unless it is simply entering it into the "Catalog #" field for the album which I already do.
    2. You select the album's tracks in a view in MC
I can do this!  lol
    3. You run the script   
How does one "Run The Script".  Do I just double click on the "discogs.pl" file? When I do that I get "dos" type window that opens and closes instantly.  (I have downloaded and installed the "ActivePerl 5.16.3...")

There are two dates that can be pulled - the release date and a year-only value of the date.

On the top of a discogs release page, you'll see a "Released" field, and the date there may be a year, year/month, or year/month/day value, as a string.   This will be stored in the MC field Date Issued, by default (this would be a user-defined field).  You would see this as the 'rdate' key in the config file.

This part I have done.  I changed the discogs-Config.txt field to match mine which is "ReleaseDate".


The year-only value, seen on a page as "Year", gets assigned to MCs Date field.  The "Year" value is typically seen at the top of a discogs Master page.  This is the 'date' key you'll find in the config file.

Ok.  When I was going through the discogs-config.txt file originally for some reason I missed the "Date" field.  But I see it there now.

So I guess I am good to go.  except for the part about "Run The Script" lol

Carl
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 01, 2014, 09:25:39 pm
The part you've quoted is from the top, introductory section.  It's just an overview of what you'll be doing.  The real meat is in the various Usage sections.  It really does walk you through the process, including how to find the discogs release ID.  Search "Usage: Common":

Quote
Usage: Common

* To see alist of command line options supported by discogs.pl, use the --help option:

    perl discogs.pl --help

* The discogs.pl script and the following usage scenarios require a Discogs Release ID.  To find this value, go to www.discogs.com, and search for an Artist.  Navigate into the artists discography listing, and select an album.  There will be one or more album releases; select the one you believe is yours (match your catalog #, or label, or use some generic entry if you don't care for that level of detail - just be sure the release has the metadata you want).  Once you are on the release page for the album, you'll see the Discogs Release ID as the last component of the URL, and also in the upper right corner of the page with a little record icon (but you only want the numeric part).
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jctcom on March 01, 2014, 09:58:24 pm
Oops.

Got to the "Download and install ActivePerl 5.16" and for some reason after doing that I thought I had been at the end of the file.

Sorry my bad.

I will continue and report back. 

Carl.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jctcom on March 01, 2014, 10:40:55 pm
Ok so I have completed the steps and performed my first test on the Jimi Hendrix Album "Hendrix In the West" with Discogs Release ID: 694011

All went well except for the Date.  All of my Artists / Albums have always had the "Year" in the upper right which is what they are sorted on.  This "Year" has always been the "Original Release Year" of the album.  In this case "1971".  

This process pulls and puts the "Current Release Year" (1980 in this case) in that space as it appears on the release for the version of the album attached to that discogs ID.

What discogs field can I use to pull the "Original Release Year" that appears when you click on the "All Versions of this release".  This is the year I would want to sort on.  I don't want an album that was re-released in 2010 to appear at the bottom of my list when the original album was released in 1970.

I have attached my discogs-config.txt just in case.  But I think it is correct with Field: Date = Date?
Thanks.

Carl.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 01, 2014, 10:52:36 pm
I'll probably have to make a second call to get the master release date.  Stay tuned.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jctcom on March 01, 2014, 10:54:53 pm
Thanks. That will be great.

Carl.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 01, 2014, 11:46:24 pm
Here's a pre-release of 1.05.  It pulls the date from the master entry, so should be correct.  I haven't put a lot of testing into this yet.  Unzip the attachment, and replace your discogs.pl script.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jctcom on March 02, 2014, 12:29:33 am
Thanks. now it is putting the correct date in.

Is there an easy way for me to have it not put in the track numbers?  With my Vinyl stuff I have created an additional field which I put in manually A1, A2, B1, B2, B3 etc...

But I like the actual track numbers to go numerically in order from 1st to nth.

If it's a problem or will create grief for others who might want to use the program then don't worry about it.  I will just do the "Fill track numbers automatically. deal.

But so far so great!

Carl.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 02, 2014, 12:46:48 am
Add the options:

    --sequence --nodiscnum

to your command.  The --sequence option uses the order listed in the view, and the --nodiscnum option says to ignore the disc name/number and just count upwards 1 to n for track numbers.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jctcom on March 02, 2014, 12:53:11 am
Thanks.  I will try that presently.

Once I have this command fixed can I just create a .bat file and run that?  Is there a way for me to tell it the discogs ID that way?

Also I wanted to add the "Artist" name into my "Album Artist" field since 95% of them are the same.

I added the line: Field:  artist   = Album Artist     ("Album Artist" of course being the name of my "Album Artist" field) but it didn't work.  What did I do wrong?

Carl
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 02, 2014, 01:08:48 am
Most of the command line options you can add to the config file.  See the Options section.  Obviously, you can't add the discogs ID, nor the -c discogs-config.txt options.  However, I find it faster to select all the tracks, get the release ID, and add that value to the selected files in MC.  Then, you don't need the --rid option at all, since discogs.pl will grab it from your selection.

If you want to map the 'artist' key to Album Artist, then just change the single artist field entry in the config file.   It won't work to add additional Field entries with identical keys - only the last one would be in use anyway.  So, change the existing:

   Field:  artist          = Artist

to:

   Field:  artist          = Album Artist
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jctcom on March 02, 2014, 01:15:55 am
Cool.  So then I could run it as a batch file once I've added the discogs ID?

I think since the Artist Field is usually correct at the time I am tagging I might just go ahead and change the artist to my "album artist" field.

So can you get this to read my mind and fill out all my other custom fields!?  lol

Carl.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 02, 2014, 01:22:22 am
Right, a batch file would work fine.

I'm working on the --mindread option.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jctcom on March 02, 2014, 02:56:26 am
I get confused about "Publisher", "Copyright" and "Label".

Recently I had made a decision for my database to use the definitions as listed on Discogs.  But they seem to be different on different sites.  However the "Label" is usually pretty consistent.

I noticed that you seem to pull the "Label" into the "Publisher" field.  Are they generally considered the same?  Because they are not always listed the same on Discogs.  I think I may even have created my own "Label" field because I don't think it existed.

For instance on this release:  http://www.discogs.com/Jimi-Hendrix-Blues/release/2521556

Where the label is "Classic Records" and the Publisher is listed as "Experience Hendrix, L.L.C.".

In this particular case I know it would not pull the Publisher per say because it is in the "Notes" and not in the Credits spot but I have seen others that list the Publisher in the credit's spot and it is not the same as the "Label".

I guess what I am wondering is on Discogs when there is a proper listing for "Publisher" under the Credits section does the "Publisher" field not have a separate designation from the "Label" field?

Thanks.

Carl.

Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jctcom on March 02, 2014, 03:45:42 am
Hi MrC.

I think I have discovered an issue for Multi-Disc albums where they have headers included headers included such as this one:

http://www.discogs.com/Jimi-Hendrix-And-Gypsy-Sun-Rainbows-Live-At-Woodstock/release/2562147

The headers seem to be imported as track titles.

Carl.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: lepa on March 02, 2014, 05:49:21 am
However, I find it faster to select all the tracks, get the release ID, and add that value to the selected files in MC.  Then, you don't need the --rid option at all, since discogs.pl will grab it from your selection.
So could I use MC's send to external with no parameters i.e. no [filename]. I would just select the files and send to external which would launch the bat file
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: JimH on March 02, 2014, 07:28:41 am
Right, a batch file would work fine.

I'm working on the --mindread option.
The delay is unacceptable.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 02, 2014, 12:57:07 pm
I get confused about "Publisher", "Copyright" and "Label".

Recently I had made a decision for my database to use the definitions as listed on Discogs.  But they seem to be different on different sites.  However the "Label" is usually pretty consistent.

I noticed that you seem to pull the "Label" into the "Publisher" field.  Are they generally considered the same?  Because they are not always listed the same on Discogs.  I think I may even have created my own "Label" field because I don't think it existed.

Copyright is a legal concept that determines the ownership rights of a work.  Only the Copyright holder has the authority to define how a work may be used, published, modified, etc.  I see no practical value in capturing and saving this value into MC.

Label is the name of the company that has been given the rights to recreate and mass-produce a copyrighted work.  The label company may change one or more times over the life of a copyrighted work.   This field (along with the release date) is useful to capture, as it helps identify the specific release of the work, especially since over time the work may be released in many different forms of packaging and variations in content.  Unfortunately, label companies get consumed by other label companies, and the same packaging release may have two different label names.

Publisher is not so clear to me in the context of music releases.  Since works are primary identified by label and catalog number, it seems not important.  Given the dearth of publisher information in discogs releases, it seems unimportant to others as well.

I used the Publisher field in MC, because there is no stock Label field.  You are free to configure your own field name in the config file.

For instance on this release:  http://www.discogs.com/Jimi-Hendrix-Blues/release/2521556

Where the label is "Classic Records" and the Publisher is listed as "Experience Hendrix, L.L.C.".

In this particular case I know it would not pull the Publisher per say because it is in the "Notes" and not in the Credits spot but I have seen others that list the Publisher in the credit's spot and it is not the same as the "Label".

Media companies are very interested in getting their names stamped on everything they can, usually with no practical consumer benefit or value.

I guess what I am wondering is on Discogs when there is a proper listing for "Publisher" under the Credits section does the "Publisher" field not have a separate designation from the "Label" field?

You can map the credits to whatever field you want.  Since Publisher is being used by default for the label, you might not want to use that field (or change the label mapping).  I configure the credits mapping for publisher to 'ignore'.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 02, 2014, 12:57:45 pm
Hi MrC.

I think I have discovered an issue for Multi-Disc albums where they have headers included headers included such as this one:

http://www.discogs.com/Jimi-Hendrix-And-Gypsy-Sun-Rainbows-Live-At-Woodstock/release/2562147

The headers seem to be imported as track titles.

Oh, that's a curious one.  I'll take a look and post an update as soon as I can.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 02, 2014, 01:03:05 pm
So could I use MC's send to external with no parameters i.e. no [filename]. I would just select the files and send to external which would launch the bat file

I replied to the same basic question in #8 above.  MC's Send To calls the external program for EACH track.  It does not operate in a batch mode.  So, for a 16 track selection, MC will call the program you've defined 16 times, regardless of what parameters you supply in your Send To command.  Now, if MC had a Send To Batch command, that would be different.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 02, 2014, 01:07:39 pm
The delay is unacceptable.

I'm fired.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: lepa on March 02, 2014, 01:20:04 pm
I replied to the same basic question in #8 above.  MC's Send To calls the external program for EACH track.  It does not operate in a batch mode.  So, for a 16 track selection, MC will call the program you've defined 16 times, regardless of what parameters you supply in your Send To command.  Now, if MC had a Send To Batch command, that would be different.
Ok thanks and sorry, should have read your previous answer more carefully.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 02, 2014, 01:37:45 pm
Hi MrC.

I think I have discovered an issue for Multi-Disc albums where they have headers included headers included such as this one:

http://www.discogs.com/Jimi-Hendrix-And-Gypsy-Sun-Rainbows-Live-At-Woodstock/release/2562147

The headers seem to be imported as track titles.

Carl.

Ok, here's your update (unzip and replace discogs.pl).  I was not aware that the tracklist might contain type = 'header' information, so this is ignored now.  I can see this information might be useful; we'll see if anyone wants it.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: JimH on March 02, 2014, 02:13:03 pm
;)
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jctcom on March 02, 2014, 02:43:42 pm
Ok.  I will have to look for another album with Headers on Discogs that I need to update.  I will post back when I get there.

Carl.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jctcom on March 02, 2014, 08:04:32 pm
Found another anomoly for you.  Not sure how you would tackle this though.

If you look at the following release:

http://www.discogs.com/JS-Bach-Yo-Yo-Ma-The-Six-Unaccompanied-Cello-Suites-6-Suiten-F%C3%BCr-Violoncello-Les-6-Suites-Pour-Viol/release/999363

You will see the track numbers are numbered:  1.01, 1.02, 1.03,,, 2.01,2.02, 2.03 and so on.

The discogs.pl took all those 1's and 2's at the beginning of each track number and used those as the track numbers. So you end up with 18 track 1's and 18 track 2's. 

I think the problem is that Discogs does not seem to have a fixed convention for numbering multi-disc sets.
Sometimes it is D1-01 and sometimes like above and I think sometimes different from those depending on who made the entry.

But it did ignore all the headers nicely.

Have fun.

Carl.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 02, 2014, 09:07:39 pm
Thanks for the link.  I'll take care of this tomorrow, it's an easy enough fix.  I knew there would be variations on how disc/tracks were presented.

This is also a good case where the headers might be useful; maybe they can be optionally stored in some other field, or prefix the track title, since having n different "Prélude" tracks is not so interesting.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jctcom on March 02, 2014, 09:18:44 pm
Yeah that would be nice.

The only problem I see is that they aren't used any any real consistent manner on Discogs. (and probably most other places)

btw.  I don't do much classical music.  The cataloging of them seems quite daunting.  What do people do with all those meaningless (To me lol) numbers etc...  This one was actually fairly straight forward.  Like you I just added the headers to the beginning of the tracks.  But some of those numbers and designations can be quite long making for an unreasonably long title when it comes to renaming the tracks etc...

Carl.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 03, 2014, 06:41:51 pm
Build 1.05 is now posted.

Version 1.05 Changes:
    - Fix: key 'date' is pulled from the master entry, so it is the original date of the album, not the
      release date (which is 'rdate').
    - Fix: incorrect track names and total track count when the tracklist contains header information.
    - Fix: some discogs entries use dot instead of dash as a disc/track separator.
    - New: key 'heading' mapped to Comments captures possible track headings in the discogs track list.
      A heading might contain the name of a particular work or grouping of tracks, for example:
      Suite No. 1, S. 1007 (G Major / G-dur / Sol Majeur).
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jctcom on March 06, 2014, 01:07:18 am
Hey MrC.

There is an issue with Headers / Comments which I don't think even you will be able to fix.  (How's that for a challenge eh?)

In looking at this album  (and I think many are the same)

http://www.discogs.com/Yes-Fly-From-Here/release/3075942

The Header only applies to the first 6 tracks on each of the LP and CD's there.

The problem is that there isn't actually anything to indicate where the header might stop applying if there is only one header on the album or disc.

In this particular case you could probably say it ends at the LP on the LP disc but not for the CD.

I will have to go through my discs to see what other albums might have Header's only for the first suite where they might not apply for the rest of the disc.  But I can tell you right now some good places to start will be Rush and other albums from Yes (Especially their early stuff)

Carl.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jctcom on March 06, 2014, 01:20:48 am
I am guessing people will have to do 1 of 2 thngs.

Either edit out the Comments for the tracks that the headers don't apply to afterwards.

or do like this guy did:

http://www.discogs.com/Yes-Close-To-The-Edge/release/2689712

and edit the discogs entry to add an ending header so that the system knows when that header stops applying (and your program as well).  (Now wasn't that thoughtful of him  ;D)

Carl
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 06, 2014, 12:34:36 pm
Garbage In, Garbage Out.

Headers are inaccurate if there is not some corresponding form of delineation separating items for which the headers do not apply.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: lepa on March 15, 2014, 03:33:05 am
Hi,
Just to report that --nodiscnum seems to add 0 to Disc #. It's only visible in MC if I choose art and music. MC then shows [varies] in the Disc # field. in Foobar2000 shows the zero as a disc number.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: JimH on March 15, 2014, 07:21:17 am
The view may not be showing Disc #, but you can add it by right clicking on the title bar at the top of the list.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: lepa on March 15, 2014, 09:16:35 am
The problem is that the field should be empty "" not "0"
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 15, 2014, 03:20:28 pm
This (Disc # == 0 issue) was introduced recently.   I have a dilemma.

In MC, there is no mechanism to clear a value via MCWS or MPL import, so a field that should have no value (such as Disc # for single disc albums) cannot be cleared.  If I do not set the value to 0, this leaves mis-tagged values incorrect; if I set the value to 0, an actual value of 0 is stored, but in the UI it appears as empty (since MC doesn't tightly distinguish between 0 and empty for numeric fields such as Disc # when using a field's display format).

What is really needed is a way to force the clearing of a field.   I'll post a request for this in the Beta forum.

In the meantime, I'll revert the change and not write out Disc # when --nodiscnum is set, or for single-disc albums.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: lepa on March 15, 2014, 03:25:25 pm
I can also delete tag manually. This is such time saver script that it is no problem to do that. I just noticed this because art was not getting that zero and showed [varies] when every album files was selected (and well obviously in foobar2000 I noticed the zeros)
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 15, 2014, 03:27:15 pm
Let me get a fix out for you.  It is more important that you don't have to do more manual work, esp. since most albums are single-disc.  Just note that if you have incorrect disc # values now which should be cleared, the script can't do that for you and you'll have to correct those.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 15, 2014, 03:45:54 pm
Version 1.06 is posted.  I only did brief testing, so please let me know if you see any issues.

Version 1.06 Changes:
    - Fix: Return the release year when a master is not available.
    - Change: omit outputing the Disc # field when the value is 0.  Although MC doesn't tightly
      differentiate between 0 and empty, a 0 value appears in file tags and is seen by other apps.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 16, 2014, 03:07:10 pm
I found an interesting case yesterday while creating version 1.06.  The album 2112 by Rush has a single track on side 1 of an LP, or is indexed as a single track 1 on a CD, yet is listed in Discogs as 7 sections of track 1:

1-I       Overture    4:33
1-II       The Temples Of Syrinx    2:12
1-III       Discovery    3:29
1-IV       Presentation    3:42
1-V       Oracle: The Dream    2:00
1-VI       Soliloquy    2:21
1-VII    Grand Finale    2:14

I suppose I need to merge such entries into a single track 1.  Maybe it should be like this:

    Track #:  1
    Name:  I-Overture (4:33); II-The Temples Of Syrinx (2:12); III-Discovery (3:29); IV-Presentation (3:42); V-Oracle: The Dream (2:00); VI-Soliloquy (2:21); VII-Grand Finale (2:14)

The durations could instead be starting times of each section so you know where that section begins.

Any thoughts?
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: InflatableMouse on March 16, 2014, 04:12:30 pm
I found an interesting case yesterday while creating version 1.06.  The album 2112 by Rush has a single track on side 1 of an LP, or is indexed as a single track 1 on a CD, yet is listed in Discogs as 7 sections of track 1:

1-I       Overture    4:33
1-II       The Temples Of Syrinx    2:12
1-III       Discovery    3:29
1-IV       Presentation    3:42
1-V       Oracle: The Dream    2:00
1-VI       Soliloquy    2:21
1-VII    Grand Finale    2:14

I suppose I need to merge such entries into a single track 1.  Maybe it should be like this:

    Track #:  1
    Name:  I-Overture (4:33); II-The Temples Of Syrinx (2:12); III-Discovery (3:29); IV-Presentation (3:42); V-Oracle: The Dream (2:00); VI-Soliloquy (2:21); VII-Grand Finale (2:14)

The durations could instead be starting times of each section so you know where that section begins.

Any thoughts?

How about this?

I. Overture (4:33); II. The Temples Of Syrinx (2:12); III. Discovery (3:29); IV. Presentation (3:42); V. Oracle: The Dream (2:00); VI. Soliloquy (2:21); VII. Grand Finale (2:14)

or (as its tagged in my library):

I. Overture (4:33) / II. The Temples Of Syrinx (2:12) / III. Discovery (3:29) / IV. Presentation (3:42) / V. Oracle: The Dream (2:00) / VI. Soliloquy (2:21) / VII. Grand Finale (2:14)
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: theoctavist on March 16, 2014, 04:51:01 pm
I've been looking at this, and have a question.

What is the advantage of this command vs having said metadata pulled/written by xyz tag app? a couple of them pull from discogs/musicbrain/amazon by default.


is the advantage the fact that you are keeping the files within JRiver instead of farming tasks out?

still somewhat new with JRiver utilities, so questions are plentiful.

Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 16, 2014, 05:56:32 pm
There are plenty of apps that will pull from external sources, but they will operate in their own way, pull data according to their own rules and specs, but not operate on MC fields directly.  I wanted to create some scripts that could pull data from discogs, amg, etc., and have them populate MC fields in the Library (either directly via MCWS or via MPL import) rather than update external file tags and have MC read those tags.

A couple of features were important to me and other users:

   1. user-defined MC field names
   2. credits mappings
   3. selective field assignment (via inclusion or exclusion rules)
   4. ability to populate MC metadata pre-rip (fills metadata values for the CD entry in the CD database, which carries forward to ripped tracks)

I'm not sure if this addresses what you are asking about, but maybe its a start.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: theoctavist on March 16, 2014, 09:10:44 pm
fantastic info. thank you.  So it does have many advantages over mp3 tag. Gonna give it a whirl.

thanks again.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: lepa on March 17, 2014, 03:33:47 pm
Various Artists issue. Artist field is populated with "Various" instead of track artists
http://www.discogs.com/Various-Atlantic-Rhythm-And-Blues-1947-1974/release/3868258
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 17, 2014, 04:54:19 pm
Various Artists issue. Artist field is populated with "Various" instead of track artists
http://www.discogs.com/Various-Atlantic-Rhythm-And-Blues-1947-1974/release/3868258

Thanks for reporting.

I've fixed this, but would like some feedback before releasing.

I've added a new key/field for 'albumartist' => Album Artist.  This is useful for cases where a predominant album artist should be used, such as Mark Isham for the sound track to Body Shots, where only track 1 is by another artist (Moby):

   http://www.discogs.com/Mark-Isham-Body-Shots-Music-From-The-Motion-Picture/release/2137557

In this case, both track artist and album artist will be output.

However, for some albums like the one you reference, since MC already does Album Artist (auto) detection, there's really little reason to output "Various" for an Album Artist field.  But this could be handled in two ways:

   1) I leave it to the user to add albumartist as an ignore field on a case-by-case basis, or
   2) I detect when the album artist is some form of the term "various" and suppress album artist in this case so that MC's Album Artist (auto) does the right thing.




Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 17, 2014, 07:43:21 pm
I've posted version 1.07, as the Artist/Album artist issue is worth making available.  I'll deal with handling the sub-tracks issue mentioned above later (it is non-trivial, as basically the data on discogs is bad, so I have to unify it in an external data structure).

Version 1.07 Changes:
    - Fix: module Bit::Vector was not being imported, causing the script to abort when using
      the --range option.
    - New: key 'albumartist' mapped by default to Album Artist is available.  When it is some form
      of the term "various", it is suppressed (the assumption being that track artists exist), so
      that MC's album artist (auto) logic will apply.
    - Fix: album artist was being used in all cases instead of track artists, which was problematic
      for mixed-artist or compilation albums.  See also the description for the new key 'albumartist'
      above.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: lepa on March 18, 2014, 08:23:48 am
  2) I detect when the album artist is some form of the term "various" and suppress album artist in this case so that MC's Album Artist (auto) does the right thing.
I'm in favour of this one. This is how I've done it manually. So it is multiple artists album if there is no "main" artist

...and thanks for the fix above
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 18, 2014, 11:25:31 pm
I've made a fix for the multi-part track issue I reported in Reply #66 (http://yabb.jriver.com/interact/index.php?topic=87032.msg603884#msg603884), and followed InflatableMouse's suggestion as implemented in his library (using " / " as the sub-track separator).

Before I release the update, because the fix feels very special-cased, I'm wondering if anyone knows of other similar Discogs entries.  That is, releases where some track is split into sub-tracks, but on the CD that track is indexed as a single track (i.e. ripping produces a single track).
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: lepa on March 19, 2014, 10:35:54 am
Not sure if this one fits the case. (I only have LP version) In my rip I have single Devil's Triangle track.
http://www.discogs.com/King-Crimson-In-The-Wake-Of-Poseidon/release/3872923
http://www.discogs.com/King-Crimson-In-The-Wake-Of-Poseidon/release/3236598
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: InflatableMouse on March 19, 2014, 11:18:28 am
That is, releases where some track is split into sub-tracks, but on the CD that track is indexed as a single track (i.e. ripping produces a single track).

No time indices unfortunately.

Track 4: http://www.discogs.com/Dream-Theater-A-Change-Of-Seasons/release/387427

Track 11: http://www.discogs.com/Dream-Theater-Falling-Into-Infinity/release/387421
Lame thing is its mentioned in the notes. Mine is tagged as:
Trial Of Tears [I - It's Raining / II - Deep In Heaven / III - The Wasteland]

Track 2, disc 2: http://www.discogs.com/Dream-Theater-Black-Clouds-Silver-Linings/release/2512944

Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 19, 2014, 11:43:35 am
No time indices unfortunately.

Track 4: http://www.discogs.com/Dream-Theater-A-Change-Of-Seasons/release/387427

Track 11: http://www.discogs.com/Dream-Theater-Falling-Into-Infinity/release/387421
Lame thing is its mentioned in the notes. Mine is tagged as:
Trial Of Tears [I - It's Raining / II - Deep In Heaven / III - The Wasteland]

Track 2, disc 2: http://www.discogs.com/Dream-Theater-Black-Clouds-Silver-Linings/release/2512944

Thanks, but these are not quite what I meant.  Each of these is a single row in Discogs, unlike the Rush 2112 entries which split a single indexed track into multiple rows.

Technical: each row in the Discogs page reflects an entry in a single tracklist array, and I use the listed disc #/album #/track # on discogs (stored as a position value) to figure out which of your tracks matches one from that array.  And as implemented, the position maps to a single location in the array, and its a two way relationship.  But the (incorrect) splitting of a single track into multiple rows breaks that implementation, and so I massage the array, coalescing the sub-tracks back into a single entry so that the mapping is restored.  At least one discogs author of a 2112 release used a proper sub-tracks split within a single track, but most have not.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: InflatableMouse on March 19, 2014, 11:50:32 am
Ah right. I didn't actually look at 2112 on Discogs  :-[

How about this one?

http://www.discogs.com/Romeros-The-Romeros-Play-Vivaldi/release/2610775
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 19, 2014, 11:50:51 am
Not sure if this one fits the case. (I only have LP version) In my rip I have single Devil's Triangle track.
http://www.discogs.com/King-Crimson-In-The-Wake-Of-Poseidon/release/3872923
http://www.discogs.com/King-Crimson-In-The-Wake-Of-Poseidon/release/3236598

Ok, so it appears for the CD release that the 7A-7C should be a single entry, which I'll combine.  This is yet another pattern case I'll have to implement (A-Z or a-z being used as the sub-track identifier/index).  Shall I do the same for the LP release (B3a-B3d)?  It seems reasonable given that you have only a single track.  But with LPs, since you can't "rip" them based on pre-defined indexes, I suppose you could arbitrarily split a song into any number of files.  This almost begs an option to defeat the feature, but that's more work that just adding the option.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 19, 2014, 11:57:26 am
Ah right. I didn't actually look at 2112 on Discogs  :-[

How about this one?

http://www.discogs.com/Romeros-The-Romeros-Play-Vivaldi/release/2610775

This one is OK.

Vinyl uses a <side letter><track number> pattern and I expect these.  It was patterns like <side letter>-<sub-track> or <side letter><track number><sub-track>, such as A-II that or B3c that are my only clues that rows should be coalesced.

For CD releases, tracks should be in the format of <track #> or <disc #>-<track #> and I use incorrect entries such as 1-IV as a clue.

Its all a bunch of heuristics to deal with incorrectly entered position values.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on March 19, 2014, 05:06:24 pm
I've posted version 1.08, which deals with the few cases of split-track entries we've been discussing.  Please test out some of the URLs above to see how they work for you.  If you find others, and they don't work, post the URL and I'll make corrections if possible.

Be sure to install the new required module on Windows (see below).

Version 1.08 Changes.
    - New: combine multi-part pseudo tracks into a single track entry.  The individual titles
      and durations are combined into single long track title for the track.  For more info,
      see: http://yabb.jriver.com/interact/index.php?topic=87032.msg603884#msg603884 .
      This fix requires an additional module.  On windows, in a command shell, type:

        ppm install Roman

      and hit Enter to install the module.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: theoctavist on April 06, 2014, 02:44:24 am
I've posted version 1.08, which deals with the few cases of split-track entries we've been discussing.  Please test out some of the URLs above to see how they work for you.  If you find others, and they don't work, post the URL and I'll make corrections if possible.

Be sure to install the new required module on Windows (see below).

Version 1.08 Changes.
    - New: combine multi-part pseudo tracks into a single track entry.  The individual titles
      and durations are combined into single long track title for the track.  For more info,
      see: http://yabb.jriver.com/interact/index.php?topic=87032.msg603884#msg603884 .
      This fix requires an additional module.  On windows, in a command shell, type:

        ppm install Roman

      and hit Enter to install the module.



having some issues.  this does wonderfully well on single disc albums, but it never fails that whenever multiple discs are involved, in spite of the track order in MC being lined up exactly as it is on a 2 x cd entry on discogs, the titles, numbers get all jacked up. any ideas?
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on April 06, 2014, 03:21:30 am
Do you have a Discogs URL for me to test?
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: theoctavist on April 06, 2014, 06:35:56 am
Do you have a Discogs URL for me to test?

yessir!

http://www.discogs.com/Wilco-Kicking-Television-Live-In-Chicago/release/677050
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on April 06, 2014, 11:45:25 am
It works fine here.  Perhaps you did not supply the -s sequence option, which uses your track order irrespective of Disc # and Track # in your metadata.  Without it, discogs.p will use your disc #/track # pairing to determine which entry of yours aligns with which entry in the discogs listing.  I'm suspecting you don't have the Disc # field populated.  So use -s.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: theoctavist on April 06, 2014, 05:35:45 pm
perfect! thank you.  sorry for the unnecessary Q!

Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on May 21, 2014, 04:35:53 am
@Nathan,

Edit: pre-release removed.

Here's a pre-release which includes the changes made to amg.pl earlier, as well as a new key 'playbackrange', which by default will set the Playback Range field.  I've chosen to use milliseconds so that you can adjust precisely if necessary without having to convert.  If you'd prefer HH:MM:SS, let me know.   By default, the config file disables this key (see: Options: -e playbackrange), since most users probably don't want this set.  Just comment out the option, or specifically include it with -i playbackrnage (place this option *after* the -c config file option, since later options override earlier ones).  See the Changes list for the list of recent changes.

Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jmone on May 21, 2014, 07:18:44 am
MrC - Works really well!...  but a couple of things (one album test)
- HH:MM:SS:xxx is what MC (LAV) shows when playing the chapters and it is human readable unlike milliseconds (see red box in the attached pic).
- Looks like there is a small unaccounted for gap between the song length from Discogs and the chapter start points on the BD as the further you get into the playlist you start to hear more and more of the end of the last song.  Unfortunately these gaps between the songs don't seem to be exactly the same, though on "Average" gap is 740ms (ranged from 583 to 843 on the first 7 tracks of this album). 

I don't think the Playback Range needs to be millisecond accurate as the gaps between songs are pretty wide but an optional offset value to add at the each of each song makes sense as by the end of the album it is way off (by seconds). 

Does that make sense?

I take it there is no way to read this chapter info from LAV at this point?

Thanks
Nathan
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on May 21, 2014, 12:57:00 pm
Here's an updated pre-release that includes the features you asked for.  See the Changes file.

Edit: pre-release removed.

Example usage fragment:

    ...  --filekey --rid 5397834 -i name,playbackrange,duration --trackgap .75

Add a new field "Duration (ext)" and you'll get the duration values populated (unless you've excluded it, or if you're using includes and you've include the 'duration' keyword as shown above).

By the way, if MediaInfo can read the chapter start/stop values, I already have a pscriptor scriptlet that uses it, and another one that sets playback range.  It would be easy enough to modify it to instead pull these values.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jmone on May 21, 2014, 04:16:22 pm
Almost there I think, two issues:
- I get an error saying "Value ".75" invalid for option trackgap (real number expected)" though it runs fine with a value like "1"
- The playback range calculation is out by 60minutes (see pic) after the 1 hour mark

MediaInfo does not present the chapter start values and I think the above is a great alternative till JR have the time to just create the particles by chapters (and at which point it will be exactly accurate).

Thanks
Nathan
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jmone on May 21, 2014, 04:27:49 pm
Actually, a "better" behaviour given the imprecise nature of the Track Gap would be to add it to the end time so the playback range is continuous but the start time falls in the correct place but it is not cutting off any content on either side of it.

Hard to explain, so eg from this ablum we have the following time and playback range with say a "1sec" trackgap

Track 1 = 4:33 --> 00:00:00 - 00:04:33
Track 2 = 2:25 --> 00:04:34 - 00:06:59

It would be better if the "trackgap" is actually added to the track length, so the playlist would like

Track 1 = 4:33 --> 00:00:00 - 00:04:34
Track 2 = 2:25 --> 00:04:34 - 00:06:59

Thanks
Nathan
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on May 21, 2014, 05:12:10 pm
Sorry, rookie mistakes.  I int'd when I should have mod'd.

I don't have any issues with using .75.    I'm using:

   ... --rid 5397834 -s --trackgap .75

I've updated the calculation to tack the gap to the end of playback, which caries to the start of the next item.  That's how I originally had it, but changed my mind for some reason.

Here's the update.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jmone on May 21, 2014, 05:25:20 pm
Mmmm rest looks perfect but I keep getting the error on .75

Code: [Select]
C:\Users\Nathan\Desktop\discogs>perl discogs.pl -c discogs-config.txt -i name,playbackrange --mediatype Video --filekey --trackgap .75

*** Value ".75" invalid for option trackgap (real number expected)

Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on May 21, 2014, 05:40:02 pm
Ah, I see it now on Windows.  It is caused by a difference in behavior on Win vs. Mac of a standard module I'm using.  The Win behavior is wrong.  I've re-uploaded the pre-release above with a fix.

A workaround would be to add a leading 0:

   --trackgap 0.75

but with the update above that is not necessary.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jmone on May 21, 2014, 05:51:11 pm
Yup - leading 0 works!  I've got to go but will test more later today

Quote
That's how I originally had it, but changed my mind for some reason.

It is probably because we are calling it trackgap but what we are trying to do is hit the "middle" of the track gap not actually create one.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on May 21, 2014, 05:51:55 pm
I'm open to better names.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: MrC on May 22, 2014, 11:32:09 am
I've merged the discussion about obtaining particle playback ranges to pscriptor, since it will involve a new pscriptor scriptlet to call MediaInfo (or other).

   http://yabb.jriver.com/interact/index.php?topic=85990.msg613636#msg613636 (http://yabb.jriver.com/interact/index.php?topic=85990.msg613636#msg613636)
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: lepa on September 09, 2014, 10:10:05 am
Is this script still working or did discogs change something in the API?
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: fucheeno on September 15, 2014, 11:49:51 am
Is this script still working or did discogs change something in the API?

I too would love to know.  I have been trying to get this working on both Mac and Windows without any luck.  Thanks.
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: noisyscott on August 01, 2015, 05:46:53 pm
Thanks for the script MrC.  Looks really promising, but I can't seem to get it to work. I believe I have all of the perl modules installed correctly (on a mac), but it fails with the following:
Code: [Select]
Uncaught exception from user code:
Clipboard does not appear to have copied track data.
Select the files in MC and they will be copied automatically.
Or use the --create option to create dummy files
Stopped at discogs.pl line 198.

Perl command history just for reference:
Code: [Select]
sudo perl -MCPAN -e 'install Text::CSV'
sudo perl -MCPAN -e 'install JSON::XS'
sudo perl -MCPAN -e 'install Roman'
sudo perl -MCPAN -e 'install Mac::Pasteboard'

What is strange is that I can verify that MC copies over the file paths by first selecting files in MC, then switching  to the Terminal and pasting the file paths, like so:
Code: [Select]
/Users/scott/Music/JRiver/Links/xfer/13th Floor Elevators/13th Floor Elevators - Elevator Tracks (1966) (jpstooges Vinyl Rip)/01 - I Don't Ever Want To Come Down.flac/Users/scott/Music/JRiver/Links/xfer/13th Floor Elevators/13th Floor Elevators - Elevator Tracks (1966) (jpstooges Vinyl Rip)/02 - Before You Accuse Me.flac/Users/scott/Music/JRiver/Links/xfer/13th Floor Elevators/13th Floor Elevators - Elevator Tracks (1966) (jpstooges Vinyl Rip)/03 - Make That Girl Your Own.flac/Users/scott/Music/JRiver/Links/xfer/13th Floor Elevators/13th Floor Elevators - Elevator Tracks (1966) (jpstooges Vinyl Rip)/04 - You Can't Hurt Me Anymore.flac/Users/scott/Music/JRiver/Links/xfer/13th Floor Elevators/13th Floor Elevators - Elevator Tracks (1966) (jpstooges Vinyl Rip)/05 - Splash One.flac/Users/scott/Music/JRiver/Links/xfer/13th Floor Elevators/13th Floor Elevators - Elevator Tracks (1966) (jpstooges Vinyl Rip)/06 - Tried To Hide.flac/Users/scott/Music/JRiver/Links/xfer/13th Floor Elevators/13th Floor Elevators - Elevator Tracks (1966) (jpstooges Vinyl Rip)/07 - You're Gonna Miss Me.flac/Users/scott/Music/JRiver/Links/xfer/13th Floor Elevators/13th Floor Elevators - Elevator Tracks (1966) (jpstooges Vinyl Rip)/08 - Everybody Needs Somebody.flac/Users/scott/Music/JRiver/Links/xfer/13th Floor Elevators/13th Floor Elevators - Elevator Tracks (1966) (jpstooges Vinyl Rip)/09 - Satisfaction.flac/Users/scott/Music/JRiver/Links/xfer/13th Floor Elevators/13th Floor Elevators - Elevator Tracks (1966) (jpstooges Vinyl Rip)/10 - I'm Down.flac/Users/scott/Music/JRiver/Links/xfer/13th Floor Elevators/13th Floor Elevators - Elevator Tracks (1966) (jpstooges Vinyl Rip)/11 - Roller Coaster.flac/Users/scott/Music/JRiver/Links/xfer/13th Floor Elevators/13th Floor Elevators - Elevator Tracks (1966) (jpstooges Vinyl Rip)/12 - I Feel Good.flac
Title: Re: discogs.pl: a Discogs metadata scraper and dummy file creator
Post by: jmone on September 26, 2015, 07:31:45 pm
This thread has been locked and while it contains useful additional info, the script utility has been removed and is now superseded with the single MCUtils script package (http://yabb.jriver.com/interact/index.php?topic=100449.0)