Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Tolga]

Is there a way to upload a bulk of (key, field, value) tupples to MC via MCWS efficiently?

I need this for various utilities I am writing like the one that embeddes content MD5 hash into the DB to ensure data integrity. Sending updates to MC File/SetInfo one at a time works but it is very inefficient. Is there a better method to upload bulk of data? I believe Library/Merge may have similar functionality but the format of uploaded file is not documented. An option that allows MPL files as input should do the trick.

[Lasse_Lus]

have you checked/tried prod's sqlmytunes ? http://yabb.jriver.com/interact/index.php?topic=56405.0

[MrC]

I don't know if this is an option or helps, but if you create a playlist with your tuple, replacing the file key with the full path to the file, you can import the entire thing in one shot.

[Tolga]

I don't know if this is an option or helps, but if you create a playlist with your tuple, replacing the file key with the full path to the file, you can import the entire thing in one shot.

I wonder if this can be used to overwrite existing values in the db, or filetags. I doubt it but worth trying. It is also a little hacky to to be part of a plugin code.

[Tolga]

have you checked/tried prod's sqlmytunes ? http://yabb.jriver.com/interact/index.php?topic=56405.0

looks interesting indeed.

[MrC]

I wonder if this can be used to overwrite existing values in the db, or filetags. I doubt it but worth trying. It is also a little hacky to to be part of a plugin code.

Overwrite: Yes.  Clear: No.

[Matt]

If you have a proposal for MCWS function additions, just spec it out here.

If it's reasonable, we'll add it.

[MrC]

Ok, old topic, but I have a need (desire) now.

I'd like to be able to set a series of file properties.  Currently SetInfo allows only one Field/Value pair:

    http://localhost:52199/MCWS/v1/File/SetInfo?FileType=Key&File=21169586&Field=Album&Value=TestAlbum

How about support multiple Field/Value pairs?

    http://localhost: ... &Field=Album&Value=TestAlbum&Field=Rating&Value=0&Field=Genre&Value=Junk

This would save me dozens of calls per track.

[GrantDG]

Could this be done with database Sync?

MCWS/v1/Library/Get
MCWS/v1/Library/Merge

[Matt]

How about support multiple Field/Value pairs?

    http://localhost: ... &Field=Album&Value=TestAlbum&Field=Rating&Value=0&Field=Genre&Value=Junk

I like the idea of setting multiple at once, but not the idea of using the same parameter name multiple times.  Doing that would be challenging for us with the current architecture, and I'm not sure if we should count on proxies, etc. not reordering parameters anyway.

How about accepting an MPL block (optionally as a posted parameter)? 

Or using Field1, Value1, Field2, Value2, etc.?

Another approach would be a special delimited string, but at that point, we might as well use MPL like the rest of the MCWS functions.

[MrC]

I had thought about that, and figured you'd provide good feedback to better alternatives.  Either approach is fine with me; MPL seems a little excessive, but that's OK.

Since I already have MPL being output, it would be easy to convert it to a block.  I presume I'll need to turn newlines, line feeds, and question marks into HTML entities.

[MrC]

A little bump on this.  Individual field updates are painfully slow.

For example, I'm updating the following 18 fields (these are keys to field names) for 18 tracks, and it is taking 35 seconds to complete:

   album, apick, artist, composer, date, genre, mediatype, moods, name, rating,
   ratingmc, review, styles, themes, tpick, tracknum, turl, url

That's 324 calls (+ 1 call per/track to get the filekey), or about .11 seconds / call.

It is far faster to output to MPL and manually import it via File > Import Playlist.

Ideally, it would be great to be able to send a multi-Item/multi-Field MPL, keyed by either Filename or Filekey to do a single bulk update.  Fields that don't exist, like when using File > Import Playlist, should be silently ignored (currently they throw a 500 server error).

[Matt]

I haven't forgotten about this, I'm just not sure the best way to proceed.

Is it important to be able to do more than one file in a call, or just more than one field?

Would a generic pipe-lining system be useful (ie. MCWS/v1/Pipeline) where you post a bunch of calls all at once?  I'm not quite sure how the return values would work in that case.

[MrC]

Thanks.

One multi-field file per call would be fine, assuming about 1/10 second per call, an album would take 1 to 2 seconds, and that seems reasonable.  In this case, probably the preamble is unnecessary, and only the Item contents are important.  But its easy either way.  For my needs, I'm updating one file at a time and always have the Filename.

I don't know if you saw the last sentence in my post above, which I added later.   If multiple files are possible or practical (each Item keyed by Filename or Filekey), being able to send an encoded multi-Item MPL would be a bonus.

Regarding errors, I'd like to see 500 server errors split into hard errors (e.g. protocol error, invalid syntax) and softer errors with response contents indicating the error (e.g. "Field "Foo" does not exist", for example), so that the soft errors can be detected and ignored, but hard errors could trigger an abort.

I'd have to look into how the URL package code I'm using deals with pipelining.

Edit: I thought about this some more.  I can think of at least one tool I would write that would take advantage of fast writing of  many files in a single shot.  So, I'm leaning towards thinking the number of items/fields should be arbitrary, just like a structured MPL is currently.

Edit 2:  Further investigation reveals the modules I'm using don't support pipelining, so I'm hoping you don't go this route. :-)

[Matt]

Next build:
SDK: MCWS/v1/File/SetInfo supports setting multiple fields / values per call by CSV encoding a list for Field and Value parameters and also setting a parameter List=CSV.

We thought CSV was easy to deal with and reasonably well understood.  In many cases, a simple comma delimited list will work.  It only gets more complicated when there are escape characters.  RFC 4180 specifies how to handle these cases.

Here's an example call that does some escaping:
http://localhost:52199/MCWS/v1/File/SetInfo?File=1166&FileType=Key&Field=Artist,Album,Name&Value=MrC,%22Was,%22,Here&List=CSV

[Matt]

Regarding errors, I'd like to see 500 server errors split into hard errors (e.g. protocol error, invalid syntax) and softer errors with response contents indicating the error (e.g. "Field "Foo" does not exist", for example), so that the soft errors can be detected and ignored, but hard errors could trigger an abort.

At a basic level, a function either works (200) or doesn't work (500).

When it doesn't work, it often provides extra clues.  For example, if you use bad CSV and call the updated MCWS/v1/File/SetInfo, you get a 500 and this XML:
<Response Status="Failure" Information="Invalid CSV data."/>

Does that help?

[MrC]

Yes, that's great.  I presume it will also yield an "Invalid Field Name" under such circumstances.

[Matt]

Yes, that's great.  I presume it will also yield an "Invalid Field Name" under such circumstances.

Yes, if you do a single field / value.  When you use CSV, it's a little harder to know what to do for error handling.  Currently if the CSV looks good, it'll return success even if none of the field names are found.

If there are errors where it's not giving helpful information, just report them and we'll improve it.

[MrC]

I think something is not working correctly when values contain , or ? characters.  I'm URL encoding values, and then CSV-ifying them.  The value for Name below contains both comma and question mark.

If I remove Name from the write, it works fine.  I wonder if you are doing the reverse of what I'm doing on receipt, i.e. URL decode then reading the CSV?

Code: [Select]

mcws_send_items: Filename: M%3A%5CTesting%5CAbbey%20Lincoln%5CYou%20Gotta%20Pay%20the%20Band%5C04%20Brother%20Can%20You%20Spare%20a%20Dime.mp3
mcws_send_items: Added field(name): Name -> Brother, Can You Spare a Dime?
mcws_send_items: Added field(album): Album -> You Gotta Pay the Band
mcws_send_items: Added field(mediatype): Media Type -> Audio
http://localhost:52199/MCWS/v1/File/SetInfo?List=CSV&Value=Brother%2C%20Can%20You%20Spare%20a%20Dime%3F,You%20Gotta%20Pay%20the%20Band,Audio&Field=Name,Album,Media%20Type&FileType=Filename&File=M%3A%5CTesting%5CAbbey%20Lincoln%5CYou%20Gotta%20Pay%20the%20Band%5C04%20Brother%20Can%20You%20Spare%20a%20Dime.mp3
500 Internal server error
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<Response Status="Failure" Information="Invalid CSV data."/>


[Matt]

I think something is not working correctly when values contain , or ? characters.  I'm URL encoding values, and then CSV-ifying them.  The value for Name below contains both comma and question mark.

I haven't tested this yet, but just thinking about it, I'd expect you to work in the other order: first make CSV, then URL encode it for use as parameters.

In the reverse, we get URL parameters and URL decode them first.  Then we're left with regular strings, or in this case, CSV.

Does that make sense?

[MrC]

Yes, it does make sense.  The first test looks good.  On to more testing...

The first album took a whole 1.2 seconds, including the external page scraping lookups.  Very nice.

Thank you.

[MrC]

Testings looks good.  It is very fast and nifty.

I (and others) thank you! :-)

Time to release some code...

[InflatableMouse]

I (and others) thank you! :-)

Indeed. And thanks for testing MrC.