Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[BryanC]

Is there any way to perform a batch look-up? I've got over 500 episodes so there is no way I can do them one-by-one. I'm hoping that doing everything in MC instead of Sickbeard) will fix my series and season art problems.

[Matt]

Not today, but hopefully in the next month.

[maid]

Can we get IMDB ratings in the Movie look up or something similar? 

[JimH]

Is there any way to perform a batch look-up? I've got over 500 episodes so there is no way I can do them one-by-one. I'm hoping that doing everything in MC instead of Sickbeard) will fix my series and season art problems.

The obstacle is the user interface.  Would you be happy with a mode that allowed you to update, say 50 at a time, with no user input?

[BryanC]

The obstacle is the user interface.  Would you be happy with a mode that allowed you to update, say 50 at a time, with no user input?

Absolutely. Ideally, this would happen on import too (I think someone dropped a hint that this might be implemented soon). If we can trust Carnac enough to correctly fill basic fields, we should trust the lookup function to return the correct metadata for those guessed values.

[pauly139]

On the batch mode interface, could you retain the 'Use Image' and 'Use Data' checkboxes so we can only import the data? That would be great as I use separate images for the episodes.

[JustinChase]

The obstacle is the user interface.  Would you be happy with a mode that allowed you to update, say 50 at a time, with no user input?

Would doing it by Series, or even just Season make it easier?  Seasons tend to have about 15-24 episodes, so that might be a "logical" way to split them anyway.

[glynor]

Would doing it by Series, or even just Season make it easier?  Seasons tend to have about 15-24 episodes, so that might be a "logical" way to split them anyway.

I was pondering the same thing.

[pauly139]

Doing it by Series / Season seems a great idea. That way  I imagine we would just select the appropriate series from the search results as we do at the moment, and then let it populate the associated episodes.

[glynor]

Actually... After playing with it a bit, we need a way to do multiple dissimilar files (multiple series and seasons) at once.  I'll have more thoughts later...

[JustinChase]

A couple more issues with this still to be ironed out...

SickBeard put the episodes (mkv files) for a show into series/season subfolders for me.

I tried to play one of the shows, but it started playing, then just went black, and I had to stop it.  I noticed that the tag showed a duration of 0, but the file size was 2.6GB.  I 'Updated library (from tags)...' and 'Got Movie & TV info' and it updated the duration.  I figured this was good, so I selected the rest of the shows, and updated library on those also, and all these files then had the cover art for the first show I had just done this to.  This seems wrong.  It should not be using the cover art from this one show for all the rest.  I assume this is related to the issue I reported elsewhere that explained that the art for the mouse-over tool-tip is using the most recent cover art downloaded using this new tool.

Additionally, I'd like to request that the lookup default to exact match before similar matches.

I looked up "Rome S1E2" and it showed me

Rome: Power & Glory
After Rome: Holy Way & Conquest
Rome Rise and Fall of an Empire
Rome Revealed
Rome Wasn't Built in a Day
Rome

The 6th one was the one I wanted.  I think it should have been the first choice, since it was the "exact" match.

[MrHaugen]

I also think that this needs to take into account more than one TV Show or Season. If we are ever getting something like this on import, it will often be lots of different shows. Multiple movies also needs to work, and as Justin says, it needs to pick the closest match first. For movies, this should be enough with Name and Year to get an exact match. Some of us already use Year in the directory name. Any 4 digits in parentheses should be counted as a year, and used for search. Either in directory or file name.

Multi updates for series should be transparent, and you should not have to pick a TV Show or a movie from a list, as long as the data from Carnage is good. A few guide lines for directory structure might be needed if people want this to be reliable and automatic.

Some applications like Sickbeard downloads info from TV Shows before it's even on the machine. This could be a very interesting approach, as you would have the means to check what have been and what will come when. The metadata would first be associated with the imported files when there is a match.

[glynor]

I also think that this needs to take into account more than one TV Show or Season. If we are ever getting something like this on import, it will often be lots of different shows.

Exactly.

When I got back home from my recent trip this weekend, I went in to use this feature on the shows it had recorded while I was gone.  This quickly made me realize that limiting the batch-lookup to only things where [Series] and [Season] match would be a big mistake.  Most of the time, you aren't going to Import a whole season of a particular show at once (at least not if you're not pirating them, and even if you are, it'll still likely be one-at-a-time unless you wait for the season to end).  You are going to Import/Record them as they air, all jumbled up.

I also had a bunch of trouble with a few particular [Series] generally.  I was hitting two issues:

1. If your chosen series name doesn't match the series name on thetvdb.com, it can be fiddly to get the "match".  For example, try a show tagged [Series]="CSI".  That's how they come in for me tagged by Carnac.  The results window lists a CSI, but it doesn't work for anything that is actually an episode of CSI.  There is also a listing for CSI: Las Vegas, which should be right, but that also doesn't work.  To get it to work, you have to scroll way down (about 10 items in the list) an pick "C.S.I.".  The only way to figure it out was to try them all and wait for the info area to light up.

I imagine for my episodes of "Star Trek - TNG" (which is how I like them tagged), it would be similar.

2. Special characters seem to break it.  At least certain, common, ones.  Specifically, I could not ever get any results or use the feature at all on my episodes of Law and Order.  "Law and Order" doesn't match anything.  I searched thetvdb.com on the web and saw it tagged them "Law & Order", but this also doesn't work if you type it in and search manually.  I could probably search for just "Law" and then scroll through a HUGE list and find it, but what a pain.  I suspect the ampersand is the cause, probably not being escaped.  Colons also seemed to cause problems, as I had lots of trouble with the other Law & Order and CSI variants (Law & Order: SVU would only match if you searched only SVU, and CSI: New York only matched if you searched CSI and then picked the appropriate match from the list).

So, you have some non-escaped special character weirdness in there.

[glynor]

In considering it over the weekend, I really think the best way to solve this would be to add a [SeriesID] relational field to the database that is auto-filled the first time you use this dialog on a particular [Series] tag:

1. The first time I ever tag an episode of CSI, I scroll through and pick that "C.S.I." item from the list, just like it works now.
2. MC then records the Series ID from thetvdb.com in the [SeriesID] field, and "links" it to any show tagged [Series]=CSI (the way I like it, and the way it comes in from Carnac by default).
3. Any show that has a valid, non-blank [SeriesID] field can be batch-filled with no prompting.
4. If you select multiple files, and some have a blank [SeriesID] field it should prompt for those the first time it encounters them.
5. Any future auto-filling of cloud metadata should be limited to shows with a valid, non-blank [SeriesID] field.
6. If the user sets the [SeriesID] to 0 manually, then files from this series never get auto-tagged (or maybe you'd need some other special [Series]-specific switch for this).  That would be useful for shows that you might have that never have the right info in thetvdb.com and you'd rather them just be skipped and never show the picker dialog.

For number four above, we'd need a different dialog with a list where you can see the files as they get info assigned.  I'm envisioning something like the Conversion dialog, where up top it has the list of files to be looked up, and shows you progress as it goes.  I'm thinking that for shows with a [SeriesID] it would just "speed past" them, but for shows where this field is blank, it would stop on them and let you pick it manually from the "bottom part" of the dialog, which would look basically just like the current one.  This would also be handy for looking up multiple movies at once.

Alternatively, if you keep the metadata lookup dialog just like it is (the lookup window is modal and only can show one "item" at a time), then I only really see two choices: display an error if the user selects any file with a missing [SeriesID] instructing them to do those separately first, or repeatedly pop up the dialog as it encounters them.  In this case, I'd strongly recommend opting for the former choice.  The latter option could result in a mess of repeating pop-up dialog boxes if someone goes in and does Control-A and tries to do them all at once (which is basically how the AutoMeta plugin works, and it is a nightmare).

What I DO NOT think will work is requiring the user to always have their tags match the ones on thetvdb.com.  A bunch of their [Series] names are weird or badly formatted.  I also think that artificially limiting the number of files you can submit at once would be weird.  If you did the "Conversion Queue" idea I described above, then that would solve this problem.  If a user selects 400 items, it would pull up a list with 400 and at least give them a count as they go.  And, it would have a "Cancel" button to abort the rest if the user realizes they bit off more than they could chew.  But, this would allow you to tag a huge selection (all of my back episodes of The X-Files for example) if you know they are all properly tagged with [Season] and [Episode] fields.

If you limit it to an arbitrary number like 50, then you'll have to manually break up your submissions into little hunks.  What if you accidentally select 51 files?  Would it error, or just skip the extra one at the end?  If it skips it, then how would you know that once got missed?

Lastly, when we get there, AutoImport should automatically fill the metadata for new files it imports where the [Series] that gets assigned means it also has a valid, relational [SeriesID].  That way, most of my new recordings would very quickly just fill with the proper metadata.  For a new Series that I'd never recorded or tagged before, I'd just need to manually select them once, which would fill the [SeriesID] and then from then on any files that "arrive" via AutoImport would just get tagged correctly.  The only exceptions would be shows that get recorded that don't get the right [Episode] and [Season] tags.  These are more rare, but some shows don't consistently have this info in the filename (like episodes of NOVA on PBS, for example, sometimes they do, and sometimes they don't).

[glynor]

6. If the user sets the [SeriesID] to 0 manually, then files from this series never get auto-tagged (or maybe you'd need some other special [Series]-specific switch for this).  That would be useful for shows that you might have that never have the right info in thetvdb.com and you'd rather them just be skipped and never show the picker dialog.

Thinking about just this piece a little more...

[SeriesID]=-1 would probably be the way to go.  [SeriesID]=0 would be the default (since 0 is blank), so that wouldn't work.  Then, in the Metadata Lookup dialog box, you could just add a Don't Lookup Metadata for this Series checkbox down at the bottom.  When the user checks the box, it sets [SeriesID] to -1 and skips those shows in the future.  If you want to "undo" this, you'd simply select a file from this Series, and clear the [SeriesID] field in the Tag AW back to blank/0.

[MrHaugen]

Very good points glynor!

SeriesID is probably the best way to go. I'm all for the way seriesID is used to identify exactly what show it is, but I also think that info should be a first hit kind of thing in the end. But that is AFTER we have sorted out some of the misses. And the user should be aware of the risks. I hope that data users already have is not overwritten though, or that this can be a choice. The same goes for an option to have the SeriesID exact match before a Series auto update is done, or simply to take the best and first results. Those who are pretty sure of the names in thetvdb should not have to manually select it. And if it DOES give you a wrong result on 1 of 40 TV Shows, I think you'll spot it fast on the images alone, and go to standard view to update it manually.

Either way, this field have to be relational as Glynor mentions. I would also think about separating this field as a clear connection to theTVDB.com! As there might be other ID's in the future. tvdbSeriesID would be fitting I think.

[JustinChase]

Very good points glynor!

tvdbSeriesID would be fitting I think.

Agreed.  I think TVDB already has a seriesID, so maybe we just need to match that internally to avoid the potential for confusion?

I just checked, it does.  In the case of "Rome" it's...

Rome  English 73508

[glynor]

Agreed.  I think TVDB already has a seriesID, so maybe we just need to match that internally to avoid the potential for confusion?

I just checked, it does.  In the case of "Rome" it's...

Rome  English 73508

Right.  Sorry, I didn't explain this.  I already knew that was how thetvdb.com worked.  That's how the existing AutoMeta plugin matches a particular [Series] to a particular "Series" on thetvdb.com.

So, yes, the system would just copy the SeriesID from thetvdb.com over to an equivalent relational field in MC's Library.  I don't care what they call the field, I was just using [SeriesID] as an example.

The same goes for an option to have the SeriesID exact match before a Series auto update is done, or simply to take the best and first results. Those who are pretty sure of the names in thetvdb should not have to manually select it. And if it DOES give you a wrong result on 1 of 40 TV Shows, I think you'll spot it fast on the images alone, and go to standard view to update it manually.

I really think that the existing dialog (with the "batch queue" addition discussed above) is probably a very good way to assign the [SeriesID] tag initially when it is blank.  It is a pretty good system as-is, it just doesn't allow for mismatching [Series] names (if I call it CSI but thetvdb.com calls it C.S.I).  That is no big deal if you only have to deal with it once for any given [Series] though.  And, it would allow you to fix it manually if you want by simply clearing the field and re-doing the Lookup Metadata step.

The Auto-Filling of metadata from thetvdb.com (or any "cloud" source) on import has to be looked at more, but I think that implementing a good batch system would go a long way towards solving this problem.  I would definitely not want it making any guesses about the [SeriesID] in an automated fashion.

Picking this one time is not too much to ask, I think.  As long as you only need to do it once per [Series].

Once it knows the Series ID, then any file you import that has an exact match of a particular [Series] tag, would automatically get the [SeriesID] tag (because it is relational).  Then, the lookup would work just like it does now, except that you wouldn't have to manually pick the name of the show from the list, because it is already sure what that "name" is from the [SeriesID] field.

The way it works now is really pretty good about not blowing away existing data.  If you try to "tag" a file with the system that has blank or missing [Season] or [Episode] tags, or even if they are there but don't match anything on thetvdb.com (like if you have a file tagged [Series]=CSI, [Season]=8234, [Episode]=-222, for example), it just basically skips the file.  The only danger would be if a FRESH import somehow gets the right [Series], and wrong [Season] and [Episode] tags, BUT the wrong ones do happen to match valid episodes of that particular show in thetvdb.com's database.  This has never happened with my shows.  I've seen Carnac import the [Season] and [Episode] tags blank before, but not filled with incorrect info (well, there was one example, but it was an edge case and I think Matt already fixed it).

In other cases, existing metadata should always "win".  But that isn't a problem because I was only talking about filling this automatically for new imports, not for existing files.  If the new import has a JRSidecar.xml file already there, then that info should (and already does) "win" over anything Carnac spits out.

For existing files you'd have to kick it off manually, and you'd have the "batch list" up at the top showing the progress and changes as you go, just like the current system, but in a more list-friendly format.

[MrHaugen]

I would definitely not want it making any guesses about the [SeriesID] in an automated fashion.

I would And I think lots of XBMC and Mediaportal users would like the same. It's much less time consuming to look at a list of all your TV Shows once it's updated and correct the very few that gets a miss, rather than going through each and one of them and checking with a dialogue. It's even worse when you HAVE to start up that pesky workstation to do the changes.  I'm up to a lot of TV shows now, and I would not like it at all that I have to add ID to each of them. Even if it's a one time thing. It can very well be 80 times I have to do it. And each of the times I have to start my workstation to do the work, or getting my mouse and keyboard for my HTPC. I really, really want to get rid of those peripherals

But it's not THAT important for now. Now we need something that works, and works good enough for most. A good basis. After that is done, improvements can be made. This would already be a significant improvement. So, I'm all for it, either way.

[glynor]

How often do you really add new [Series] to your library?

It wouldn't be an every day thing.  Probably a handful of times per year.  Sure, to get the existing library tagged initially it would be a pain (I have hundreds of [Series] so I feel your pain), but it still wouldn't be that bad.  And, it would be easy to make a Smartlist or View that would only list Series with missing [SeriesID] fields, so you could have a go-to list that would slowly widdle down as you got it done.

From then, it would just happen when you import something totally new that you'd never imported before.  Frankly, I wouldn't bother to tag it or keep it at all if I was just "trying the show out".  It would only apply to a new show that I watched, decided was good enough to keep recording, and want to have good tag data on from then on.

EDIT:  I think maybe you're missing the point.... A relational field wouldn't require you to add the [SeriesID] to each and every FILE.  You'd only add it to ONE file from a particular show, and then it would automatically apply to future (and previous) files with that same [Series] tag.

Just like if you add an [Artist Description], you don't have to tag each file with the [Artist Description] field manually.  You simply add it as a relational field, pointing at the [Artist] field, and then add the description once to one file tagged with that [Artist] name, and it automatically applies to all other files in the database with that matching [Artist].

So, the only time you'd have to see that dialog manually was when you import a NEW show that you'd never imported or looked up before (so at the Fall Premiere time, or when you get into a new show you'd never watched before).

[glynor]

The good thing about this proposed system is that most users wouldn't have to know about or even care about the [SeriesID] field at all.  All they'd know is that the very first time they import a file from a brand new show, they right click on it, and say "Get Info about TV And Movies" (or whatever it is called, I'm not at my MC right now).  They'd have to pick out the proper Series from the search list, and then it would fill all the metadata about that one episode.

Then, next week when they download/record the next episode, it would just automatically fill in the [Name] and [Description] and [Series Description] and cover art and all of that for them, as long as Carnac could extract the [Series] name, [Season], and [Episode] tags.  It would Just Work.  If Carnac "missed" and didn't get one of those three required fields, it would be skipped.  Then, to fix it, you'd just have to fill in the missing info, right click and get metadata, and it would fill without showing the search or requiring any user interaction.  The dialog would just open and the file would be "done" and you could just verify the info and click "close".

[MrHaugen]

I know it's done once per TV Show, and not for each file. You do not have to explain. I understand everything you've explained, very well But for those of us who check out new TV Shows on a regular basis, this is not enjoyable to even do the manual lookup once. People who have 100 different shows and increasing would agree I think. As I say, this is not the most important at ALL. But it should be a goal in my opinion.

[JustinChase]

I personally would not like it to "guess", but if it finds an EXACT match, to use it, and not ask me.  I think most shows will find an exact match, so, of the 80 (or hundreds) of shows, I bet 10% or less dont find exact matches.  I could live with 10-20 "please verify" messages once, and very occasionally moving forward.

[glynor]

I know it's done once per TV Show, and not for each file. You do not have to explain.

Okay, sorry.  I thought you did at first, but then I was confused because you mentioned "It can very well be 80 times I have to do it. And each of the times I have to start my workstation to do the work, or getting my mouse and keyboard for my HTPC. I really, really want to get rid of those peripherals".  That seemed like it was something you were worried about on an ongoing basis.  Do you get "into" 80 new shows weekly?  Do you care if they're tagged properly since 90%+ of them are going to be crap and just deleted?

Anyhow... Sorry, that's why I got confused.  Sure, I'll have about 300 to do the first time to get my existing library up to snuff.  That'll be a crappy hour or so.  But, that's why we also need a dialog that can handle batches better with a Queue list.  

I personally would not like it to "guess", but if it finds an EXACT match, to use it, and not ask me.  I think most shows will find an exact match, so, of the 80 (or hundreds) of shows, I bet 10% or less dont find exact matches.  I could live with 10-20 "please verify" messages once, and very occasionally moving forward.

Perhaps it would work.  With any system that employs guessing, I think there is serious danger.  But then, I'd have never guessed they could have gotten the Carnac feature so good so fast.  Matt can work some serious magic when he puts his mind to it.

In tagging my 50 or so recordings last night, the current system DID find some matches that were WRONG.  For example, I had some files that came in tagged "NOVA" that were actually from NOVA ScienceNow.  Carnac decided that ScienceNow was part of the Episode Name, not the Series.  The problem is that thetvdb.com DOES have a separate Series ID for NOVA ScienceNow, as opposed to just vanilla NOVA, and since NOVA has been running for 39 seasons, the ScienceNow episodes all had "matches" in the dialog, they were just wrong.

I caught it because I could look at it and see that the proposed [Name] and [Description] didn't make sense for that particular episode.

The other thing that would concern me was this:

It's much less time consuming to look at a list of all your TV Shows once it's updated and correct the very few that gets a miss, rather than going through each and one of them and checking with a dialogue.

The problem is that how would you know they were wrong from the list?

MC would be filling the [Name] and [Description] fields.  Unless your list showed the original file name, and you were darn sure to check them manually before ever using the Rename, Move, and Copy tool, you wouldn't be able to SEE that they were wrong.

The [Series] name would be "right" (or it would look right, because it would be an exact match for a show you have in your Library already).
The [Season] and [Episode] tags would be right.
The [Name], Cover Art, and [Description] would be wrong, but you wouldn't know because they would match what you'd expect for a show with the above data.  So, how would you know to fix it?  You wouldn't know until you played the file and said, wait, this isn't a NOVA episode about Nazis, it is a NOVA ScienceNow episode about butterflies.  What the heck happened?

That's kind of my point.  It would be easy to spot the "failures" if they were just untagged.  It would be HARD to spot the failures if they were tagged, but tagged wrong.

[Daydream]

Problem is there's content you don't have tags for (recordings, manual downloads, etc) and content you have tags for (from somewhere else). If there's no tags there will always be instances where guessing is difficult and I for one, regardless how 'automated' (complacent? ) I get, I will always like to inspect it. If the aim is to please as many people as possible maybe some kind of an option to trust Carnac implicitly might work. And maybe explain how Carnac works - regex, hardcoded templates, pixie dust. I for one, don't know.

For the stuff you already have tags for (the 300 series; or if there's somebody with 300 series and has no metadata for them yet, there should be a reward for such a feat), it'll help in more than way if MC will be able to read more types of tags. Yeah I'm looking at them, nfo files.

Just my two cents.

[JimH]

Problem is there's content you don't have tags for (recordings, manual downloads, etc) and content you have tags for (from somewhere else). If there's no tags there will always be instances where guessing is difficult and I for one, regardless how 'automated' (complacent? ) I get, I will always like to inspect it. If the aim is to please as many people as possible maybe some kind of an option to trust Carnac implicitly might work. And maybe explain how Carnac works - regex, hardcoded templates, pixie dust. I for one, don't know.

Carnac doesn't guess if you already know.  Carnac won't step on your toes.

[rick.ca]

Exactly.

Really. Then why is this followed by six lengthy posts I haven't time to read?    

Sorry, but I don't understand what's so complicated about all this. I really don't. Yes, It's essential to get and use some unique ID for each series. But the lookup process should work in two modes: silent and interactive. In silent mode, it will retrieve data for episodes for which the season has be previously identified, and for which it's able to find a match without undo ambiguity (it should probably risk infrequent error for the sake of not foregoing close matches). The result of a silent mode update will be the system getting as much data as possible without user assistance. After a such an update, the user will have ample means (using existing features) to select files for which no data was retrieved. Running the process on these files in interactive mode, the user would be prompted to resolve ambiguities. After doing that once for a series, it generally shouldn't be necessary again (there are occasional ambiguities about episode and season numbers).

Such a system is simple and robust. It allows a user to name a series whatever they please. When that causes a search to fail, the user will be asked to make the match, and that's the end of it. If this system also facilitates some sort of rules-based scheduling (as I believe it should), it would also be possible for a user just to correct tags causing a problem (e.g., "Oops, maybe I should name that The Office (UK)") and let the system get the data the next time it runs. The ID should be regular field (Series-rational in the case of a series) so it can be changed by the user. That's what will allow the user to use whatever title they want without messing-up the system.

[glynor]

Really. Then why is this followed by six lengthy posts I haven't time to read?

Hah!  Maybe if you read those posts, you'd see that I was trying to discuss the details of that exact system that you just re-described in less detail.  

[rick.ca]

Hah!  Maybe if you read those posts, you'd see that I was trying to discuss the details of that exact system that you just re-described in less detail.

On scanning them, I thought so, but wasn't sure. I was also concerned you were throwing in some brilliant bits I was going to miss out on.

[struct]

I wish to encourage Matt etc to take their time and do this correctly, or at least do it in a future proof way that requires me to do no/little extra work as you go forward.  Whatever comes out needs to be robust, expandable and step in the "grand plan".  If that means waiting a couple of months for some infrastructure to be built, so be it.

We were given a nice taste with series images, but I have gone through a manual selection process to get the right image, and now we are going to have to do this same thing again to get the correct seriesid (unless there is some record kept within MC that I can't see).  I would rather not be given the series image option (as I can't help myself  ).

For this particular topic I would prefer some type of whitepaper on what you want to do and a likely framework to get there.  We then comment on this, rather than piecemeal release and comment with no clear idea of where you want to end up. 

Craig

PS Slight distraction, but why can't I manually edit the image file tag (but pvdimport can set it for me?).

[rick.ca]

PS Slight distraction, but why can't I manually edit the image file tag (but pvdimport can set it for me?).

All PvdImport is doing is writing the image file "beside" the video file. MC then records the location, builds a thumbnail, etc. [Image File] is an information field—not something that can be changed directly. It's the same field that provides information about album covers. Those may be "inside file" with no image in the file system. Tools for adding and changing the image associated with a file are included in the Cover Art menu. So instead of changing the value in that "field," you would use Cover Art > Add from file.

PS. I agree it would be wonderful to be able to comment on a white paper. It would save some of us a lot of time commenting on things that are never going to happen, and others angst over development going somewhere it's not. But that's not how things are done around here.

[raldo]

[...]
The result of a silent mode update will be the system getting as much data as possible without user assistance. After a such an update, the user will have ample means (using existing features) to select files for which no data was retrieved. Running the process on these files in interactive mode, the user would be prompted to resolve ambiguities.
[...]

What about a JRiver automatic metadata status tag which could be used to filter metadata retrieval results?

Values:

not ambigous: MC managed to retrieve data with very little possibility of error
ambiguous: Some user interaction (choosing series etc.) must be done before data can be retrieved.
resolved: Ambiguities has been resolved by user.
dirty: User has changed data

This could be used for automatic retrieval too. Upon new imports, run carnac, run meta eengine, tag according to above state.  

Occasionally, a user must check if there is ambigous data and resolve.

[glynor]

I wish to encourage Matt etc to take their time and do this correctly, or at least do it in a future proof way that requires me to do no/little extra work as you go forward.  Whatever comes out needs to be robust, expandable and step in the "grand plan".  If that means waiting a couple of months for some infrastructure to be built, so be it.

Agreed.

I don't think they intend to go off half-cocked anyway, they are building a bit of overall metadata collecting infrastructure.  Clearly it has some sort of "end game" in mind, though the implementation details are still being built as they go, so things can shift.  (And even though it is clear there is some semblance of a plan, they won't always necessarily tell us exactly what that is, because then we'll hold them to it or scream bloody murder.)

But, overall... That's why we're discussing it, I guess.  They read these threads, and think about them, and use some of it, and not other parts.

For this particular topic I would prefer some type of whitepaper on what you want to do and a likely framework to get there.  We then comment on this, rather than piecemeal release and comment with no clear idea of where you want to end up.

 

But that's not how things are done around here.

Right.

[glynor]

That's an interesting idea, raldo.  It could display in Details view with an icon or status light or something.  I don't know... Maybe.  You might not need it.  I think their system will be pretty good, and there won't be many false positives or missing info to deal with.

They could also use that as a mechanism to mark a series as "exempt" from metadata lookup.  But, that might be just as easily solved with just the one [SeriesID] field.

Hmmm....

No "dirty" though either way.   

[raldo]

[...]
You might not need it.  I think their system will be pretty good, and there won't be many false positives or missing info to deal with.

"Won't be many" is a few to many for those with OCD.

Edit: Btw, the coverage of the TvDb decreases the further away from the US you get.

[rick.ca]

"Won't be many" is a few to many for those with OCD.

Oh, I think I could handle it.

I suppose the utility of a "meta data status" tag would depend on how the system works. Missing data is usually a fairly obvious indicator, but it certainly wouldn't hurt to make it more obvious. A date stamp indicating successful data retrieval may be all that's needed. If it's empty after an import, then it's evident the user needs to run an update in interactive mode. I don't think "resolved" is a meaningful state—just write the date stamp when successful. Finally, the field could be used to disable (or "exempt") data retrieval for single files—simply by entering something other than a date (it would have to be a special field type). The same so method could be used to disable data retrieval for a series—using the ID field (e.g., entering "disabled" instead of a series name or ID).

I've left out "dirty" because it implies a system in which the user will record or make changes to the same fields used for writing data from an external source. Aside from fields that can be configured to be filled only if empty, I think that's a dangerous practice that shouldn't be encouraged. This is part of the reason why I hope for some sort of field mapping—so the user has complete control over what fields are populated by download, and which are strictly for the user.

Let's not abandon hope this will ultimately be a system that can handle multiple sources. Especially for those geographically and culturally removed from the US. I suppose there could be a separate status field for each source. Or a separate field to indicate the last source used, with a single status field indicating the result.

[glynor]

Edit: Btw, the coverage of the TvDb decreases the further away from the US you get.

Which is part of why I don't think it should guess about the [SeriesID] at all.  Leave those untouched (except for by Carnac) until the user picks the SeriesID manually via the cool, slick lookup tool (hey, they already built one of those).

My guess about the style of what they are trying to build is this:  Something elegant that works behind the scenes to get Metadata for you where it can, without bothering you very much, but which also doesn't get in your way.

That's how they've been approaching Carnac and Red October and a bunch of other recent stuff.  I think it's been working.  That design "goal" seems to lend itself to:  Guess with limited user input, but if it isn't sure, leave it alone (blank).  That's how Carnac works.  Don't override the user, and don't guess if you aren't quite darn sure.

That's basically my whole point for suggesting the [SeriesID] idea.  For me, without some mechanism to tell the system "I want this [Series] to always match this "Series" on thetvdb.com" I think I'd end up with a bunch of files it would never touch automatically.  And then I'd really have a lot to "fix" manually.

[glynor]

Hmmm...  Good idea, Rick.  A [Date Metadata Retrieved] field, like the current [Dare Imported] one.

[raldo]

Hmmm...  Good idea, Rick.  A [Date Metadata Retrieved] field, like the current [Dare Imported] one.

Fair enough. [Date Metadata Retrieved] is a good suggestion.

But you'd have (at least I'd have to) to set up smartlists (or something) to filter out those recent files which received no updates.

Btw: You can call it whatever you'd like. The intended meaning behind "Dirty" was that the user has fiddled with retrieved metadata. It's a method/function that would require some software implementation, I guess.

[struct]

yep, I get that whitepapers is not the way things are done.  It is nice to have the surprise of unexpected additions   Nevertheless I don't think our input is particularly constructive without a bit of guidance (no promises need be made/implied or otherwise inferred).

This metadata system needs to be done properly if it is done otherwise we are left with pvd or other more "complete/flexible" system anyway; a halfway house is the worst.  rick has covered a majority of what needs to be done. one only needs to look at pvd, yannfoe or other systems to see what options are required (e.g. on meta-data update choose what fields come from which source, which fields should be forced to update or which should only be updated if empty, etc).  these other systems are not all automagical and although I am the first to let the software get as much data as possible on its own where there is no ambiguity (or an option to always force a manual check for glynor's comments above) there must be a clear path for correction and a clear understanding on how force a full/partial update when an error is made. 

for someone like me who has spent far too many hours getting a lot of data into MC but wants to supplement with the new additions from MC (e.g. rotten tomatoes rating) needs some flexibility in what gets updated.  I also need a clear way to use the "perfect" data I have to force MC to get the correct new data. I don't want any interface or prompting. I have a thousand TV shows with exactly the correct imdb/name or whatever field, I need a way to tell MC what to use and what to update.

I see a great big options tab in the options dialog: 
- for media subtype b get x field/image from y source, always/never/only when empty overwrite existing value.
- repeat for each field available from all sources (grey out where not available from a source)
- definition and configuration of what is the key field to govern a lookup (eg is it the series/imdb id or is it the name [see context menu comment below])
- be able define not just an online lookup but a relative/absolute lookup in current directories for those with existing data and those coming from xbmc or other places (e.g. if I have screenshots in ../screenshots use these in preference to online sources)
- get movie info context menu gets expanded for a couple of key options (e.g. silent mode, always confirm, update based on field x+y failing that then z)
- have an option for a "processing" directory where carnac does the interpretation, finds a match (to existing entries?) and moves/renames to a defined location [yet another dialog?] (eg similar to sickbeard)

With an options menu, when fanart becomes usable in v18, i get to change the options, ask for a full update of only fanart and hey presto I have fanart but have not screwed anything else.

yes it may be all to confusing at step one with an options matrix but jriver set the defaults for the new user it is as automagical as jriver can make it, but for others there is a tweak available.  I hope it is not not left closed.

i truly hope jriver go to town on this and do a great job as they are more than capable of doing.

craig

[MrHaugen]

Good points struct. I can't do anything but to agree. This should be the long term goals.

Even if JRiver are skeptical of getting automatic info from some sources, it's important that the framework is there. The users could make the rules them self, so you are not responsible in any way, no matter what sources things come from. A good default setup from thetvdb and moviedb would do well for most I think, but the key to a truly great system would be user customizable sources, priorities, update intervals, and write/overwrite options.

[ThoBar]

Tacking on to this... why is RottenTomatoes the only one to return Genre info?

This is really frustrating, as all the sites have Genre info, and I find Genre is really important.

Also, it would be nice for the new Bulk-lookup system to be able to specify preferences for the filling of particular fields which provider.

[rick.ca]

This is really frustrating, as all the sites have Genre info, and I find Genre is really important.

The issue is somewhat bigger than this comment suggests. Not all sources have a Genre, and there's not much consistency between those that do. Most, however, use the convention of not attempting to assign just one genre to a movie. In providing a string field for recording just one genre, JRiver is pretty much alone. So even in the case of Rotten Tomatoes, only its first mentioned genre is being included.

But this is just an example of a bigger issue. While perhaps a majority of users don't care, many of us want meaningful, consistent meta data. We're not likely to agree on where it should come from, exactly what types of data we want, and how it should be recorded. We need a flexible system that will allow us to choose sources, the data elements to be saved, which fields they're to be saved to and whether or not existing data should be overwritten. Ideally, it will also allow the use of expressions to change the data saved.

In the case of genre, such a system will allow you to decide which source can best provide consistent, meaningful data (you won't want to mix sources). It will allow you to save it in a list-type [Genres] field. And even if you want to be unconventional and assign just one genre, it will provide the means for doing so.

[darichman]

We're not likely to agree on where it should come from, exactly what types of data we want, and how it should be recorded. We need a flexible system that will allow us to choose sources, the data elements to be saved, which fields they're to be saved to and whether or not existing data should be overwritten. Ideally, it will also allow the use of expressions to change the data saved.

In the case of genre, such a system will allow you to decide which source can best provide consistent, meaningful data (you won't want to mix sources). It will allow you to save it in a list-type [Genres] field. And even if you want to be unconventional and assign just one genre, it will provide the means for doing so.

I think Rick has summed this up very nicely.

The way I would see things working? An options menu or dialogue box on lookup which allows us to:
  - specify which info we want to pull and from which source on a field by field basis
  - for each piece of info/metadata
        - which source we want to take it from
        - assign which MC fields these go into (field-mapping)
        - specify whether we want to over-write existing data or only empty fields
        - not essential, but a way to set a delimiter (PVD does this) to facilitate importing into list fields
  - recognise the above might be different for different media types (TV, movies - expressions or profiles for metadata lookup would be ideal, more on this below)
  - have some way of recording when the metadata was updated (there have been several suggestions)
  - to streamline future lookups, storing IDs from the various sources, including TheMovieDb, TVDb, RottenTomatoes, IMDb
     (these could be manually assigned for ambiguous or non-unique titles - will help with disambiguation)

This could be integrated into right-click menus, or in options - even perhaps as part of the import functionality.
It would be nice to define:
  - whether we want to run the lookup
        - manually (eg right click menu on certain files....)
        - automatically (at defined time intervals if we're happy to trust MC to maintain the data automagically in the background)
  - additionally, define which files to perform metadata lookup on
        - you might decide to have this limited by media subtype or a particular directory
        - taken to its most flexible realisation, could be run on results of a particular search expression (ala File Info templates in Theatre view) with profiles for user defined searches (<- ideal option )

[rick.ca]

taken to its most flexible realisation, could be run on results of a particular search expression (ala File Info templates in Theatre view) with profiles for user defined searches (<- ideal option)

You're too easy to please. Given this is the way everything else works in the program, I'd call it the "expected option." 

I don't disagree with anything you've said, but I think how all this fits together is important. I imagine a fully automatic system, where most of the data, most of the time simply appears without any user intervention. The meta data retrieval part of it would run on auto-import and on a schedule—according to how available sources have been configured—in a "silent mode." There would also be an "interactive mode" in which the user would select files, and from the context menu run Get Movie & TV Info from...

• All sources configured for the selected files (i.e., according to those user-defined searches);
• Source 1;
• Source 2;
• Etc.

You would use that to add meta data that could not be added in silent mode. The system would prompt you to resolve ambiguities and problems as it processed each of the selected files.

In a sense, the use of these two modes have two applications. First, when adding a large number of new files, you definitely want this thing to run unattended (perhaps overnight), without stopping for any reason. At the end of that pass, 10 to 20% of the files will not be processed—but only because everyone of them really requires user intervention. Those can be done in a batch interactively—whenever the user has the time and attention for it. Overall, the update process is then as efficient as it could possibly be.

The second application is the regular daily routine of adding a handful of files (e.g., last evening's TV shows). In a properly configured system, those are very likely to be processed automatically. Even if they're not, you may choose to ignore the incomplete data for a time (e.g., the description of an episode you're not ready to watch anyway). Whenever you have the time and inclination, you would select files in need of updating (which are probably identified for you) and run the process in interactive mode. At the same time, the automatic mode might be used to update other files in respect of changing data (e.g., votes, ratings, box office, etc.).

[darichman]

Yes - to everything Rick just said. J River, please take your time and get this right

[MrHaugen]

*gentle nudge*

[JustinChase]

A couple more issues with this still to be ironed out...

SickBeard put the episodes (mkv files) for a show into series/season subfolders for me.

I tried to play one of the shows, but it started playing, then just went black, and I had to stop it.  I noticed that the tag showed a duration of 0, but the file size was 2.6GB.  I 'Updated library (from tags)...' and 'Got Movie & TV info' and it updated the duration.  I figured this was good, so I selected the rest of the shows, and updated library on those also, and all these files then had the cover art for the first show I had just done this to.  This seems wrong.  It should not be using the cover art from this one show for all the rest.  I assume this is related to the issue I reported elsewhere that explained that the art for the mouse-over tool-tip is using the most recent cover art downloaded using this new tool.

Additionally, I'd like to request that the lookup default to exact match before similar matches.

I looked up "Rome S1E2" and it showed me

Rome: Power & Glory
After Rome: Holy Way & Conquest
Rome Rise and Fall of an Empire
Rome Revealed
Rome Wasn't Built in a Day
Rome

The 6th one was the one I wanted.  I think it should have been the first choice, since it was the "exact" match.

A friendly reminder the above issues are still not working "properly"