Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Hendrik]

I vote for rick.ca's plan!

While at it, I really only wish for one thing right away: full regex support to gather all kinds of informations from filenames and use them for tagging. That would finally allow me to tag and use my neatly organized movie and series library.

Unless that already works and noone told me?

[Matt]

I vote for rick.ca's plan!

While at it, I really only wish for one thing right away: full regex support to gather all kinds of informations from filenames and use them for tagging. That would finally allow me to tag and use my neatly organized movie and series library.

Unless that already works and noone told me?

You can use Library Tools > Fill Properties from Filename

If that isn't enough, describe what you'd like and we'll see if we can make it happen.

Thanks.

[Hendrik]

That is by far not powerful enough. It only really works if your files follow one exact pattern.

Some things I would need:
- replace certain chars with spaces (dots, underscores)
- multiple pattern for the same attribute (s01e01 vs 1x01 vs 101)
- take some attributes from the folder name others from the file

I can probably think of some more..

In the end, nothing will suffice except a full regular expression parser.

[JimH]

I vote for rick.ca's plan!

While at it, I really only wish for one thing right away: full regex support to gather all kinds of informations from filenames and use them for tagging. That would finally allow me to tag and use my neatly organized movie and series library.

Unless that already works and noone told me?

I'm not sure what regex is, but you can tag your files from your filenames with MC.   Select a few files (be careful), then right click on them, choose Library Tools/Fill Properties from Filename.

[bspachman]

That is by far not powerful enough. It only really works if your files follow one exact pattern.

Some things I would need:
- replace certain chars with spaces (dots, underscores)
- multiple pattern for the same attribute (s01e01 vs 1x01 vs 101)
- take some attributes from the folder name others from the file

I can probably think of some more..

In the end, nothing will suffice except a full regular expression parser.

You may be able to accomplish a bunch of this by using the "Move, Copy, Rename..." tool with expression language. It would certainly be more tedious to write IF statements for the various filename styles you have than to wrap it all up in a single REgular EXpression, but that's how I've rolled for a few years now.

brad

[Jaguu]

I'm not sure what regex is

regex = regular expressions! I thought you were a Unix man, Jim!

Yes, sometimes I miss this function in Search/Replace, as it would allow to refine search/replaces much more than it is now. On the other hand with the tools available in MC16, you can do almost everything, although sometimes you need several steps. One handy function in Search/Replace is that you can apply it on the filenames doing the file renaming as well.

By the way, as far as I know there are various open source regex libraries that can be probably simply included in the program. See http://www.regexlib.com

[Alex B]

You may be able to accomplish a bunch of this by using the "Move, Copy, Rename..." tool with expression language. It would certainly be more tedious to write IF statements for the various filename styles you have than to wrap it all up in a single REgular EXpression, but that's how I've rolled for a few years now.

nevcairiel,

I.e. try first fix the filenames with the "Rename, Move, & Copy Files" tool. Use the expression language when applicable ( instructions by marko: http://wiki.jriver.com/index.php/Media_Center_expression_language ) and then use the Fill Properties From Filenames tool.

Here is an old reply in which I explain how I use Find & Replace and some other simple features when I tag video & subtitle files in a list view:
http://yabb.jriver.com/interact/index.php?topic=51540.msg357304#msg357304


Alternatively you could try Flexible Renamer: http://hp.vector.co.jp/authors/VA014830/english/FlexRena/  (it has wild card support)

[MrC]

There are other threads requesting regular expression support, such as this one.

There are many classes of textual replacement problems that can't be generally solved with the current expression language in MC - instead, they require specific case-by-case modifications to an expression via IF statements.  The MC expression becomes doomed on a case-by-case basis.

Directory Opus finally upgraded and updated their regular expression support from the simplistic basic regular expression-like to something more on par with the most powerful of the sets of RE implementations (eg. perl and other languages).  They are using the Microsoft TR1 implementation, which is sufficiently powerful for MC.

[Hendrik]

I do not want to rename my files, but thanks for the tips.

I have a structure on the filesystem like this:

TV Series/#show#/Season #/#filename#.mkv

What I want is all episodes to be categorized by show and season, and then shown in theater view in episode order, preferably with a cleaned name (1. #ep title#, or Episode 1) instead of the file name.

To do this, the show name should be taken from the #show# folder, and the season and episode can be extracted from the filename. XBMC does this beautifully, fwiw.

[rick.ca]

To do this, the show name should be taken from the #show# folder, and the season and episode can be extracted from the filename.

Regex would obviously be more powerful than (or add more power to) the existing expression language. But what's really needed here is some sort of auto-tagging system. That's been discussed at length already, without (it seems to me) any consensus on the relative importance of such a capability or exactly how it should work. However it's done, the result required is for media appearing in the file system (by any means—manually moved there, downloaded, ripped, recorded) to be automatically tagged on import based on file path name. The other essential component of a complete system is the automatic search for and addition of available meta data. That, IMO (again, no consensus), should include the ability to create and support records for media that does not exist. That would allow things like a new movie wish lists and records of upcoming episodes to be included in the database.

These two things together would make a great development project for version 17. If it got the same kind of focus as RO did for 16, it would surely create the same sort of buzz. And not just with existing users, but with huge numbers of potential customers just waiting for MC to become a complete solution for video. Surely it's time to end their suffering!

[Lasse_Lus]

...should include the ability to create and support records for media that does not exist. That would allow things like a new movie wish lists and records of..

already there rick, notes...

[rick.ca]

already there rick, notes...

how can a note can be tagged and handled in the same manner as a file? How does a meta data plugin add data to a note? How do I include notes in a Theatre View along with other file-based records?

[Edit]
I'm surprised to see my original expression of genuine surprise has been removed. Now I feel compelled to clarify these questions are now out of context and not meant to be rhetorical. I don't use the Notes feature and, while I doubted it, I really thought there might be something important I was missing. I suppose the fact my post was edited by the moderator suggests otherwise.

[gvanbrunt]

I have a structure on the filesystem like this:

TV Series/#show#/Season #/#filename#.mkv

You can do this with fill filename from properties like this:

Directory text box: [Show]\[Season]\
Filename text box: [Episode]

That is if the filename is episode. You can use expresions if only part of the filename is the episode #etc. Also since Show and Season do not exist, you would have to create custom fields for them...

If you really do need Regular Expressions, there are many utilities (as you likely know) that can rename files etc using regex. You can clean them up a bit, then use the Fill Filename from properties.

[Matt]

I've been thinking about this, but I'm not sure the best way to proceed.

I think regex is neat and would be a good thing to support.  But it's not too useful for non-programmers (i.e. most users).

Do you think we could make an auto-magic filename parser that could recognize common templates, and automatically get information from them?  Something that was hands off for most users would be slick if it could be done.

[Jaguu]

But it's not too useful for non-programmers (i.e. most users).

Well Matt,
 
some simple regex rules are part of Microsoft Word (see Word Help: Search/Replace). There are probably a few hundred miliion copies of Word around the world. And this rules are usually taught in basic Word courses. Simple office secretaries know about those rules. Any Editor I know (like notepad++) do have them. In most cases you can activate / deactivate regex rules.

[MrC]

I've been thinking about this, but I'm not sure the best way to proceed.

I think regex is neat and would be a good thing to support.  But it's not too useful for non-programmers (i.e. most users).

Do you think we could make an auto-magic filename parser that could recognize common templates, and automatically get information from them?  Something that was hands off for most users would be slick if it could be done.

Suggestion - create the templates using regular expressions, and then allow the templates to be edited like calculated expressions are today.  This gives you the power of what regular expressions were designed to do (recognize and capture textual patterns), and makes standard templates available for users.

I think it would be a mistake to attempt to use MC's current excel-style functions, as they are not general pattern-recognizers.

Instead, rely on the community to provide MC users with regular expressions that amend and modify pre-defined templates.  This solves the problem once and for all for JRiver.  There is no reason why you should be burdened by the necessity of adding new one-off, specific functions such as RemoveCharacters().

What you want is (and I'm making up the syntax for the LHS):

{date (year)} = \d{4}

or a little more specifically and strictly:

{date (year)} = [12]\d{3}

Common file path parsing is just as easy, where common regex-based templates can  be used to both cull parent directories names to be used as title, artist, etc. as *well as* embedded dates, or other fluff that should be included as part of, or removed from, the metadata.

[rick.ca]

Suggestion - create the templates using regular expressions, and then allow the templates to be edited like calculated expressions are today.  This gives you the power of what regular expressions were designed to do (recognize and capture textual patterns), and makes standard templates available for users.

This is the way to go. Users are not going to appreciate the auto-magic system that doesn't work in their situation. True, many will not be happy having to ask for help, but at least a flexible regex-based system will pose no barriers to others trying to offer it. And it can still be presented with all the "common templates" you might use anyway. So you have the best of both approaches, and it would still be "hands-off" for most users.

In any kind of pattern recognition system, a "template" is necessarily going to be a list of regex, functions or rules applied in order until a match is found. Since this much at least has to be exposed to the user, it shouldn't be difficult to provide a library of expressions according to what they do—indicated by way of an illustration of the form of file pathname each matches. The user would pick one from a list, then, if necessary, modify the actual expression.

My experience with supporting the same function in Personal Video Database suggests most users will not be interested in how such a system works, they'll just expect it to do so. Many new users will have collections of filenames for which patterns will be very difficult or impossible to match. They're not likely to respect the fundamental choice that remains regardless of what the system is capable of—configure the system so it recognizes all existing filenames, or change the filenames so they're easier to recognize. Limiting the capability of the system will only result in users resenting the necessity to change their filenames or demanding the "magic" be modified to suit their circumstances.

A flexible and powerful regex-based system will not only be more useful to all users, it will be much easier to maintain and support.

[jmone]

I had a look at my Videos and how they are named and stored on the drives.  A bit like with Audio files, I have a mix of using Directly Structures / Naming convention that provide all the info needed to populate the needed fields of (for example in TV Series):
- Media Sub Type
- Series
- Season
- Episode
- Name

The issue is the complexity and variation even I have (for example for TV Shows I tend to have Series Name as part of the structure and then also the filename).  Then multiply this over the number of users.

One approach that works well in the middleware space for data transformation is allowing the user to graphically select or "paint" the bits they want to map to a specific field so they can create their own import template(s).  You could do this either post import or as a part of the import routine, eg see my crude pic

[Matt]

jmone's screenshot sort of shoots down the idea of using a single expression for a given folder.  Within that folder there are all sorts of naming styles.  And my guess is that he's more tidy than a typical user.

But looking at his list, it doesn't seem like it'd be that hard to make an algorithm to pick out the good bits of information.  Some things are obvious like "S3E21".  Adding a simple list of programs would make the algorithm even better.

[Hendrik]

Regular expressions can contain alternate expressions for the same match, you can do all that with it, and more.

[MrC]

Without any discernible pattern, it be probably impossible to *know* how to parse a string of almost meaningless characters into meaningful tags, at least over an entire collection.  There is a point of diminishing returns between provide simple patterns run several times vs. a monolithic exponential backtracking pattern.  Given the exact data jmone has presented, I could write an RE that solves the specific cases.  So there is some GIGO here.

But that's not the point I think.  Rather, it is to provide the tools necessary to recognize patterns that do exist.  For every S3E21, there might be dozens of other ways to encode season/episode (S3;E21, S3-E21, S3 E21, Season3 Ep. 21).

I helped somebody in the Directory Opus forums, which used essentially a very large case switch  (>600 code lines) to match exactly type of thing you're talking about here.  One 40 char RE solved the same problem.  His script tried to pull apart and canonicalize book author, title, series, and notes from a file name; it was 1000's of lines.  Much of it was replaceable by a couple of REs.

Do you really want to be in the game of having to manage this stuff perpetually?

[jmone]

jmone's screenshot sort of shoots down the idea of using a single expression for a given folder.  Within that folder there are all sorts of naming styles.  And my guess is that he's more tidy than a typical user.

Maybe not.  Prior to coming to MC, all I had for Media Portal / Nero Media Home was the directly structure approach so you tended to structure this stuff pretty well, along the lines of:

Media
  --> Audio
  --> Video
       --> Movies (then all under here by Movie Name for me)
       --> TV Shows
            --> Series Name (then all under here but with a File Name that included Series Name, Series #, Eps #, Eps Name etc)
     etc etc


Once using MC however, I'm no where near as worried about how this stuff is represented on the disc as I (of course) use the meta data fields....

But looking at his list, it doesn't seem like it'd be that hard to make an algorithm to pick out the good bits of information.  Some things are obvious like "S3E21".  Adding a simple list of programs would make the algorithm even better.

I've gone through my entire video collection and for what it is worth the following is a random example of each of the various structures I use.

[Matt]

The thing I keep hearing is that "XBMC just does this automatically."  Maybe I'm not understanding, because setting up regular expressions is the opposite of something that just automatically works.

Can someone explain exactly what other programs do that they like?  Thanks.

[MrC]

Well, that's curious.  Because the XMBC documentation says things likes:

TV show file naming conventions, for TV shows to work properly in the Video Library, the file naming conventions used have to match the regular expressions used in XBMC

...

HOW-TO add and scan media files into Video Library for the first time
XBMC Online Manual
Introduction
...

Start by educating yourself:

    Video Library
    Set Contents
    TV Show documentation
    Movie documentation

Step Two - Internalize the documentation

Now that you have read the existing documentation, read it again (yes, this is important and will save you headaches later).

    Video Library
    Set Contents
    TV Show documentation
    Movie documentation

Step Three - Follow the documentation
...

... and then it continues in additional pages about how to help XMBC determine the metadata.

Btw. I can imagine it feels like you're drinking through a fire hose...

The REs... JR or the community would establish initial rule sets.   And then these act as a sequence/chain of filters, or transformation rules, that help canonicalize the input into a more normative form, which can then be used as the basis of confirmations / look-ups.  A feedback dialog could be used to show a Before -> After scheme that shows populated fields, such as that done by the existing Rename dialog for pathname transformations. For simple cases, where naming structures are relatively straight-forward, the job is easy.  The dialog should allow rule chains, and have some basic defaults rules (and perhaps chains), such as Underscores_to_Spaces, Remove Date, Sequence_Episode, etc.  By using simple chains instead of a single complex RE, feedback can be given along the way, and users can explore what works.  Patterns that don't match are skipped, no transformations or capturing occurs (and these would be handled in subsequent runs or by adding additional rule in a chain).

The community could help to add / catalog additional rules.  Great thing about using REs - they are *well* documented and there is *plenty* of help available, and they are in common use (but many may not even know they are used under the surface).

Anyway, I hope this is helpful, and not trivial or belaboring.

[rick.ca]

The thing I keep hearing is that "XBMC just does this automatically."  Maybe I'm not understanding, because setting up regular expressions is the opposite of something that just automatically works.

First of all, I hope it's understood that XBMC and probably most other programs that perform the same sort of function (for whatever purpose) are using regex. Furthermore, whether you expose the regex to the user or not, it's still the most mathematically elegant way to recognize patterns. But how you code the solution is your business, so I assume your question really pertains to properly defining the user need...

I'm not familiar with it, but I'm sure "XBMC just does this automatically" means that once configured correctly (and, yes, that necessarily includes providing regex suitable to the circumstances), it just works. In other words, with a set of regex that will match any form of filename I might use for video, I can then just include my video folder(s) in Auto-Import and all my files will be properly recognized and tagged. Equally important, if I encounter one exception, I know the fix can be to modify one regex or add another, and I'll never have to worry about that exception again.

I hesitate to question your ability to create an algorithm that does it all, but I suspect you must be making the assumption users will magically follow one of some finite set of file naming schemes you believe is sensible. No, I don't need to go that far. I don't believe you can create one fixed algorithm that will handle even just all of different perfectly sensible ways to organized files in a file system. And even if you could, why would you want to maintain such a thing?

Some things are obvious like "S3E21".

Yes, there's probably no more than a dozen commonly used ways to denote the series and episode. But that's just one component of several for just one media type. Even if you could create an algorithm that would handle most of the commonly used schemes, you would then be faced with educating users about what is recognized and what is not. And then you'll find yourself trying to advise users to rename their files while they're asking you to modify your algorithm (and, take my word for it, they'll all tell you they have at least 1,000 files matching the pattern you left out).

Adding a simple list of programs would make the algorithm even better.

Incorporating a list of titles recognized by a meta data provider being used is not a bad idea. For searches of the meta data database to work, you would have to use the correct name anyway. But the user-friendly way to implement that would be to do so internally and not attempt to enforce it on the filename (i.e., your system might ask once, "do you mean this series?"). This particular list, however, doesn't appear to be a good choice. From the number of omissions I can readily spot, I would wonder too about the accuracy of the items on the list.

What is it about the system I outlined in my previous post that concerns you? It seems to me you could go as far as you please in making such a thing user-friendly. First of all, simply being populated with a list of expressions that will match all the schemes most users are likely to use will result in a it-just-works-out-of-the-box experience for most users. Most of the remaining users could deal with whatever doesn't work by picking something from a list that appears to be a match, modifying it if necessary, and placing that expression in the right position in the list of expressions. As I mentioned, that pick list would be of file pathname descriptions, not the regex themselves. Only the 0.1% of users who are actually comfortable with regex would actually choose to add a regex directly.

Like a number of similar matters in MC, the issue is not about providing something that "just automatically works." That's because nothing "just automatically works" all the time. To be truly user-friendly, your solution needs to do that and provide the means to easily configure the thing so it does work when it fails to do so automatically. Even if users find the unavoidable failures annoying, what they'll want to hear is, "Don't worry—it can be configured to do anything," not, "You're SOL—change your filenames."

[MrC]

...and a list of titles is nothing more than a long list of degenerate REs. ;-)

[rick.ca]

The community could help to add / catalog additional rules.  Great thing about using REs - they are *well* documented and there is *plenty* of help available, and they are in common use (but many may not even know they are used under the surface).

And something that would make regex much easier to deal with in this particular application...The configuration system could offer a preview of the existing files matching a selected regex, and display the variables that would be extracted. That would be far more effective than trying to rely on other readily available regex testers that would not have direct access to the actual filenames in question.

...and a list of titles is nothing more than a long list of degenerate REs. ;-)

Hmm. If we had such a system for expression fields, and/or a Regex() function were added to the Expression Language, we could replace all these boring tags with expression fields dynamically update from the filename.

[MrC]

And something that would make regex much easier to deal with in this particular application...The configuration system could offer a preview of the existing files matching a selected regex, and display the variables that would be extracted. That would be far more effective than trying to rely on other readily available regex testers that would not have direct access to the actual filenames in question.

Yeah, exactly.  I was thinking that capture groups would be ordered list of db fields, so not only would $1, $2 (aka \1, \2) be captured as the numbered captures, they would also be assigned to the ordered list of fields, as in:

(some title RE)(some date RE)\.mov -> [title] [date]

where [title] and [date] are pulldown field selectors and columns that could be added.  Adding a new capture group populates the next added column.  Additional columns could be added to duplicate existing captures.  It exactly equivalent to perl's :

($title,$date) = (some title RE)(some date RE)\.mov

Hmm. If we had such a system for expression fields, and/or a Regex() function were added to the Expression Language, we could replace all these boring tags with expression fields dynamically update from the filename.

Oh, man, now I'm getting excited.

[rick.ca]

Oh, man, now I'm getting excited.

Good. Then I can count on you to explain how changing any such fields should also dynamically change the filename, and all the implications of that. My brain hurts.

[Hendrik]

I'm not familiar with it, but I'm sure "XBMC just does this automatically" means that once configured correctly (and, yes, that necessarily includes providing regex suitable to the circumstances), it just works. In other words, with a set of regex that will match any form of filename I might use for video, I can then just include my video folder(s) in Auto-Import and all my files will be properly recognized and tagged. Equally important, if I encounter one exception, I know the fix can be to modify one regex or add another, and I'll never have to worry about that exception again.

XBMC comes with a default regex which works already pretty good, and for the advanced users you can modify the regex in a configuration file. On their forum there are several threads where people keep improving the regex, with some hints where to put it in the config file for people just wanting to update it.