Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[glynor]

Note: This was split from this original thread. I've inlined some of the original content to make this discussion make more sense.

Substring searches are relatively slow operations (not horrible, but slower than some). Because of this, and because they can be fragile (they don't persist if you re-arrange your disk structure), it is usually best to limit [Filename] contains type searches where possible in your media views.  Stacking many of them can make a view slow to open and refresh.

If you need to parse the filename for some kind of metadata, you can usually accomplish the same thing, better, by using a Tag on Import rule to apply a tag to the files in question at import. Then the substring search only needs to happen once at Import time, rather than for each file in the View every single time you open or refresh it.

Thanks for the tips on tagging, will it also be faster for my use? (which is hiding some files from the view), I guess I would first have some kind of "Hidden" tag and then make a rule checking for that tag?

Yes. I accomplish this kind of thing with "magic keywords" (though if you want, you can make specific custom Fields).

So, I have a special keyword "Hide from Views".  All of the top-level Media Views (except "Advanced" where I want all files to show at all times) in my Library have the rule: -[Keywords]=[Hide From Views]. I can apply this manually when needed, or automatically at Import with a Tag on Import rule (and do both, depending on the situation).

If preferred for your situation, you could do the same with a custom [Hide from Views] Field set to Check view style. I use [Keywords] because I have other "magic" keywords for other purposes, and [Keywords] are a pretty interchangeable field if I ever need to look at or use the metadata in another application.

A few other tips to keep your View searches responsive are:

* Order counts. Remember that Views with no filters at all will show ALL files. The first filter added is applied to every single file in your Library, one at a time.  Then, the next filter in the list gets only the output of the previous one (so, it only has to parse the files that "survived" the first filter).  So, if you are going to include a relatively slow operation, include it at the "bottom" of the search list whenever possible.

* Don't duplicate searches unnecessarily. Remember that Media Views are cascading searches.  If you have a View under Audio in the tree (which already has a [Media Type]=[Audio] filter), then it is useless to include this search in any of it's child views (assuming they are set to include the parent searches, as they are by default), and this just slows it down making it do a search on every file that will always result in True. Move filters up as far in the View hierarchy as is possible for best results.

* Any search or category that does things "on the fly" with files on disk, rather than the Library metadata, is slow, because it has to parse the file structure off of the disk (which is slow and limited by your disk's random access speed).  This especially includes things like IsMissing searches, and the Location category. Limit use of these features for when it is absolutely needed (especially IsMissing is best reserved for special "utility" Views).

* Be wary of complex searches that use Global Variables. They can be handy, but they are slower than other operations because they have to write to the Library, rather than just read. Limit use for when absolutely needed (and there are often ways around this).

[glynor]

Two last things I forgot:

* Circular searches are the death of performance. Be very careful that your searches aren't causing circular logic that loops. MC will try to detect these and short circuit them (rather than hanging) but it still makes your views and searches run horribly slow. Be wary of Searches that point to Searches. You can do this (and it is very powerful) but you have to take care to ensure the two searches aren't "inter-related" and re-running the same logic back and forth.

* Don't do something (over and over and over) with a calculated field or expression if it is "intrinsic" to the file and won't ever change. Instead, make your expression into a Tag on Import rule and apply it to a regular Library Field.

Each Calculated field or Expression has to be calculated each time the field is displayed in a column or used in a search, for every single file in your Library. This is, obviously, slower than just reading from a regular Library Field. So, if you have some characteristic about the file itself, which will never change depending on circumstances, that you want to use, then make a Tag on Import rule and apply it to the file at Import time. Like the substring search thing I mentioned at the outset, this means the calculation only needs to be done once, rather than every time the thing is visible.

Calculated fields and expression columns are best reserved for when the "answer" will change over time. Try not to use them for things that are "intrinsic" to the files and won't ever change after they're created (to calculate characteristics of the files, for example).

Almost none of these things (except IsMissing, circular searches, and Location columns) are super-slow all by themselves. But every little bit helps, and if you avoid "stacking" bad on top of bad, then you will have much better experiences with MC's performance. Basically, have "machine empathy". If you're asking the machine to do the same operation over and over, limit it when possible.

[flac.rules]

Thanks for the info glynor, great stuff. A final question, the "filname(name)" is not a regular library field then? And if not, how do i see the difference between a library field and not a library field when i press the tag-button?

[glynor]

[Filename (Name)] is calculated based on [Filename] (it is like a built-in calculated field). You don't have to worry about that, though. It is just that substring searches (and especially regex) are slower than simple comparisons. That's just a general Computer Science rule for working with databases.

With my later "calculated fields" tip, it isn't something you have to be crazy about (I have a TON of calculated fields), but something to be aware of:

* Don't use them if you don't have to. If you need to, they're fine.
* Don't include a ton of them as columns in a View if you don't really need them there, and they're way off the "edge" where you never see them.
* Be aware that each Category that you add to a view that does calculation (rather than just displaying regular fields) will slow it down. One or two are fine (especially if the View is limited to a subset of files), but a view that displays 100k files, and has 4 or 5 calculated fields used as Categories, is going to open slooooow.

[glynor]

I should note...

[Filename] is a regular Library Field.  [Filename (Path)] and [Filename (Name)] are calculated fields, essentially, which do a substring search on [Filename].  It isn't a huge deal, but if you are going to do a substring search on the filename, then just do it against [Filename] rather than one of the other two "already substring parsed" versions. That way, you aren't making the engine do two substring searches for no reason.

So, if you're going to look for files in your "Music" directory, do this:
[Filename]="\Music\"
instead of
[Filename (Path)]="Music"

The first will be slightly more responsive. We're talking handfuls nanoseconds, but they add up, especially if you are doing them to a bunch of files.

However, it is also worth noting.  [Filename] is a regular Library field.  But, the expression filename() and the other File Path expressions do parse the filesystem (getting answers "live" from disk) and are slower. When possible, use [Filename] and not filename() to do your stuff for searches.

[ferday]

Just to add a little since we all use MC a bit different and have different levels of programming knowledge LOL

I refuse to tag on import (personal and not logical reasons and I know I'm not the only one) so I find calculated fields, especially with variables, work well to limit to small folders like incoming video / incoming audio, etc., then mapping the results to a static field (same results as tag on import but the manual version).  If the tag is going to change I'll keep it in a smartlist and/or a specific view for that tag only

I have found regex to be faster than variables but slower than expression language comparisons, obviously all three methods don't always intersect so you may not have a choice sometimes.  I'm learning those intersections now and it's been an eye opener

And the most important thing I've ever discovered to keep MC fast and fantastic is to spend the time to tag, tag, tag and do it right and do it intensely. All those hours I've spent especially on older files and images have been rewarded in spades now with fast and easy searches and views. 

And thanks glynor as always for taking the time.  I never used (hide from view) tags but it's a great idea

[ssands]

Thanks for the valuable info, Glynor.

Would you consider adding this info to the Wiki?

I think it would be a worthwhile addition.

[6233638]

So, if you're going to look for files in your "Music" directory, do this:
[Filename]="\Music\"
instead of
[Filename (Path)]="Music"

The first will be slightly more responsive. We're talking handfuls nanoseconds, but they add up, especially if you are doing them to a bunch of files.

For what it's worth, I knew that this should be the case, but I just checked a handful of views which I created back when I first started using Media Center.
Switching out [Filename (path)] to [Filename] made a very noticeable improvement in loading up some of my more complex views.
 
Thanks for the great information in this topic.

[glynor]

Would you consider adding this info to the Wiki?

I think it would be a worthwhile addition.

I did already drop a link to this thread in the Media Views article.  That whole section is basically all incomplete, though. The Media Views article is good, but the sub-articles for Standard, Theater, and Media Network Views aren't done at all.

It is just a huge project.

[glynor]

For what it's worth, I knew that this should be the case, but I just checked a handful of views which I created back when I first started using Media Center.
Switching out [Filename (path)] to [Filename] made a very noticeable improvement in loading up some of my more complex views.
 
Thanks for the great information in this topic.

Cool. Glad it helped.

If you have a ton of files and you're doing filename filtering, little things really do add up.

[ssands]

I did already drop a link to this thread in the Media Views article.  That whole section is basically all incomplete, though. The Media Views article is good, but the sub-articles for Standard, Theater, and Media Network Views aren't done at all.

It is just a huge project.

Thanks. It is huge. Your work is appreciated.