INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Global var issue and musings  (Read 12412 times)

MrC

  • Citizen of the Universe
  • *****
  • Posts: 10462
  • Your life is short. Give me your money.
Global var issue and musings
« on: July 05, 2013, 01:15:36 pm »

In the 18.0.207 thread, I wrote:

Quote
I think I'm seeing a bug when selecting a value in a pane column, when the column values are derived from global variables.  First, here's a link to the movie that shows the problem:

    https://www.dropbox.com/s/kv9gmv2885mddya/possible%20bug.mp4

Here is how the view is created:

Set rules for file display below.  These just initialize two variables to 0, and then later increment either of the variables depending upon the value of the field [Custom]:

[=save(0,exc_[Artist])1]=1 [=save(0,inc_[Artist])1]=1 [=if(isequal([Custom],EXCLUDE), save(math(1 + load(exc_[Artist])),exc_[Artist]), save(math(1 + load(inc_[Artist])),inc_[Artist]))1]=1

The library has two calculated user fields which are defined as:

   _v_exc:    load(exc_[Artist])
   _v_inc:     load(inc_[Artist])

The movie shows these in the file list column.  It is clear that clicking on them selects the files that produced the cell value.  So this works.

The movie also shows that the view has a calculated field - this pane column's expression demonstrates the problem:

   math( [_v_exc] & [_v_inc] )

The calculated values are correct - the pane shows both values 0 and 1, as expected, since some of the 90 files have the EXCLUDE keyword in Custom.

However, notice how clicking on either value produces the wrong file list - its all files or no files.

To which Matt replied:

Quote
MrC, you're in uncharted territory.

When you pick '1' in your AND expression pane, it's effectively saying "show files that match these two searches":

Code:

[=save(0,exc_[Artist])1]=1 [=save(0,inc_[Artist])1]=1 [=if(isequal([Custom],EXCLUDE), save(math(1 + load(exc_[Artist])),exc_[Artist]), save(math(1 + load(inc_[Artist])),inc_[Artist]))1]=1 [Media Type]=[Video]

[=math( [_v_exc] & [_v_inc] )]=[1]

The search compiler combines searches, sorts token ordering, etc. to help performance.  It's not really designed have dependencies that require a deterministic evaluation ordering, which only arise when there are save and load calls.

It's possible we could make the compiler smarter for cases like this.

However, I'm not thrilled with packing up the global variable store like this as it seems like a complex solution.  Is there some construct that would allow the same functionality without using variables like that?  Feel free to start a thread.

I understand the explanation, and had previous suggested to others that this is how MC might be working.  I realize these hack techniques are working by a fortuitous side effect, which has surpassed the limits by trying to impose a deterministic evaluation order.  So that's cool.  And I entirely agree, the current hacked solutions are far too complex and JRiver probably should spend no time trying to make this work.

Here's my assessment of the desired functionality:

   - a way to deterministically evaluate an expression sequence across the list of files

Generally the technique is employed for performing some math/counting, or for string manipulation.

A more straightforward method to accomplish this (removing the gross side effect employed now) would require an ordering to MC's operations with a new intermediate stage:

   1) Filter Stage: Set rules for file display (and Search) to filter the list of files
   2) New: Post-filter Iteration Stage: Iterates over these resulting files evaluating an expression sequence
   3) Display Stage: Presents the generated tokens based on view configuration

Stage (2) would require a loop as such:

   Foreach expression
       Foreach file
           evaluate expression

This order is require so that values can be initialized across all files, and then subsequent calculations performed:

   a = 0  (for all files)
   a += 1 (for all files)

There are cases when an expression only needs to be run once (e.g. number_of_tracks = 0), and cases where it needs to run for every file (number_of_albums_per_artist).  Performance optimizations could be done by providing two evaluation sections or lists:

    1. Run Once:
        Evaluate run-once expressions

    2. Run Always
        Foreach run-always expression
           Foreach file
               evaluate expression

Allowing things like this (non-MC pseudo-code):

    1. Run Once:
           1a.  n_tracks = 0

    2. Run Always
           2a.  n_albums_[artist] = 0    (for each file)
           2b.  n_albums_[artist]++     (for each file)

Clearly item selection in the view would have to restart the sequence of operations (which MC essentially seems to do already).  Selection seems tantamount to filtering via Search.

I have no idea how much impact this has on MC code.  But folks want to count things, and aggregate cross-file data.  And these folks seem willing to accept a performance penalty for such capabilities.
Logged
The opinions I express represent my own folly.

vagskal

  • Citizen of the Universe
  • *****
  • Posts: 1227
Re: Global var issue and musings
« Reply #1 on: July 05, 2013, 01:52:38 pm »

To a layman like me it looks like the logic proposed by MrC could be used to introduce global search and replace for Regex as well (and an operation applicable to each item in a field). That would, for me, be even more useful.

But since I understand only a fraction of what Matt and MrC said I might be totally wrong. I just wanted to give input on the priority of new features (that would both be so advanced that normal users including me would still probably need help from MrC to implement).
Logged

MrC

  • Citizen of the Universe
  • *****
  • Posts: 10462
  • Your life is short. Give me your money.
Re: Global var issue and musings
« Reply #2 on: July 05, 2013, 03:20:40 pm »

This topic is primary focused on a method for using the expression language on the set of files: for each expression, evaluate it for each file.  This is inverted logic from the current operation.

Replacement would work within the current implementation: for each file, apply an expression (which just happens to massage a single file's data in some fashion).  It's no different than Left(), or RemoveRight(), or Replace(), for example.

I wholeheartedly agree with your priority bump!  Time for some more advanced constructs so that other tools can go into the bit bucket.
Logged
The opinions I express represent my own folly.

vagskal

  • Citizen of the Universe
  • *****
  • Posts: 1227
Re: Global var issue and musings
« Reply #3 on: July 05, 2013, 05:52:15 pm »

Thanks for the support regarding the suggested priority! I will shut up now since I obviously do not know what you are talking about.
Logged

MrC

  • Citizen of the Universe
  • *****
  • Posts: 10462
  • Your life is short. Give me your money.
Re: Global var issue and musings
« Reply #4 on: July 05, 2013, 07:50:27 pm »

I probably didn't do a very good job clarifying the subtlety, so sorry about that.

For tasks like these, we're using some trickery that just happens to work because of they way MC operates.  It is a side effect in essence, but I've encountered a limitation for which I've not found a workaround.  The trick relies on how MC figures out which files are to be displayed based on the Search query in Set rules for file display, and that MC allows Custom expressions posing as a query.  The trick needs each rule in the query to run first and in order, so values can be pre-calculated in order to show values in a file list cell or in panes columns.

The technique fails in some cases, like the one at the top of the post.  It seems some expression values are calculated and cached prior to those set in the Set rules for file display, so selecting panes values produces incorrect results (see the movie, if you're curious to see it working in some cases but failing in another).

In explaining how to perform these global calculations, it has become clear the trick is too complicated for users to modify for new values, and is too finicky.  But the pattern of requests has been consistent - a means to perform some calculations on the set of files in a view, before the values are shown in the view, so that those values are selectable and usable in the view itself (the first parts of the movie).

Anyway, maybe this helps clarify the ideas a bit for you.
Logged
The opinions I express represent my own folly.

vagskal

  • Citizen of the Universe
  • *****
  • Posts: 1227
Re: Global var issue and musings
« Reply #5 on: July 06, 2013, 12:49:27 am »

Thanks again for the explanation! No offence at all taken. I only meant that this is your thread and your suggestions, and I will not intrude any more by pushing for something other that turned out to have no connection to the issues brought up.
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41821
  • Shoes gone again!
Re: Global var issue and musings
« Reply #6 on: July 08, 2013, 07:46:13 pm »

I wonder if a new class of expression functions that operate on the set of files would be better than using Save(...) and Load(...) with global variables?  Imagine a function to get the average rating, the number of tracks, the total duration, etc. for the current file set.  I think we could cover most the needs with a few functions, and they'd be simple to use and be an order of magnitude faster than using string-based variables.

This idea brings back your request to have a way to say 'only run this expression once' instead of 'run this expression once for each file' when building a pane.  It would be best if the expression engine could just figure this out automatically so that a use wouldn't have to think about this detail.
Logged
Matt Ashland, JRiver Media Center

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41821
  • Shoes gone again!
Re: Global var issue and musings
« Reply #7 on: July 08, 2013, 07:53:01 pm »

I wonder what constructs you'd need to categorize into buckets like:
Classic Rock - 600 files - 50 Artists
Folk - 420 files - 37 Artists
etc.

I think the most efficient would be to have two expressions define the category:
Group by: [Genre]
Display: [Genre] - Summary(# Files) - Summary(# Artists)
Logged
Matt Ashland, JRiver Media Center

MrC

  • Citizen of the Universe
  • *****
  • Posts: 10462
  • Your life is short. Give me your money.
Re: Global var issue and musings
« Reply #8 on: July 08, 2013, 09:16:40 pm »

Just jotting down some thoughts...

I know you have some of this stuff stashed away in the Grouping database, so making those values available would certainly be used.  Most of the requests are for per-album, per-artist/album artist (auto) stats.  Maybe your Summary() function, or Stats(), or (pseudo-)fields such as [duration (total)].  I can see that too many fields become cumbersome.  Its not clear to me that there is really much difference between using one or the other other than its availability in the Field pick-list.

Some of the other requests require conditional operations (e.g. if [rating] is not empty, do operation).

For more complicated calculations, my first instinct would be to have an iterator function, which operates on a supplied category list, and applies an expression on the specified properties for the current file.  But since the expression language doesn't support lval assignment, math operations, nor an obvious place in the UI to specify such a construct, I'm not sure what it looks like.  What would the syntax be for something like the following, and where would be be entered:

  result <op> foreach(category list, expression)

  duration_total += foreach([album],[album artist (auto)], [duration,0])

The red stuff is what's missing.

Edit: to be clear, this is all very low priority for me personally.  I'm happy to provide help where needed to others, and get along just fine with what's available now. So consider this a discussion of needs by proxy.
Logged
The opinions I express represent my own folly.

mark_h

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1851
Re: Global var issue and musings
« Reply #9 on: July 09, 2013, 01:55:03 am »

I wonder if a new class of expression functions that operate on the set of files would be better than using Save(...) and Load(...) with global variables?  Imagine a function to get the average rating, the number of tracks, the total duration, etc. for the current file set.  I think we could cover most the needs with a few functions, and they'd be simple to use and be an order of magnitude faster than using string-based variables.

As long as you don't remove Save() and Load() functionality....  I have about thirty global variables in use throughout MC now (smartlists/user fields/view columns etc)  I know we agreed that variables were "temporary" and "could go away" but I cannot live without them...


Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41821
  • Shoes gone again!
Re: Global var issue and musings
« Reply #10 on: August 13, 2013, 11:22:20 am »

I wonder what constructs you'd need to categorize into buckets like:
Classic Rock - 600 files - 50 Artists
Folk - 420 files - 37 Artists
etc.

I think the most efficient would be to have two expressions define the category:
Group by: [Genre]
Display: [Genre] - Summary(# Files) - Summary(# Artists)

Next build:
NEW: Expression based categories can define separate grouping and display expressions so that you can group by something like artist, but then display something more advanced with counts, number of albums, etc.
NEW: Group-based expression functions like GroupCount(...) and GroupSummary(...) are supported in the new category display expressions.

It should work in Standard View, Theater View, Gizmo, etc.

I'll start a thread sometime in the next few days to explain this better and give some examples.  I think it opens up a lot of neat possibilities.
Logged
Matt Ashland, JRiver Media Center

MrC

  • Citizen of the Universe
  • *****
  • Posts: 10462
  • Your life is short. Give me your money.
Re: Global var issue and musings
« Reply #11 on: August 13, 2013, 02:24:19 pm »

I see that GroupSummary() will provide averages of values, such as Number Plays.

Is there a mode to get a summation instead?
Logged
The opinions I express represent my own folly.

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41821
  • Shoes gone again!
Re: Global var issue and musings
« Reply #12 on: August 13, 2013, 02:38:19 pm »

I see that GroupSummary() will provide averages of values, such as Number Plays.

Is there a mode to get a summation instead?

It decided per field the most logical summary style.

Functions to ask explicitly for mode, mean, min, max, etc. would be a nice addition.
Logged
Matt Ashland, JRiver Media Center

mark_h

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1851
Re: Global var issue and musings
« Reply #13 on: August 14, 2013, 03:58:08 am »

I'm not getting anywhere with these new expressions.  Any examples to help me understand how I might use them?
Logged

MrC

  • Citizen of the Universe
  • *****
  • Posts: 10462
  • Your life is short. Give me your money.
Re: Global var issue and musings
« Reply #14 on: August 14, 2013, 04:05:36 am »

GroupCount() and GroupSummary() can be used in the new TAW's Stats area.

They can also be used in a Categories expression by using different expressions to group by and display.  For example:

Expression to group by:
[File Type]

Expression to display....
[File Type] /(GroupCount()/)

This would group by file types, but show you file types followed by the count of the given file type.
Logged
The opinions I express represent my own folly.

mark_h

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1851
Re: Global var issue and musings
« Reply #15 on: August 14, 2013, 04:41:21 am »

Ah, that's very useful.   Funnily enough I'd recently just used variables to do just the same on a view, so this shortcut is very handy.
Logged

vagskal

  • Citizen of the Universe
  • *****
  • Posts: 1227
Re: Global var issue and musings
« Reply #16 on: August 14, 2013, 12:24:33 pm »

Sorry, I do not fully understand.

The example you gave worked, but it did not work when I tried to display [File Type] /(GroupCount() files, GroupCount([Artist]) artists/) - The GroupCount([Artist]) always showed 0 (I tried it with [Name] instead of [File Type]). When is GroupCount([field name]) useful.

I was not able to use the new expressions in thumbnail text. Is there any way to do that?

If we I can get this to work I think I would need a FormatGroupText(multiple items, one item, no item) expression to get a nice display.

EDIT: I realised now that is should be GroupCount(Artist), then it works. What are the valid parameters inside GroupCount()?
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41821
  • Shoes gone again!
Re: Global var issue and musings
« Reply #17 on: August 14, 2013, 01:01:52 pm »

What are the valid parameters inside GroupCount()?

Empty for a numeric count of the group.

Any field name for a count of unique values in that field for the group (ie. number of artists, number of albums, etc.).
Logged
Matt Ashland, JRiver Media Center

vagskal

  • Citizen of the Universe
  • *****
  • Posts: 1227
Re: Global var issue and musings
« Reply #18 on: August 14, 2013, 01:04:04 pm »

Thanks! Now that was a lightning fast reply!

EDIT: I guess there is a valid reason why fields are not written with brackets inside precisely that function, but it confused me. I wonder if I can put an expression inside GroupCount()... Lots of things to try out. Thanks!
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41821
  • Shoes gone again!
Re: Global var issue and musings
« Reply #19 on: August 14, 2013, 01:19:15 pm »

EDIT: I guess there is a valid reason why fields are not written with brackets inside precisely that function

A field in brackets is shorthand for the Field(...) function.

So:
[Artist]

Expands to:
Field(Artist)



So:
GroupCount([Artist])

Becomes:
GroupCount(Field([Artist]))

Becomes:
GroupCount(Abba)

The program doesn't understand GroupCount(Abba), but it would understand GroupCount(Artist).
Logged
Matt Ashland, JRiver Media Center

vagskal

  • Citizen of the Universe
  • *****
  • Posts: 1227
Re: Global var issue and musings
« Reply #20 on: August 14, 2013, 01:35:54 pm »

Thanks for the explanation. I guess that makes sense to programmers, but it confused me. I hope all will be explained in detail when MrC has had the time to update the wiki.

The recent developments on working on an entire group of files looks promising. Maybe someday we can have the global regex search and substitute for at least all items in a list in one file I have come to appreciate in other software (for text manipulation)...
Logged

MrC

  • Citizen of the Universe
  • *****
  • Posts: 10462
  • Your life is short. Give me your money.
Re: Global var issue and musings
« Reply #21 on: August 14, 2013, 01:56:06 pm »

I'm working on it, but have tackled a larger undertaking which will make (my) life easier in the future (hopefully).

As to Matt's explanation, you need to understand context a bit when you are working with expressions.  The characters:

   [Album]

are literally read by MC's expression parser, and replaced with whatever value is inside Album for the given file.  As Matt indicates, [Album] is really an abbreviation for the function and argument Field(Album).

When you use [Album] inside some function or as an expression, as in:

    Left([Album], 2)

MC reads the expression and parses from the inside of the parens outwards, doing evaluation and interpolation when and where it can and must.  So the expression goes through this sequence of transformations (assuming the value of Album is White Album):

    Left([Album], 2)
    Left(Field(Album), 2)
    Left(White Album, 2)

Thus the Left() function is passed the characters White Album to operate on.

The Group*() functions need to know internally which field you want to operate on, because they do different things depending upon the field passed.  These two are very different for the reasons mentioned already:

   GroupCount([Album])          Wrong
   GroupCount(Album)             Right

because GroupCount() knows nothing about a field called White Album.  The Group*() functions want the name of the actual field, and not its interpolated value.
Logged
The opinions I express represent my own folly.

vagskal

  • Citizen of the Universe
  • *****
  • Posts: 1227
Re: Global var issue and musings
« Reply #22 on: August 14, 2013, 05:00:27 pm »

Thanks, MrC! I understand. It makes perfect sense to programmers (and me when you or Matt explains it). My argument was, however, that it was not intuitive for a regular user. But I hope for more functions to work on several files/items in a list, so it might be good to begin to learn about this difference.
Logged

vagskal

  • Citizen of the Universe
  • *****
  • Posts: 1227
Re: Global var issue and musings
« Reply #23 on: August 15, 2013, 12:25:17 pm »

I still cannot get this to work in thumbnail text. Is it supposed to, and how would that be done in that case?

PS. Would it be possible to introduce a pane preset, like for columns? Amending pane expressions in a lot of views with the same panes but different include flies' rules is a lot of work.
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41821
  • Shoes gone again!
Re: Global var issue and musings
« Reply #24 on: August 15, 2013, 01:17:32 pm »

I still cannot get this to work in thumbnail text. Is it supposed to, and how would that be done in that case?

It works for categories and the new tag window.

If you want it for other places, please provide details and propose what would define the group.

Thanks.
Logged
Matt Ashland, JRiver Media Center

vagskal

  • Citizen of the Universe
  • *****
  • Posts: 1227
Re: Global var issue and musings
« Reply #25 on: August 15, 2013, 01:33:59 pm »

I would like to be able to use the Group() functions in thumbnail text when in a categories view (with thumbnails, of course). The group would be the category after which the current view/level is displayed. A simple example would be a categories view with this order:
[Album Artist (auto)]
[Album]

When looking at the [Album Artist (auto)] level the group would be [Album Artist (auto)]. GroupCount(Album) would then in the thumbnail text display the number of albums for that [Album Artist (auto)]. When at the [Album] level of that view the thumbnail text could display the number of artists for the individual album by using GroupCount(Artist) in the thumbnail text (I always set the [Album Artist] field for non VA albums so this works for me).

If it is possible to get this to work in thumbnail text, a cool addition would be to have a GroupRange() expression to show Albums released XX-XX by one artist.

I might have misunderstood something completely. In this case, please disregard.
Logged

MrHaugen

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 3774
Re: Global var issue and musings
« Reply #26 on: August 27, 2013, 10:55:24 pm »

I'm looking for a way to use this for counting number of episodes in each season of my TV Shows and check if there's anything missing between the first episode and the last. I would also like to use this to count all Watched episodes and Not Watched episodes in each season of all my TV Shows. Then have this as an expression in the info pane so I know which season to dive into.

Is this currently possible with this GroupCount() or GroupSummary() expressions? I've tried with adding a pane column with GroupCount, and it counts alright. But I can't seem to set two items like TV Show and Season, and get the number of files for that combo. Just one group at a time. Do I have to use some global variables for this, or should this stuff be able to handle it?
Logged
- I may not always believe what I'm saying

MrHaugen

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 3774
Re: Global var issue and musings
« Reply #27 on: September 09, 2013, 02:16:02 pm »

Nobody knows, or are everybody so uncertain because of uncharted territory?
Logged
- I may not always believe what I'm saying

MrC

  • Citizen of the Universe
  • *****
  • Posts: 10462
  • Your life is short. Give me your money.
Re: Global var issue and musings
« Reply #28 on: September 09, 2013, 02:23:46 pm »

Think of these functions as simple and limited counters or summarizers.

The GroupSummary() function accepts a "key", which is a field name, and if Matt has coded a summarizing function for this "key", you get a value back.  So it won't work to supply random arguments.

The GroupCount() function is similar, but just adds up the number of things in a category.

Both of these functions are limited in that they can only work currently in locations where there is a set of categories being determined (like a panes column, or in an upcoming new tag action window replacement).

You're wanting something that is still higher level, and other than what we've discussed in the past, there are no other techniques.

Did the method I provided a while ago in another thread not work?
Logged
The opinions I express represent my own folly.

MrHaugen

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 3774
Re: Global var issue and musings
« Reply #29 on: September 10, 2013, 02:33:37 am »

Ok. I see. That's a bit disappointing. I was hoping for a more open system to count everything from everywhere. Matt, can you please consider adding the summary function to work for the actual items of a TV Show and Seasons? As well as a counter on watched and not watched status for the same two? It would give us some cool info tools in the Small Info pane when browsing! This have been a function in other MC's for lots of years, and users love it.

The functions would be rather simple simple to add I believe?
Episode summary: Count number of episodes in TV Shows as well as each Season. Added bonus would be if it was possible to check the number of episodes in each season against the Starting and ending episode number, and logically figure out if there are any episodes missing.
Watched/Not Watched Summary: This could include number of files in each TV Show or Season that have been watched and that's not watched.



MrC. The other suggestion you gave me only worked partially. I'm using lots of views to hide episodes already watched. The counter mechanism you showed me would not work in those cases. As it relies on what's viewed then and there. Not what files actually exists for that category. So, for me it's no good I'm afraid. And it's not ideal even if I had not filtered out watched items, as it does not work on the category above the actual file level :(
Logged
- I may not always believe what I'm saying

MrC

  • Citizen of the Universe
  • *****
  • Posts: 10462
  • Your life is short. Give me your money.
Re: Global var issue and musings
« Reply #30 on: September 10, 2013, 11:41:22 am »

Ah, got it.

It might work to place the counting stats code in the parent view level, which will have access to all the files.  I don't remember the expressions I wrote, so we'd have to revisit the topic.  If you're interested, please reply with the thread reference.
Logged
The opinions I express represent my own folly.

MrHaugen

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 3774
Re: Global var issue and musings
« Reply #31 on: September 11, 2013, 04:13:13 am »

Here's the thread. I'm not sure it this can be done in a level above. Even if it could, it would be a problem displaying the info pane in other levels than the file level I suppose. Perhaps it could format the caption of the season somehow?
Logged
- I may not always believe what I'm saying
Pages: [1]   Go Up