Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[wer]

Thanks.  Lepa and I were talking about this, as he was having issues with slowness of these functions.

I thought it odd, as these functions perform a search essentially the same as filtering in a panes view, and filtering in a panes view is extremely rapid.

Are these Group...Query functions using a totally different search methodology?

[wer]

Matt, there's a lot that could be done to make Regex() more usable, but for starters, could it be changed to either return all non-overlapping matches to a single capture group (like Boost normally would) populating R1..R9 as appropriate, or at least, using the existing multi-explicit-capture-group syntax, not throw away the first 5 matches if match 6 isn't there, because the string was too short? Regex matching against lists of variable length is problematic.

[Matt]

I'll make the next build have this:
NEW: The Regex expression function takes a -2 mode that outputs all the captures as a semi-colon delimited list.

Thanks for the idea.

[marko]

I'll make the next build have this:
NEW: The Regex expression function takes a -2 mode that outputs all the captures as a semi-colon delimited list.

Thanks for the idea.

I thought at first that this would allow for producing a list, from a list, like so...

(I really can't wrap my head around regex, so forgive the laborious explanation coming...)

If the source list contains, amongst other things:

A/1;A/2;A/3;A/4

MrC kindly provided regex to search for "A anything ;" and then remove the "A/". This much I can just about keep up with.
The result, is 1, because the regex stops as soon as it finds a match.
I want, "1;2;3;4"

When I first read this change, I thought this would do it, but, I can't get there. Is this because I misunderstood and it's still not possible, or because I'm not constructing correctly?

I initially thought that by changing the mode here to -2, it would place a string of all matches into [R1], but, I now think that what this actually does is return a list from [R1];[R2];[R3] etc.?
As my expression only ever returns an [R1], my goal is still unattainable, right?

Code: [Select]

if(regex([keywords],/#(!People[^;]+)#/,0),replace([R1],!People\,),)&datatype=[list]
I sat down ages ago to add this to the wiki, and that still hasn't happened yet! Reminds of when we used to send our kids to tidy their rooms.... Never happened

[zybex]

Marko,
I'm not sure I understand what you're trying to do, but I can see several ways of doing what I think you want.
the following expressions assume you want to take out the prefix "!People\" from list elements:

1. Why not just do replace([Keywords],!People\,) ?

2. You can use ListMix() to apply a function to each element of the list:
ListMix(/#Replace([L1],!People\,)/#, 0, [Keywords])
ListMix(/#Regex([L1],!People\(.+), 1)/#, 0, [Keywords])

3. Regex with mode -2:
Regex([keywords],/#!People\([^;]+)#/,-2)

[marko]

Thanks for replying... Here's exactly what I want...

Take this string, a [Keywords] example...

Code: [Select]

!Events\Annual Christmas Get-Together;!People\Family\MarkW;!People\Family\Nettie;!People\Friends\Alex;!People\Friends\David;!People\Friends\John Stevens;!People\Friends\Louise Stevens;!People\Friends\Megan;!People\Friends\Tracy C;!Places\!Edinburgh\Platform 5
Extract all instances of !People\...\...;
Remove !People\ from each instance
return new list: Family\MarkW;Family\Nettie;Friends\Alex;Friends\David;Friends\John Stevens;Friends\Louise Stevens;Friends\Megan;Friends\Tracy C

Is that possible?

[lepa]

Thanks for replying... Here's exactly what I want...

Take this string, a [Keywords] example...

Code: [Select]

!Events\Annual Christmas Get-Together;!People\Family\MarkW;!People\Family\Nettie;!People\Friends\Alex;!People\Friends\David;!People\Friends\John Stevens;!People\Friends\Louise Stevens;!People\Friends\Megan;!People\Friends\Tracy C;!Places\!Edinburgh\Platform 5
Extract all instances of !People\...\...;
Remove !People\ from each instance
return new list: Family\MarkW;Family\Nettie;Friends\Alex;Friends\David;Friends\John Stevens;Friends\Louise Stevens;Friends\Megan;Friends\Tracy C

Is that possible?

I thought this was exactly the point of -2 although havent test it yet myself.
wer gave here some regex example how to catch all the instances between similar patterns. Of course at that time there wasn't -2 option yet so it didn't work then.
https://yabb.jriver.com/interact/index.php/topic,119385.msg867397.html#msg867397

[marko]

I thought this was exactly the point of -2 although havent test it yet myself.

Me too, untill I got up close and personal with it...

I want to capture Everything between !People and ;, and then remove all instances of !People\
My issue being, the regex function stops after it has captured the first instance. I want it to carry on, finding each instance, placing each into [R1] to [Rn], then, I believe, the -2 mode could output my list. As it is with what I have at the moment, I only ever get as far as [R1], so have to fall back on 20 nested if(isequal( functions...

Code: [Select]

listbuild(1,;,if(isequal(listitem([keywords],0,;),people,8),replace(listitem([keywords],0,;),!People\,),),if(isequal(listitem([keywords],1,;),people,8),replace(listitem([keywords],1,;),!People\,),), and so on...20 being an arbitrary number that ensures all people are always captured. I think this is having a pretty bad performance affect in theater view, where on-screen responses to navigation from the remote in my image library take a good three seconds a press.

I was hoping regex could provide a more streamlined and efficient solution, but think that the answer is still, no, hoping someone can say, actually, yes, try this...

-marko

[zybex]

Regex mode -2 should work for that, but I just tested it and it looks like -2 is not yet available/working.
I made that work with this:

Code: [Select]

Replace(ListMix(/#if(Compare(find([L1],!People\),>=,0),Replace([L1],!People\,),)#/, 0, [keywords]),; ;,;)
- Uses ListMix to evaluate each item in the list [keywords]
- for each item, if it contains "!People\", replace the "!People\" with nothing and return the rest; else return blank;
- cleanup the resultng list with Replace() to remove empty list items (I thought there was some other function to do this...)

[lepa]

Code: [Select]

Replace(ListMix(/#if(Compare(find([L1],!People\),>=,0),Replace([L1],!People\,),)#/, 0, [keywords]),; ;,;)

Hah, that's clever  I also thought there was a function to exclude empty items but I couldn't find any. ListClean() could be a good canditate for that...

[zybex]

One new function proposal that would help here:

ListGrep(list, text, mode)

Returns all elements of list which contain text.
Mode 0 - case insensitive
Mode 1 - case sensitive

ListGrep(redneck;blue;red roses, red) = "redneck; red roses"

Then the above code could be reduced to

Code: [Select]

Replace(ListGrep([Keywords], !People\), !People\,)

[marko]

Looks interesting, thanks zybex. Might be a day or three before I can play and feedback, back to work in the morning.

ListGrep would be great too 👍

[RoderickGI]

Regex mode -2 should work for that, but I just tested it and it looks like -2 is not yet available/working.

The new mode isn't available until MC26.0.84, which isn't public yet. However, Marko has been working with a version that includes the mode.

It would be great if someone could test and provide examples for the use of Mode -2 when a Public Release comes out that includes it. Then we can update the Wiki.

[JimH]

zybex, you have mail.

[marko]

Had a very quick play with this, and, it works... with a few 'buts'...

I had to wrap your expression in two expressions... ListClean, removing duplicates still left some trailing semi colons, so, I tested for that, removing if necessary, resulting in quite a cumbersom expression, that nonetheless, worked...

Code: [Select]

if(isequal(Right(ListClean(Replace(ListMix(/#if(Compare(find([L1],!People\),>=,0),Replace([L1],!People\,),)#/, 0, [keywords]),; ;,;),1),1),/;),removeright(ListClean(Replace(ListMix(/#if(Compare(find([L1],!People\),>=,0),Replace([L1],!People\,),)#/, 0, [keywords]),; ;,;),1),1),ListClean(Replace(ListMix(/#if(Compare(find([L1],!People\),>=,0),Replace([L1],!People\,),)#/, 0, [keywords]),; ;,;),1))
I think there's a bug in the the ListMix function because, if there are brackets in the list items, a syntax error is produced, only for that item, the rest are processed correctly. For example...
After running through the expression above, this keywords string:

Code: [Select]

!Events\!Concerts\Take That (Caroline & Vada); !People\Family\Caroline; !People\Family\Vada; !Places\!Glasgow\The Hydro
Returns:

Code: [Select]

Expression Error (Syntax Error); Family\Caroline; Family\Vada
I won't be able to test this against Theater View performance on the HTPC until at least Sunday. Thanks for your help with this zybex.

Do you really think the regex "-2" mode will catch these? How would you get it to to place all the "People" list items into separate captures? My understanding was, that once it dealt with the first instance, it would consider its work done, and stop.

-marko.

[zybex]

Try this for some magic:

Code: [Select]

Replace(Replace(ListMix(if(Compare(find(/#[L1]#/,!People\),>=,0),Replace(/#[L1]#/,!People\,),@@), 0, [keywords]),; @@,),@@; ,)


[Matt]

I think there's a bug in the the ListMix function because, if there are brackets in the list items, a syntax error is produced, only for that item, the rest are processed correctly.

It looks like parenthesis were causing problems.  Next build should fix it.  Thanks for the report.

[Doof]

Still taking requests? If so, if there isn't already a way, I'd love to be able to read environment variables into a couple of my expressions.

Also, I've recently hit a situation where the ability to do date calculations would be super helpful. Even if you could just clue me in to how to manipulate the internal floating-point value to add/subtract seconds?

[mattkhan]

Even if you could just clue me in to how to manipulate the internal floating-point value to add/subtract seconds?

It is days since epoch where epoch is December 30th, 1899 at 00:00:01

+1s is therefore +1/86400

[Matt]

Still taking requests? If so, if there isn't already a way, I'd love to be able to read environment variables into a couple of my expressions.

I'll add this tomorrow.  Thanks for the suggestion.

[RoderickGI]

Also, I've recently hit a situation where the ability to do date calculations would be super helpful. Even if you could just clue me in to how to manipulate the internal floating-point value to add/subtract seconds?

Also, if you want to do math on the floating point number, you need to use the raw format of the field. i.e. [Date,0] shows the raw floating-point number.

So to add 35 seconds to a date stored in the [Date] field: Formatdate(Math([Date,0]+(35/86400)), Date HH:mm:ss)
Or to add 35 seconds to the current time: Formatdate(Math(Now()+(35/86400)), Date HH:mm:ss)  That works because Now() is a function not a field, and returns a floating-point number and not a formatted date.

Also, I think the epoch begins at December 30th, 1899 at 00:00:00, as best I have determined it from past testing in MC. The floating-point number calculation never yields an exact zero value. But what is a second among friends? 
I always thought that was a strange date and time to start from, rather than December 31th, 1899 at 24:00:00 (January 1st, 1900 at 00:00:00), but maybe that was an error in counting leap years or something... Oh, I see it is the Microsoft COM DATE epoch, and the day shift was because of a leap year!

[RoderickGI]

The tooltip for ListMix() includes information and examples from FormatNumber. A copy/paste error.

[RoderickGI]

Try this for some magic:

Code: [Select]

Replace(Replace(ListMix(if(Compare(find(/#[L1]#/,!People\),>=,0),Replace(/#[L1]#/,!People\,),@@), 0, [keywords]),; @@,),@@; ,)

I tested and deconstructed this expression because I thought I would learn something.

Some notes;
1. This expression will only work correctly on MC26.0.84 and earlier, because of this change in MC26.0.85.

26.0.85 (5/27/2020)
5. Changed: Standardized on ";" for the list delimiter instead of "; " for all the expressions.

2. While it might be the safest thing to do, and doesn't seem to change the result, I found that the [L1] components didn't need to be escaped using the /#...#/ pair.
3. This works on the sample [Keyword] list Marko provided. I didn't test beyond that.
4. The expression "fails" and returns "@@" when no instances of "!People\" are found. Fixed by wrapping another Replace() function around it.

What I learned;
a. I was trying to work out what the @@ operators were supposed to do... then realised that they are just placeholders for the Replace functions later.
b. This only works because ListMix() is a list manipulation function, and passes the [L1] value down to functions within its scope.
c. This type of expression does my head in, and I have to read it s.l.o.w.l.y.
d. Version 2 of the Syntax Highlighter for JRiver Expression Language makes reading this code so much easier. Thanks Wer!

So on MC26.0.85 and above, use;

Code: [Select]

Replace(Replace(Replace(ListMix(if(Compare(find(/#[L1]#/,!People\),>=,0),Replace(/#[L1]#/,!People\,),@@), 0, [keywords]),;@@,),@@;,),@@,)or

Code: [Select]

Replace(Replace(Replace(ListMix(if(Compare(find([L1],!People\),>=,0),Replace([L1],!People\,),@@), 0, [keywords]),;@@,),@@;,),@@,)

Thanks for sharing Zybex.

But we really need to get the Regex mode -2 tested and working... which is what I was going to look at today... but Rabbit Hole, down I went. 

[wer]

But we really need to get the Regex mode -2 tested and working... which is what I was going to look at today... but Rabbit Hole, down I went. 

The reason I proposed what has been christened the -2 mode in the first place was for exactly situations like this.  Without it, the Regex() cannot take an industry standard regex expression and return all matches at once against a variable length string.  The -2 mode should make operations like this simple.

I haven't been able to test it however because it's still in beta only.

[zybex]

Thanks RoderickGI, nice analysis

The change to ";" will surely break some code out there. I agree with it though.
I really dislike chained functions like Replace(Replace(Replace...), but most times it's the only way. For me it highlights the need for some extra functions/modes:

- Trim(expression) could take a second argument to Trim that instead of whitespace. Trim(expression, ; ) would trim semicolons on both sides

- ListClean() needs a mode 2 to remove empty elements from a list

- ListGrep(list, value) to return list elements containing "value". It would simplify this expression tremendously. Alternative name: ListFilter()

Regex mode -2 can be used to do this, but it would need some really awkward regex expressions in most cases.

[zybex]

The -2 mode should make operations like this simple.

It should work, but the regex expressions needed can become quite tricky. For this case, it would be something like this (The RemoveRight() it to remove the last semicolon):

Code: [Select]

RemoveRight(Regex([Keywords],/#[;^\s]+!People\\([^;]+;)#/,-2), 1)
Or a general expression to filter a list for elements containing "word":

Code: [Select]

RemoveRight(Regex([Keywords],/#[;^\s]+(.*word[^;]*;)#/,-2), 1)
I've just tested this on the latest beta and mode -2 only returned the first match. Seems it's not working 100% yet.

EDIT: Perhaps I misunderstood how -2 works or how it's implemented. After some more tests, I see it's currently working like this:

a) Regex(this 123 sample 456 text, /#(\d+)#/, -2)               => this outputs 123 (expected: 123;456)
b) Regex(this 123 sample 456 text, /#(\d+)\D+(\d+)#/, -2)       => this outputs 123;456

I expected that -2 mode with a) would output all \d+ occurrences.
The b) case is not as useful as it requires you to know the exact layout of the string you're searching. It's not possible to apply -2 to a list of random length to filter the elements, for instance - my regex examples above to filter the lists will not work with this current -2 mode.

[zybex]

This is a teaser for something I'm doing - yesterday I started building an interactive Expression Language editor where you can just type and immediately see the results.
This is still very unpolished (half-day work), but check out the 2 screenshots. One shows a rendered Tooltip expression as it would show on the Movie Tooltip view, the other one is a scratchpad I used to test several different expressions.

Planned features:
- instant evaluation (by MC via the API, I'm not reimplementing the language)
- syntax highlighting, color customizable
- rendered (HTML) and Text output views
- HTML tag support: font, b/u/i, img with full src path or "tooltip:" for thumbnails, logos, etc
- multiple scratchpad tabs (like notepad++)
- test expression against any movie (currently this in running inside ZRatings, and when you change to some movie in the main Datagrid the expression re-evaluates for that movie)
- function argument syntax/help
- insert fields and function templates from a list
- color selector for <font> ... WYSIWYG is possible, but too much work for now.

Hopefully next weekend I'll have some test version available.
Feedback so far?

[wer]

EDIT: Perhaps I misunderstood how -2 works or how it's implemented. After some more tests, I see it's currently working like this:

a) Regex(this 123 sample 456 text, /#(\d+)#/, -2)               => this outputs 123 (expected: 123;456)
b) Regex(this 123 sample 456 text, /#(\d+)\D+(\d+)#/, -2)       => this outputs 123;456

Zybex, if -2 is working as I intended, it would output 123;456 as per your example A.  That was the whole point of it.

The functionality described in B is useless, because it cannot be used against a string with an unknown number of matches (and it uses non-standard syntax of course).  What you describe in B is also redundant with the already existing R1..R9 functionality.

I can't comment on what Matt actually implemented because I haven't seen the beta, but returning an arbitrary number of matches with a single regex was the goal.

[Matt]

One new function proposal that would help here:

ListGrep(list, text, mode)

Returns all elements of list which contain text.
Mode 0 - case insensitive
Mode 1 - case sensitive

ListGrep(redneck;blue;red roses, red) = "redneck; red roses"

Then the above code could be reduced to

Code: [Select]

Replace(ListGrep([Keywords], !People\), !People\,)

Next build:
NEW: Added a ListGrep expression function to filter a list to a matching string.

Thanks for the idea.

[marko]

Next build:
NEW: Added a ListGrep expression function to filter a list to a matching string.

Thanks for the idea.

Brilliant. This will be useful in three or four areas for me. Thank you both.

[marko]

This is a teaser for something I'm doing - yesterday I started building an interactive Expression Language editor where you can just type and immediately see the results.
<snip>
Hopefully next weekend I'll have some test version available.
Feedback so far?

Looks amazing. Would help enormously when compared to tweaking in notepad++, copy, paste into MC, go back, tweak more, copy, paste, etc.

Will it only work on movie fields?

@Rod,
Thanks for the breakdown and amended expression, which works flawlessly.

[zybex]

It works on any [field], but it's always in relation to some "File" entity (movie/episode/song/...). Not sure how it behaves with the GroupQuery functions.

The expression is evaluated by MC, so whatever works on MC should work here - unless MC has some extra functionality which is not exposed on the API, which is very much possible.

Still, even like it is I find it's already very useful.

[Doof]

It is days since epoch where epoch is December 30th, 1899 at 00:00:01

+1s is therefore +1/86400

you need to use the raw format of the field. i.e. [Date,0] shows the raw floating-point number.

D'oh! I had found an earlier thread and/or wiki article that clued me into the 1/86000 of the epoch value but I couldn't ever get the the math to work. It was this last bit about the raw value that I was missing! Thanks guys! This is going to VASTLY simplify a few of my expressions.

I'll add this tomorrow.  Thanks for the suggestion.

Thanks Matt!

[Doof]

Is there any way to output information from a different file by referencing its key or another identifier?

[Doof]

GroupCountQuery() seems to be a pretty taxing operation for MC to perform on a... largish (200k) library. Could it be sped up at all if we could limit the number of files the query needs to run against? For instance, a query like GroupCountQuery([Album], [Track #]) could be limited to only searching within music files by doing something like GroupCountQuery([Album], [Track #], [Media Type]=[Audio] -[Media Sub Type]=[Audiobook]). Would something like that speed it up?

[wer]

It's puzzling why GroupCountQuery should be significantly slower than the filtering done in a panes view.  It's essentially the same search against the db, and filtering in a panes view is extremely speedy.

[wer]

I can confirm Zybex's findings, that Regex -2 is not working as intended.

The intention was to return all matches of an arbitrary length string that contained an indeterminate number of matches.

For example, in the Boost regex engine (used by MC) we can do the following:

Input: Humphrey Bogart [Role One];Sydney Greenstreet [Role Two];Mary Astor [Role Three]
Regex: [\w\s]+(?=\s\[)
Output(semicolon delimited): Humphrey Bogart;Sidney Greenstreet;Mary Astor

That one regex would output all the actor names in such an input, regardless of whether there was one actor or 20 listed.

The above is exactly what Regex-2 should do.

However, the Regex-2 function only outputs the first match, Humphrey Bogart.

As implemented, to use Regex-2, you must still possess precognition, because you have to know in advance the number of matches that will exist.  This does not work with lists (strings) with a variable number of elements.  You must still provide a number of redundant capture groups EXACTLY equal to the number of matches to be found in that particular string. If you have too few groups, you will not get all matches. If you have too many groups, the Regex() function returns null.  (And that was exactly the problem with the existing R1..R9 functionality.)

This is not helpful, as it is redundant with the pre-existing Regex() functionality and the way it loads results into the R1..R9 variables. Plus, in addition to requiring prior knowledge of the number of captures present, you have to produce a somewhat ridiculous regular expression with multiple redundant capture groups.

The purpose of this enhancement to the Regex() function was to get around those constraints.

[wer]

I've found some issues with GroupSummaryQuery, Matt...

GroupSummaryQuery does not work correctly sometimes IF the field name has spaces in it. 

GroupSummaryQuery(Album Artist,Duration) can be used, but GroupSummaryQuery(Artist - Composition,Duration) can not. [Artist - Composition] is a custom field. Another example, custom field [Artist-Album] can be used, but the identical field [Artist - Album] can not.

The [Artist - Album (Year)] field also can not be used.

My personal favorite because it is so strange:
GroupSummaryQuery breaks (returns null) if the VALUE of the grouping field contains a letter with an umlaut (e.g. Tannhäuser).  So, GroupSummaryQuery(Composition, Duration) works correctly if [Composition]=Tannhauser but breaks if [Composition]=Tannhäuser    A double quote character " has the same effect.

[marko]

One new function proposal that would help here:

ListGrep(list, text, mode)

Returns all elements of list which contain text.
Mode 0 - case insensitive
Mode 1 - case sensitive

ListGrep(redneck;blue;red roses, red) = "redneck; red roses"

Then the above code could be reduced to

Code: [Select]

Replace(ListGrep([Keywords], !People\), !People\,)

This is working an absolute treat. Thank you very much to all involved.

[Matt]

Maybe the regex needs to empty the array between runs?

[wer]

Maybe the regex needs to empty the array between runs?

I'm not sure what this is in response to.  I don't see the connection to what Zybex and I were seeing with Regex-2.  Is there something we need to explain better?

[Matt]

Could someone post a simple RegEx(..., -2) example that isn't working properly?

[wer]

Sure, that's what my previous post had.  Here it is:

Input: Humphrey Bogart [Role One];Sydney Greenstreet [Role Two];Mary Astor [Role Three]
Regex: [\w\s]+(?=\s\[)
Output(semicolon delimited): Humphrey Bogart;Sidney Greenstreet;Mary Astor

That one regex would output all the actor names in such an input, regardless of whether there was one actor or 20 listed.

The above is exactly what Regex-2 should do.

However, the Regex-2 function only outputs the first match, Humphrey Bogart.

Another example from Zybex's post:

a) Regex(this 123 sample 456 text, /#(\d+)#/, -2)               => this outputs 123 (expected: 123;456)

[Matt]

I'm not clear on this example:
Regex(this 123 sample 456 text, /#(\d+)#/, -2)

Isn't there only one capture, so outputting 123 is right?

[wer]

No.  \d+ matches all numbers in the string.  We're trying to return ALL matches.

The current behavior of -2 is functionally equivalent to the existing R1..R9 behavior, in that you have the following problem:

Write a single Regex function that obtains all the numbers from the [Data] list. The data field may have a different number of elements for each file, as follows:

123 Main;1000 Oak;500 people
2000 years;2500 dollars;12 cars;13 ghosts of scooby doo
4 calling birds;3 french hens;2 turtle doves;11 lords a leaping;5 gold rings; 6 geese a laying

Currently, that is impossible. In MC, a different Regex expression is required for each of those lists, to match the exact number of numbers present in each string.

[wer]

Please look at this example:

https://regexr.com/5628d

All numbers are matched (highlighted) with the single expression.

[zybex]

Matt,
Check the boost::regex_iterator:
https://stackoverflow.com/questions/2593288/how-to-use-c-boosts-regex-iterator

This is about repeating the regex capture on the string until there are no more matches.

[wer]

Exactly.  It's about returning all matches for a capture group, as the /g (global) flag would.

There is also the  match_results::captures function.  Take a look here, in the section on repeated captures:
https://www.boost.org/doc/libs/1_33_1/libs/regex/doc/captures.html

[Hendrik]

We don't use boost. Haven't for years.

[zybex]

We don't use boost. Haven't for years.

Good for you, that thing is getting bloated.

std::regex_iterator / std::regex_token_iterator does the same.