Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[wer]

Ok then. We were going by a forum post that said the Mac version was being modified to use libboost in order to match the Windows version.  There were also mentions of the Windows version using the MS regex engine.  So is it a secret?

Perhaps your library has regex_iterator.

Regardless of which library you are using, the goal is the same: return all matches.   


Edit: I see Zybex and I are posting the same thing at the same time...

[Matt]

Well our existing regex implementation just doesn't support returning all the matches.

We're playing with using the regex_iterator.  Too soon to say if it will work, but let's hope.

This thread keeps me busy

[wer]

I'll bet it does.  Thanks Matt.

This thread may be keeping you busy, but it's also made a lot of complicated things much easier for people, and has enabled new things that were previously impossible, so I think it's resulted in a huge improvement in MC.  And you deserve the credit, for starting it and following through on it.

[RoderickGI]

We don't use boost. Haven't for years.

What is used now Hendrik?

The Wiki is out of date. It is probably important that people know what version of Regex is used.
https://wiki.jriver.com/index.php/String_Manipulation_Functions#Regex.28.E2.80.A6.29
https://wiki.jriver.com/index.php/Regular_expression_syntax

I'll update the Wiki, if I know what to say in it.

[Hendrik]

We're using the standard regex implementation available in newer version of C++, using the default ECMAScript grammar. Its the same syntax as the boost regex, just without a megabyte or two of additional dependencies.

[wer]

Here's an idea... I haven't fully thought it through, but I think some people might appreciate it, and and perhaps suggest suggest some refinements...

Beautify(expression): Formats text for nicer display

Beautify would take whatever's in the parens, including expressions, and format it according to the following rules:

Text: Left as is
Numbers: Displayed using the system default thousands seperator, e.g. 12,324
Lists: Displayed with each element seperated by a comma, with an ampersand for the last: Element 1, Element 2 & Element 3
    (some people might prefer "and" but that takes up more space...)


Example: Beautify([Name], released in [Year], and starring [Actors], went on to make $[Gross Revenue].)

Output: Jaws, released in 1975, and starring Roy Scheider, Robert Shaw & Richard Dreyfuss, went on to make $470,654,000.


We have functions already for some of these sorts of things, although nothing simple for making a list pretty, but this would just be a shortcut for people. I didn't include anything for date fields because they already have their own function with exhaustive options.

[Hendrik]

Example: Beautify([Name], released in [Year], and starring [Actors], went on to make $[Gross Revenue].)

That sort of syntax wouldn't work, because the function would be passed one string with the tokens already replaced. Its a basic expression reality, [..] tokens are replaced before the function is called - hence all the escaping shenanigans in those functions that absolute want a full expression, and not replaced values.

The best option would be calling it specifically for every field you want formatted, like eg. the date formatting function works. In your example, for example, you also don't want [Year] to get a thousands seperator, noone understands 1,975

eg.
[Name], released in [Year], and starring Beautify([Actors]), went on to make $Beautify([Gross Revenue]).

Since the limited number of types, personally I would rather have a number and a list formatting function then some opaque "beautify" function. FormatNumber could be expanded to add an option for thousands seperators, just leaving a list function?

[wer]

Sure, I don't see why not.  The only part that can't be easily done now is beautifying the list.  So just a list beautify would probably be enough.  It's was just about making it easy for people.

[MikeO]

I am not sure if its already covered but a "Split" Function

Classical track names tend to be in a format  -- Symphony No. 1 in C Major, Op. 21 : 1. Adagio molto - Allegro con brio -- alas not standardized.

Its common to need to to split off the Composition and the Movement into Custom Tags i.e. either side of the Colon (delimiter) (at the moment it involves a Move/Copy then Find and Replace the bit you don't need.

So an expression where you can define the Split([Tag Name], Delimiter, Left/Right of Delimiter)

"Delimiter" Colon in this case (its often a Hyphen or a Semi Colon
"Left Part" or "Right Part"

Composition = Split([Name], ":", Left)
Movement = Split([Name], ":", Right

A more complex version maybe to index the delimiter in say Beethoven: Symphony No. 1 in C Major, Op. 21 : 1. Adagio molto - Allegro con brio where the Colon appears twice (or n)

[Hendrik]

You can use ListItem() for that, treating your string as a list delimited by a colon, since you can pass a custom delimiter to those functions

ListItem([Name], 0, :) - for the first part
ListItem([Name], 1, :) - for the second part

[lepa]

And to be precise with your example it needs to be " : " so
ListItem([Name],0,/ :/ )
ListItem([Name],1,/ :/ )

E:  Hendrik was faster...
Anyway here is wikilink:
https://wiki.jriver.com/index.php/List_Manipulation_Functions#ListItem

[lepa]

ListRemove(list, value, mode) default mode 0=index, 1=value

Remove by index:
ListRemove(a;b;c,1,0) ==> a;c

Remove by value:
ListRemove(a;b;c,b,1) ==> a;c

Would like to promote myself once in case this was  just not seen 
So ListRemove() would be useful and also good companion for ListFind()

[wer]

Agreed, ListRemove would be useful.

[Matt]

I'll try to sneak ListRemove into a coming build.  Thanks!

[lepa]

Great.
Thanks Matt!

[wer]

I've been doing some preliminary testing of the new Regex-2 in .90.

Test string:
F. Murray Abraham [Antonio Salieri];Tom Hulce [Wolfgang Amadeus Mozart];Elizabeth Berridge [Constanze Mozart];Roy Dotrice [Leopold Mozart];Simon Callow [Emanuel Schikaneder];Christine Ebersole [Katerina Cavalieri];Jeffrey Jones [Emperor Joseph II];Barbara Bryne [Mrs. Weber]

At first I thought that positive lookaheads weren't working, because
[\w\s\.]+(?=\s\[)
returns no results.

But then I tried:
([\w\s\.]+(?=\s\[))
which is just nesting the above in a capture group, and that returned correct results.

So I'm gathering that Regex-2 will only return a result for an explicit capture group, which I guess is a holdover from the R1..R9 functionality.

Anything else tricky we should be aware of or test for?

[wer]

I've been doing some more testing of Regex-2, and have found something that might be a bug.

Consider a string with letters and numbers...
a$b%c!

(\W) is a valid capture group for Regex(), and returns $;%;!
(\w) is a valid capture group for Regex(), and returns a;b;c

If you use alternatives with multiple capture groups, the list returned has empty entries (two semicolons together)
(\W)|(\w)     returns   ;a;$;;;b;%;;;c;!;

If you use this form instead, with only a single capture group:
(\W|\w)
you get correct results:  a;$;b;%;c;!

I think Regex(a$b%c!,/#(\W)|(\w)#/,-2)  returning a crapped up list is probably buggy, but at best it seems to me to be bad behavior that will confuse people.

If Regex-2 can only support one capture group when using Alternatives, that is a livable limitation, but I think it would be preferable to fail more gracefully.

Matt could you comment?

[zybex]

I think this behavior is actually desirable. The empty list elements can be removed with some other function (ListClean should have a mode for that...)

Consider this example:

save(Population of Germany is 83M/, France is 67/, UK is 66 millions, var1)
regex([var1],/#(\w+) is (\d+)#/,-2)

ouput: Germany;83;France;67;UK;66

This allows you to get multiple captures from each list item, separated out into a new list.
If you want both captures in the same list item you can just change the regex to have a single capture group, like you mentioned.

[zybex]

@Matt,
Number() is not capturing negative numbers.
Also not capturing things like "1,234.56" (it returns 1) or "1 234.56" correctly, which are correct ways to represent values, though in this case I'm not sure it should capture.

Perhaps add more args to the function?

Code: [Select]

Number(string, mode, decimalSeparator)mode 0 - strict capture, just -1234.56
mode 1 - flexible capture for things like 1,234.56 or 1 234,56 or 1.234,56, etc

decimalSeparator could default to "."

[zybex]

@Matt,
I've mentioned this before, but here goes again - a new mode for ListClean() to:
- remove empty entries - ;; or ; ;
- trim each entry (remove spaces)

Thanks!

[Matt]

Also not capturing things like "1,234.56" (it returns 1) or "1 234.56" correctly, which are correct ways to represent values, though in this case I'm not sure it should capture.

You need to escape commas like this:
Number(1/,234.56)

Otherwise it's another parameter in the function.

[zybex]

Duh. Silly me.

I thought Number() would always return a type that could be directly used in Math(), but apparently that's not the case - it's still a string.

So Number(1/,234.56) returns 1,234.56, but Math(Number(1/,234.56) + 1) still won't work directly.

[wer]

I thought Number() would always return a type that could be directly used in Math(), but apparently that's not the case - it's still a string.

So Number(1/,234.56) returns 1,234.56, but Math(Number(1/,234.56) + 1) still won't work directly.

Actually I'd forgotten Number() was one of mine! 

I tested and Math(Number([Name])+1)  works correctly.

However, there is a bug in the Number() function:

If [Name]=Concerto No. 1 in B flat: I. Allegro

Number([Name])  returns .
but it should return 1

[wer]

Matt, GroupCountQuery seems to be working fine now, except for one thing.

If the field to group by is a list with multiple items, the function always returns 0.

So if [Director]=Director 1;Director 2

GroupCountQuery(Director,Name) always returns 0.

If the Director 2 element is not present, so [Director] only has a single element, the function returns the count as expected.

[Matt]

Actually I'd forgotten Number() was one of mine! 

I tested and Math(Number([Name])+1)  works correctly.

However, there is a bug in the Number() function:

If [Name]=Concerto No. 1 in B flat: I. Allegro

Number([Name])  returns .
but it should return 1

This is a pretty tricky one because the dot looks like a decimal.  I'm going to try to make the parser smarter tomorrow.  Wish me luck 😄

[wer]

Well if the string had "No.1" without a space, I could understand returning ".1" but if you look carefully, there is a space in "No. 1" so returning 1 should be unquestionable. And returning just the . by itself must always be an error.

In fact, maybe "No." should be a special case because it is an abbreviation, so that "No.1" without a space would return 1 instead of .1  since I think that is what people would usually want.

I know you'll get it Matt!  Good luck!

[wer]

I just noticed something strange...

GroupSummaryQuery(Album,Track #) works as expected: it reports the total number of tracks (really it just counts different values of Track #, but because each track has a different #, it comes out the same).

But GroupSummaryQuery(Album,Disc #) returns nothing at all.  It should work the same as with Track #.

[Matt]

I just noticed something strange...

GroupSummaryQuery(Album,Track #) works as expected: it reports the total number of tracks (really it just counts different values of Track #, but because each track has a different #, it comes out the same).

But GroupSummaryQuery(Album,Disc #) returns nothing at all.  It should work the same as with Track #.

Here's the comment from building a summary string for Disc #:
// always return empty for a disc number since there's no fast way to count the number of discs
// and also because sorting links MF_DISCNUMBER and MF_TRACKNUMBER and we'd rather ignore MF_DISCNUMBER
// for grouping files during sorting

So it's by design for now.

[Matt]

Next build will have a lot of changes from this thread.  Thanks!

Fixed: ListGrep was not honoring the mode switch for case sensitivity.
Fixed: The example for ListFind had the wrong function name.
Fixed: The number function did not pull negative numbers.
NEW: Added a ListRemove expression.
NEW: Added modes 3 (remove empty strings) and 4 (trim strings) to the ListClean(...) expression function.
Fixed: List fields with multiple values did not work properly with the GroupCountQuery function.
Fixed: The number function searches for actual numbers after a dot so something like (. 1) won't get read as just a dot.

[zybex]

Thanks for listening Matt!

[wer]

Hi Matt,

Your GroupCountQuery fix did correct that one specific problem, but there's another case with semicolons:

[Director]=Joe Russo;Anthony Russo
GroupCountQuery(Director,Name) now works ok. This is what you just fixed.

[Director]=Joe Russo
GroupCountQuery(Director;Studios,Name) works correctly, counting matches with both Director and Studio

[Director]=Joe Russo;Anthony Russo
GroupCountQuery(Director;Studios,Name) still always returns 0. This case is still broken.


Additionally, if the Field to Group By contains special characters, the function always fails and returns 0:
[Director]=Alfonso Cuarón
GroupCountQuery(Director,Name)   always returns 0
GroupCountQuery(Director;Studio,Name) always returns 0

The same thing happens with any other accented character, as in François Girard, Josée Dayan, or Måns Mårlind

[Matt]

[Director]=Joe Russo;Anthony Russo
GroupCountQuery(Director;Studio,Name) still always returns 0. This case is still broken.

Studio is not a stock field.  Studios is, and using that works for me.

Additionally, if the Field to Group By contains special characters, the function always fails and returns 0:
[Director]=Alfonso Cuarón
GroupCountQuery(Director,Name)   always returns 0
GroupCountQuery(Director;Studio,Name) always returns 0

The same thing happens with any other accented character, as in François Girard, Josée Dayan, or Måns Mårlind

This should be fixed next build.  Thanks for the report.

[Doof]

Another of my "Unless somebody already knows a way to do this" requests: As I've been writing increasingly larger and more complex multi-line expressions I often find myself wishing I could "comment my code" to help with all the things that comments usually help with when coding. I'd love some kind of no-op function or expression shorthand that will let me leave comments without affecting the output.

I've been contemplating using either If(0, // This is my comment,) or Left(// I wish there was a better way, 0). They work, but I can't help but think that both of these will have an impact on performance even a little, and that adds up when this is applied to 200k+ files. I've already had to back off some of my more ambitious expressions as they rendered MC unusable, so I'm trying to optimize wherever I can. If you could somehow create a //(This is much cleaner) function that just literally does nothing with its input, that would scratch this weird itch. Again, unless somebody else already has a better way?

Or am I overthinking this and this is just a crazy idea?

[wer]

Actually that was just a typo in my post (corrected). Studios was what I was using in the expression column, as I have no [Studio] field without the s, and it was returning positive results for some films and 0 for others, which precludes an error in the actual expression column.

However, today it's working; I don't know what was going on the other day but if I can reproduce it again I'll let you know. It may have had to do with restarts; I had issues with the last update crashing when I loaded a DSP preset when I was testing that, but the problem went away after a reboot so I didn't report it. 

[wer]

Doof, I think maybe you're overthinking it. Have you tried using the Syntax Highlighter? That has helped me make sense of some gnarly expressions.  I think adding comments to short but inscrutable expressions could actually make it harder to read.  Regular expressions don't have comments, and those can be much more opaque than any MC expression.  And as you noted, you can preface everything with a null If() if you really wanted to.

[zybex]

I'd love some kind of no-op function or expression shorthand that will let me leave comments without affecting the output.
(...) Again, unless somebody else already has a better way?

The problem is also the whitespace - any end-of-line CRLF is also added to the output.
That said, this is a neat way to add comments:

- create a field called "X" (or anything else), mark it as Hidden (not necessary, just to hide it from regular field lists)
- now you can comment the code like this:

Code: [Select]

This is my actual code [X,this is a comment!]/
code continues here
The slash at the end of line removes the CRLF from the output.

[Doof]

Doof, I think maybe you're overthinking it. Have you tried using the Syntax Highlighter? That has helped me make sense of some gnarly expressions.  I think adding comments to short but inscrutable expressions could actually make it harder to read.  Regular expressions don't have comments, and those can be much more opaque than any MC expression.  And as you noted, you can preface everything with a null If() if you really wanted to.

The desire for comments is more to help future me remember what's going on. And in this case it actually was a series of regexes that made me wish I could comment what it was they were trying to match against so I could keep them straight from each other.

The problem is also the whitespace - any end-of-line CRLF is also added to the output.
That said, this is a neat way to add comments:

- create a field called "X" (or anything else), mark it as Hidden (not necessary, just to hide it from regular field lists)
- now you can comment the code like this:

Code: [Select]

This is my actual code [X,this is a comment!]/
code continues here
The slash at the end of line removes the CRLF from the output.

Brilliant! This is exactly what I was looking for! Thanks!

I created a library field called simply /. To be honest, I'm surprised MC let me do it. But now I can just do this as my comment [//, This is my comment]/, and it works exactly as expected! Since the / needs to be escaped, I used just a single slash so that when I actually type it I have to use // so it even looks like the type of comment I'm used to. Perfect!

[wer]

Neat. That is both clever and unexpected.

[wer]

So Matt, going back to Hendrik's suggestion, how about this:

ListBeautify(list): Beautifies a list for output, separating items with commas and the last item with an ampersand.

ListBeautify(A;B;C;D) outputs
A, B, C & D

Just a simple convenience function.

[Matt]

So Matt, going back to Hendrik's suggestion, how about this:

ListBeautify(list): Beautifies a list for output, separating items with commas and the last item with an ampersand.

ListBeautify(A;B;C;D) outputs
A, B, C & D

Just a simple convenience function.

Sure thing.  Anything else it should do other than outputting like A, B, & C?

Thanks.

[zybex]

Perhaps call it ListFormat() ?

ListFormat(list, mode, newSeparator)

ListFormat(a;b;c, 0) = a; b; c
ListFormat(a;b;c, 1) = a, b & c
ListFormat(a;b;c, 2) = a, b, & c
ListFormat(a;b;c, 3) = a, b and c
ListFormat(a;b;c, 4) = a, b, and c
ListFormat(a;b;c, -1, //) = a/b/c

Modes 2 and 4 are fillers, really...

[wer]

I'd originally though it might more generally beautify text for output (numbers, etc) but Hendrik didn't think the syntax would work, so the idea was to restrict it just to lists.

I think it looks better if there is NOT a comma before the ampersand.

So A, B, C & D
Not A, B, C, & D

Plus it takes up less space. That's why I used the ampersand instead of and, to conserve space, so to avoid line wraps or exceeding column widths when possible.

[Matt]

Thanks guys.  I'll use the ListFormat(...) syntax and allow numbers to specify different things.  If there's one you want once the build ships, just let me know.

Thanks again

[wer]

Ok Matt, since you've been making short work of these simple ones we've been giving you, how about a complicated one? 

FieldQuery: Return a list of matches based on a list of fields to search, from a selected scope of files.

Syntax:
FieldQuery(fieldlist,valuelist,return,mode,scope)

Fieldlist: list of fields to search, e.g. Director;Studio

Valuelist: a value to search for each field in fieldlist (could be an expression)

Return: The single field that is returned when a match is found, values from files returned as a ; delimited list

Mode: 0=contains
     1=exact match

Scope: Audio=1/Video=2/TV=4/Image=8  OR'd together  (searches globally)


Examples:

FieldQuery(Director,Ridley Scott,Name,1,2) returns
Alien;Blade Runner;Gladiator

FieldQuery(Actors,Al Pacino,Name,0,2) returns
The Godfather;Heat;Scarface

FieldQuery(Actors;Director,Al Pacino;Michael Mann,Name,0,2) returns
Heat

If you think we should, we could add a final option, context 0=global, 1=current file scope in view)

[zybex]

Slight change: no need for mode -1, the custom separator can be on mode 0 instead. This way it also works as a default mode = 0, default sep = "; "

ListFormat(a;b;c, 0) = a; b; c
ListFormat(a;b;c, 0, //) = a/b/c
ListFormat(a;b;c,, //) = a/b/c
ListFormat(a;b;c) = a; b; c

[zybex]

FieldQuery(Director,Ridley Scott,Name,1,2) returns
Alien;Blade Runner;Gladiator

This is a nice one. Makes me think, if you're were using a relational DB you can could just go ahead and add an SqlQuery() function 

Code: [Select]

SqlQuery(SELECT Name FROM Files WHERE Director='Ridley Scott' AND /[Media Type/]='Video')

[RoderickGI]

Scope: Audio=1/Video=2/TV=4/Image=8  OR'd together  (searches globally)

Scope needs some more thought, because TV is Video. You are mixing selection based on [Media Type] and [Media Sub Type].

if you're using a relational DB

Unfortunately, MC doesn't use a relational database. If it did, we wouldn't be discussing all this here.

[glynor]

Scope needs some more thought, because TV is Video. You are mixing selection based on [Media Type] and [Media Sub Type].=

No. TV is its own [Media Type], for entries in the TV Program Listings. TV Show is a [Media Sub Type], generally used for Videos, but wer is right. He did forget Documents though.

[wer]

I didn't forget Documents, I chose not to include it, the same way I didn't include Data or Playlist.  This because those are unlikely to share field contents that would make the search useful, and I wanted to minimize the number of cases.

I suggest we debate details like that after we find out whether Matt tells me I'm off my rocker for even suggesting such a thing, or not.  This is a significantly more complex function than we've asked for previously.   

[glynor]

I should note, though, that I’m not sure why such a function would need a scope like that at all. That seems like a redundant, and oddball, addition to the Expression Language in a single function.

Why wouldn’t you just limit the scope on the input side with the much more capable Search Language?