Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[mattkhan]

I've noticed multiple items have no black bar detected after a successful run (i.e. the field has 0x0xResXxResY in it after analysis, job reports as successful) but where the video clearly has black bars. I cleared the field & reran the job manually and checked the logs, the final output is

Code: [Select]

0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x142x1914x938, with 3 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x142x1912x938, with 3 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x142x1908x938, with 7 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x142x1906x938, with 8 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x142x1904x938, with 9 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x142x1900x938, with 5 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x142x1892x938, with 6 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x144x1920x936, with 348 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x144x1918x936, with 8 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x142x1920x872, with 10 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x142x1920x880, with 9 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x142x1920x886, with 10 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x0x1920x1080, with 30 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x142x1920x938, with 200 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x142x1918x938, with 271 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x142x1916x938, with 5 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 2x144x1894x936, with 4 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 2x144x1772x936, with 4 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 2x142x1878x938, with 2 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 2x142x1920x938, with 124 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 2x142x1918x938, with 27 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 134x144x1736x936, with 4 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 648x352x1312x784, with 4 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 394x144x1534x936, with 4 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 22x142x1918x938, with 5 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 24x142x1918x938, with 3 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 24x142x1920x938, with 5 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 154x144x1720x936, with 40 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 288x144x1614x936, with 4 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 166x142x1920x938, with 5 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 170x144x1712x936, with 4 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 46x142x1918x938, with 3 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 182x144x1702x936, with 40 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 570x232x1364x868, with 40 hits
0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 198x144x1678x936, with 40 hits
0019956: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 72x142x1918x938, with 5 hits
0019956: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 1598x244x1642x280, with 4 hits
0019956: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 76x142x1918x938, with 2 hits
0019956: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 204x144x1678x936, with 12 hits
0019956: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 204x144x1676x936, with 28 hits
0019956: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 88x150x1918x928, with 3 hits
0019956: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 88x150x1918x938, with 4 hits
0019956: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 218x144x1676x936, with 24 hits
0019956: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 218x144x1672x936, with 12 hits
0019956: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 220x144x1670x936, with 4 hits
0019956: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 222x144x1670x936, with 4 hits
0019956: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 114x144x1748x936, with 4 hits
0019956: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 122x144x1744x936, with 40 hits
0019956: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 504x160x1426x936, with 4 hits
0019958: 4076: Encoders: CJRVideoAnalyzeExtended::AVLog: [AVIOContext @ 0000023746F55E00] Statistics: 222161780 bytes read, 52 seeks
0019959: 4076: General: JRWorker::ProcessCommand: AnalyzeVideoBlackBars succeeded...

the tag still has 0x0x1920x1080 in it despite it apparently clearly detecting the presence of blackbars. I'm not sure how it decides how to pick one value though but, from watching the film itself, it quite clearly simply has blackbars throughout the entire film (i.e. it's completely unambiguously a constant 2.39 throughout the film).

If the analysis is confused about what to pick, I think it needs to do a better job of reporting the uncertainty.
if it's just getting it wrong then that seems like something to improve.

this isn't a one off btw so I don't think this is just one disc being odd

[Hendrik]

Code: [Select]

0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x0x1920x1080, with 30 hits

This line is the problem - it detects some frames which use the full frame and thus does not cut anything.

What it basically does is take all the detections and find the smallest rectangle that doesn't cut any video off. On a clean movie you would usually only have one or two rects, but some detect this noisily, maybe their margins are not cut cleanly or they are older movies. Why it detects the full frame, I can't tell you without the movie at hand.

[mattkhan]

Ok i see, thanks

It would be helpful to log some timestamps I think

I don't know the best way to handle this but current (misdetect with no notification) definitely not ideal

Does it read every frame or sample various timestamps?

Some ideas

Add an extra field which holds a list of all detected values
Allow picking the most common rather than largest
Add a warning if it detects a full image and a variety of common scope formats

[Hendrik]

It jumps through the video at about 10% increments and reads a couple of frames at each spot.

I suppose I can add some kind of status field that indicates how clean a match was, which would help to let us investigate why it gets to that conclusion. I mostly focused on not erroneously cutting off image so far, as thats also easier to check for from the reported values. A full 16:9 frame is a perfectly valid value - may just not match the content.

[mattkhan]

I appreciate doing this perfectly is probably impossible. I think ideally it needs to be able to distinguish between genuinely variable aspect ratio films and ones that are a fixed ratio with some odd frames then have some way to pick the best value as opposed to safest. I guess this might be a user preference though.

My preference would be to signal this via a tag so I can create a view of items that need manual review post import.

I use this to drive jrvr profile choice and I would personally prefer not to fallback to having to manually pick a profile.

[mattkhan]

fwiw I tried https://www.avsforum.com/threads/moviestarter.3245962/ for comparison and it's quick analysis came up with

Code: [Select]


Detected video format: 1920x1080 (SAR: 1), FPS: 23.976, bit depth: 8, EOTF: SDR, runtime: 151 min
Detected audio format: Surround51, 3D Codec: None

Analyzing meta data finished in 0.9 s
Detected black level: 16

Aspect ratio detection using MPV:

Sample added 00:03:01: crop: 1920x1080 (1.78)
Sample added 00:07:33: crop: 1920x1080 (1.78)
Sample added 00:12:05: crop: 1920x1080 (1.78)
Sample added 00:16:37: crop: 1920x1080 (1.78)
Sample added 00:21:09: crop: 1920x1080 (1.78)
Sample added 00:25:41: crop: 1920x1080 (1.78)
Sample added 00:30:13: crop: 1920x1080 (1.78)
Sample added 00:34:45: crop: 1920x1080 (1.78)
Sample added 00:39:17: crop: 1920x1080 (1.78)
Sample added 00:43:50: crop: 1920x1080 (1.78)
Sample added 00:48:22: crop: 1920x1080 (1.78)
Sample added 00:52:54: crop: 1920x796 (2.41)
Sample added 00:57:26: crop: 1920x796 (2.41)
Sample added 01:01:58: crop: 1920x796 (2.41)
Sample added 01:06:30: crop: 1920x1080 (1.78)
Sample added 01:11:02: crop: 1920x1080 (1.78)
Sample added 01:15:34: crop: 1920x796 (2.41)
Sample added 01:20:06: crop: 1920x1080 (1.78)
Sample added 01:24:38: crop: 1920x1080 (1.78)
Sample added 01:29:11: crop: 1920x1080 (1.78)
Sample added 01:33:43: crop: 1920x1080 (1.78)
Sample added 01:38:15: crop: 1920x1080 (1.78)
Sample added 01:42:47: crop: 1920x1080 (1.78)
Sample added 01:47:19: crop: 1920x1080 (1.78)
Sample added 01:51:51: crop: 1920x1080 (1.78)
Sample added 01:56:23: crop: 1920x1080 (1.78)
Sample added 02:00:55: crop: 1920x1080 (1.78)
Sample added 02:05:27: crop: 1920x796 (2.41)
Sample added 02:10:00: crop: 1918x796 (2.41)
Sample added 02:14:32: crop: 1920x796 (2.41)

Aspect ratio candidate found: 1.78 (76.7 %)
Aspect ratio candidate found: 2.41 (23.3 %)

Calculated primary aspect ratio: 1.78 (76.7 % within 1.68 and 1.88)
Calculated secondary aspect ratio: 2.41 (23.3 % within 2.31 and 2.51)
Skipped samples: 0 %

Rounding to: 2.4

Analyzing aspect ratio finished in 4 s


so I guess this track has something unusual about the way the black bars are encoded?

the logging is easier to follow btw given the timestamps as it means I can quickly play and check the relevant scene

[Ekpen]

Code: [Select]

0019955: 4076: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x0x1920x1080, with 30 hits

This line is the problem - it detects some frames which use the full frame and thus does not cut anything.

What it basically does is take all the detections and find the smallest rectangle that doesn't cut any video off. On a clean movie you would usually only have one or two rects, but some detect this noisily, maybe their margins are not cut cleanly or they are older movies. Why it detects the full frame, I can't tell you without the movie at hand.

Hello:
Crop info from a 1080P video is: 0x22x1980x1058
Crop info 4k video is: 0x42x3840x2118.
I have already written about this in an earlier response.
BTW how do you clear the crop field ?
I zapped my entire database field 3 times while testing .
Thanks.
George Omoregie

[mattkhan]

BTW how do you clear the crop field ?

open the tag window, right click the field and click empty

[mattkhan]

I tried ffmpeg for comparison which gives different results depending on where I start the analysis

i.e.

Code: [Select]

ffmpeg -t 180 -i 00003.m2ts -vf cropdetect -f null -

correctly detects crop=1920:784:0:148

similarly

Code: [Select]

ffmpeg -ss 00:01:00 -t 180 -i 00003.m2ts -vf cropdetect -f null -

detects the same

whereas

Code: [Select]

ffmpeg -ss 00:02:00 -t 180 -i 00003.m2ts -vf cropdetect -f null -

detects the same time period (i.e. 00:02:00 - 00:03:00) as a completely different value - crop=1920:1072:0:4

I don't know why this occurs but given that MC skips through the track, could it have the same problem?

[Hendrik]

Without access to that particular file I can't answer any questions about why.

[mattkhan]

I sent you a sample

[Hendrik]

If I do your seek command but output the frames to a file



You might see the problem. We already ignore the first 4 frames at the start, I might increase that to 8 or 10 just to be safe. Although technically seeking could result in even worse results / longer gaps, and LAV for example has a bunch of helpers to avoid outputting those, but re-creating all that in the analysis would be rather overkill (and some particularly bad files still slip by). FFmpeg itself is supposed to only output recovered frames, but seemingly this is not working properly for all videos.

For Blu-rays, I should really read the seeking table in the Blu-ray metadata, rather then blindly seeking in the m2ts, but its quite a lot of effort to do this all manually, so maybe this might come when I have a Blu-ray demuxer for Linux that I can just re-use .. sometime.

You can simulate the fix by using "-vf cropdetect=skip=8"

[mattkhan]

OK i see

if MC samples the track at intervals, how long a section does it analyse?

any downside to skipping a few more frames?

 

[Hendrik]

Shouldn't be a problem. Lets see how it behaves in the next build.

[mattkhan]

Ok great.

[mattkhan]

it seems every film I watch is a problem film, possibly because I'm watching a lot of older films recently....

Dirty Harry appears to be a ludicrously dark film and totally trips up the detection (somewhat understandably as I could barely see anything myself at times)

I know the manual profile selection is coming (soon?) which I am now minded to think is, in the end, the pragmatic solution to this problem.

However i do think there is one simple change that could be added which would improve quality of life enormously in this regard, namely just another field that can be used to indicate whether there is significant "doubt" over the reported crop.

This could be one or more of

* a boolean (true if it's doubtful)
* the max crop (currently the crop is the min crop so output the maximum crop instead)
* the most common crop
* the no of distinct crops

from the logs, all such info seems to be available so just storing that in the library would then enable the user to decide what to do next (which might simply be "watch a bit of the film and decide what to do" or it could be run some other custom analysis)

[jmone]

In testing (31.0.24) I too see the issue where (as an example) John Wick 2 was populated with 0x0x3840x2160 in the database instead of 0x280x3840x1880 (I guess) even though looking in the log summary this was in the minority, eg:

0020225: 21544: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x0x3840x2160, with 48 hits
0020225: 21544: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x280x3838x1880, with 132 hits
0020225: 21544: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x280x3840x1880, with 553 hits
+ many many other lines

I'll see if I can find the no crop scene.   

[jmone]

Nope can't see any areas without a crop in manually scrubbing through the film.  Plenty of all black, but none I could see with content in the letterbox per say.  Can we interpret the log for time stamps (eg the t:xxx)

0001385: 21544: Encoders: CJRVideoAnalyzeExtended::AVLog: [Parsed_cropdetect_0 @ 0000020A63643900] x1:0 x2:3839 y1:0 y2:2159 w:3840 h:2160 x:0 y:0 pts:382027863 t:382027.863000 crop=3840:2160:0:0

[mattkhan]

I am finding it too unreliable so far, too many errors and no way to know (without playing the track) whether it got it right or not. Richer logic in choosing a value would obviously be great but just exposing the analysis info in a useful form would be sufficient for me
 For example, dump a file next to the sidecar with scan info in a parsable format.

[jmone]

The only weird impact for me so far (I'm a all 16:9 display user) is when the crop is too much (rather than too little), eg the John Wick 4 I posted in the beta build thread.  I guess I'm not a user that would take advantage of black bar detection. 

[mattkhan]

when my wife asks out of the blue "what's going on with the letterboxes", you know something is v wrong

[jmone]

SHMBO!

[Hendrik]

I'll work on adding a analysis report into a library field, so you can easily make a smartlist to check the results. It would start with a single line with a status keyword to filter on (eg. how certain it is), and then dump possible matches after.
Once thats available I can look into a heuristic to improve the selection, as it gives me more easy access to loads of data.

[mattkhan]

I've realised that one of the discrepancies is with anamorphic DVDs

for example ffprobe reports for a certain dvd

SAR 64:45 DAR 16:9

MC reports the crop as 2x72x720x504 which yields an AR of ~1.4 but if you multiply that by 64/45 then you get ~2.03

for this particular film, if you set limit=16 and look at various timestamps then you get 720x432 and 720/432*64/45 = 2.37 which seems like the actually correct AR

so there are 2 problems here

1) bad crop, possibly due to the black level
2) handling anamorphic DVDs

but to add even more confusion (for me), this particular track has 2.36 in the Aspect Ratio field which implies something somewhere already knows the right answer Where does this value come from?

[Hendrik]

A crop of 2x72x720x504 yields a frame size of 718x432, essentially the same as you say you get, so the detection seems to be working fine. (our crop values define the active video area, by left/top/right/bottom, so width/height is right-left and bottom-top)

The aspect ratio computation takes into account both the detected black bars, and the anamorphic scaling, so thats where 2.36 comes from, which is also fine (I assume it would round to 2.37 if the width wasnt being cropped by 2 pixel)

Now what actually happens when you play this? I assume something isn't right. Maybe anamorphic scaling doesnt work properly combined with black bar removal at playback time? Because the data seems fine.

Looking at the code, it should actually handle this fine as long as you are in Preserve Aspect Ratio mode or Crop mode. The only video I could find with a similar setup only had a anamorphic factor of 1.067 (720x576 being scaled to 4:3), which isn't super obvious to see, but the resulting image rectangle seems to be spot-on.

[mattkhan]

right yes sorry, I forgot about that when posting

now I realise that the problem, for this film, is that my custom AR field is not aware of that SAR value and this value is what I base the profile rule on, i.e. I have various different configuration for scope vs 16:9 and this is based on that custom field but this is wrong for anamorphic DVDs so it doesn't pick the right profile & doesn't apply the crop/scale I need

I'm not sure how I never noticed the existing Aspect Ratio field before but it seems that contain the right info already so I can just switch to using that

seems I was barking up the wrong tree, apologies for confusion! will retest with the right configuration

[Hendrik]

No worries, appreciate all the testing and feedback so we can nail down this feature. A analysis report should be coming next week. I realized if I want to have a "how good is the analysis" info in it, I already need to do some sort of heuristic right away, so I'm trying to make it just overall smarter.

[mattkhan]

found another example where you need to skip quite a few frames before you get a good output

i.e. same as https://yabb.jriver.com/interact/index.php/topic,136262.msg944074.html#msg944074

Bourne Ultimatum BD
analysis says no crop at all but film is scope all the way through

checking with ffmpeg and comparing to its default options (e.g. limit=24) says there are 2 problems

1) it takes a long time for decoding to create a correct image, skip=20 fixes it
2) it gets thrown off by dark scenes, limit=16 fixes it

it means cropdetect=skip=20:limit=16 produces the correct 1920x800 for all periods I tested

Code: [Select]

for i in 10 20 30 40 50
do
  echo $i
  ffmpeg -ss 00:${i}:00 -t 2 -i /media/films/The\ Bourne\ Ultimatum/BDMV/STREAM/00009.m2ts -vf cropdetect=skip=20:limit=16 -f null - 2>&1|awk '/crop/ {t=substr($(NF-2),3); split(substr($NF,6),c,":"); print t,c[1],c[2],c[3],c[4]}'
done


sample output with the above

Code: [Select]

50
0.872867 1920 800 0 140
0.914578 1920 800 0 140
0.956289 1920 800 0 140
0.998000 1920 800 0 140
1.039700 1920 800 0 140

and without either

Code: [Select]

50
0.122122 1920 1072 0 4
0.163833 1920 1072 0 4
0.205533 1920 1072 0 4
0.247244 1920 1072 0 4
0.288956 1920 1072 0 4
0.330667 1920 1072 0 4
0.372367 1920 1072 0 4

and with just skip, things get v strange

Code: [Select]

50
0.872867 1360 800 198 140
0.914578 1360 800 198 140
0.956289 1376 800 188 140
0.998000 1376 800 188 140
1.039700 1376 800 188 140
1.081411 1376 800 188 140
1.123122 1376 800 190 140


[Hendrik]

I have been planning to look into the limit, but some videos also have a bit noisy black bars so its a bit of a balance. The question would be how impactful would the mis-detections be, and would any kind of reasonably smart heuristic just ignore them?

If the majority of the matches have a clean cut that also matches a common aspect ratio, then some that are off, especially if they cut the width, should be easily discarded.

[mattkhan]

the log output from MC in this case is

Code: [Select]

0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x0x1920x1080, with 47 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x142x1328x782, with 2 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x142x1328x718, with 2 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x140x1896x940, with 3 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x140x1920x940, with 1031 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x142x1456x898, with 6 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x140x1852x940, with 3 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x140x1920x828, with 3 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x140x1696x940, with 3 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x140x1676x940, with 10 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x140x1640x940, with 3 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x140x1644x940, with 7 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 0x140x1652x940, with 15 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 2x140x1110x736, with 4 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 2x142x1182x810, with 2 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 2x204x1902x940, with 11 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 2x140x1902x940, with 3 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 2x140x1906x940, with 3 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 2x140x1858x940, with 2 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 2x140x1674x940, with 5 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 2x140x1682x940, with 6 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 2x140x1730x940, with 10 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 2x140x1650x940, with 2 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 8x140x1920x940, with 16 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 408x140x1920x940, with 2 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 28x140x1920x940, with 14 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 412x140x1920x940, with 15 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 292x140x1920x940, with 3 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 296x140x1920x940, with 13 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 300x140x1920x940, with 2 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 436x140x1920x940, with 2 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 200x244x1920x940, with 8 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 456x140x1920x940, with 3 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 332x140x1920x940, with 10 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 332x244x1920x940, with 2 hits
0018882: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 460x140x1920x940, with 4 hits
0018883: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 728x140x1920x940, with 2 hits
0018883: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 112x140x1920x940, with 15 hits
0018883: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 116x142x1920x938, with 2 hits
0018883: 11788: Playback: CJRVideoAnalyzeExtended::BlackBarDetection: Crop Window: 252x140x1920x940, with 2 hits

I notice you set limit=0.1 so the black scene problem may affect this analysis, still the biggest no of hits is 0x140x1920x940 which looks correct but still not selected in favour of the no crop option with 47 hits

is that output sorted btw? I notice the top line is the one it picked

0x0x1920x1080, with 47 hits

the logs show that the period analysed around the 29min and 46min mark were problematic, both decoding errors and it require skip=16 to get valid output from the former and skip=22 for the latter

it feels like the best solution is to have a filter that detects for decoding errors if that is possible, i.e. a skip=auto feature

[mattkhan]

my current feeling is that skip is the primary source of errors, in the absence of an auto option would it make sense to do something like run the analysis with a few different skip values and check the output is the same (for the same frames)? I suppose it would make the analysis x times slower but I think accuracy is preferred to speed in this case.

later I'll run a script against a load of films and see what it produces with different skip values

[Hendrik]

Its not sorted right now, but I am sorting and filtering it for the upcoming changes. I can also increase the skip even further, and maybe investigate if there is options i can set to reduce the output of unfinished frames. Detecting if a frame is unfinished might not be possible, but I'll also check on that.

In this result, even without changes, the heuristic should pick the 1031 hit one with the proper dimensions, at least how I imagine it working. 47 hits for the full frame is just such a low percentage that it would be ignored.

[mattkhan]

I wrote a little script https://github.com/3ll3d00d/jrmc-utils/blob/master/cropdetect.py which ploughs through my library, runs ffmpeg with a few difference parameters and dumps the result to some CSVs

I also wrote another script to summarise all of those into one csv https://github.com/3ll3d00d/jrmc-utils/blob/master/analyse_crop.py which compares vs the current mc video crop

the output of this one is like, I'm just printing the most popular single crop from ffmpeg here

Code: [Select]

name,limit,skip,ffmpeg_crop,mc_crop
The Last Picture Show D1,18,24,0x0x3840x2160,0x0x3840x2160
The Last Picture Show D1,25,24,0x0x3840x2160,0x0x3840x2160
The Last Picture Show D1,18,8,0x0x3840x2160,0x0x3840x2160
The Last Picture Show D1,25,8,0x0x3840x2160,0x0x3840x2160
The Last Picture Show Original,18,24,0x0x3840x2160,0x42x3840x2118
The Last Picture Show Original,25,24,0x0x3840x2160,0x42x3840x2118
The Last Picture Show Original,18,8,0x0x3840x2160,0x42x3840x2118
The Last Picture Show Original,25,8,0x0x3840x2160,0x42x3840x2118
Your Name.,18,24,0x4x1920x1076,0x0x1920x1080
Your Name.,25,24,0x4x1920x1076,0x0x1920x1080
Your Name.,18,8,0x4x1920x1076,0x0x1920x1080
Your Name.,25,8,0x4x1920x1076,0x0x1920x1080
Malcolm X,18,24,0x4x1920x1076,0x0x1920x1080
Malcolm X,25,24,0x4x1920x1076,0x0x1920x1080
Malcolm X,18,8,0x4x1920x1076,0x0x1920x1080
Malcolm X,25,8,0x4x1920x1076,0x0x1920x1080
A Star Is Born,18,24,0x164x1920x916,0x0x1920x1080
A Star Is Born,25,24,0x164x1920x916,0x0x1920x1080
A Star Is Born,18,8,0x4x1920x1076,0x0x1920x1080
A Star Is Born,25,8,0x4x1920x1076,0x0x1920x1080

I was thinking of then doing some sort of analysis on this like

* count how many agree then ignore those
* calculate the delta between the two crops, count how many have a "trivial" pixel difference then ignore those
* count how many are insensitive to limit/skip values
* try to scrape AR from something like blu-ray.com (though I don't think they have an API so that could get annoying), assess MC and ffmpeg vs that
* count how many, if any, have obviously wrong values
* if MC behaviour changes, assess how many videos were impacted by the change
* some other stuff I haven't thought of

I thought that this might provide a way to systematically compare MC's behaviour as it evolves across a large no of videos which might help arrive at an optimal solution more quickly

I can post the results once it completes (and if you want the raw data then I can share that also)

[mattkhan]

ran that for ~600 films (ignored DVD), lots of differences but it's clear there are many trivial differences so I need to filter those out first

[mattkhan]

hard to say if this is useful but I ran the analysis anyway on ~600 films

if I allow a tolerance of +/- 16 pixels on either dimension then I'm left with 74 BD or UHD with a significantly different crop to that produced by MC atm
if I reduce this tolerance to +/- 8 pixels then this number goes up dramatically (to ~250)
if I increase it (even to 32), the number of differences barely shifts so I think those 74 films are the material differences

of those 74, the ffmpeg crop is sensitive to

limit: 29
skip: 33
both: 13
neither: 25

i.e.

25 are insensitive to the values I chose for limit/skip
13 are sensitive to both (i.e. I get 4 different crops)
33 are sensitive to skip only (so I guess these are the ones with decoding errors)
29 are sensitive to the black level limit

reminds me that I need to further filter the differences within ffmpeg to check whether the different crops produces are trivial differences, probably this reduces the sensitivity somewhat (though not sure to what extent)

NB: there is at least 1 item (2001 4k) in which MC's current analysis is currently right and ffmpeg is not, not obvious why that is so probably need to investigate that one further to look at what ffmpeg was doing

[mattkhan]

realised I set limit incorrectly, have to rerun the analysis 

[Hendrik]

Setting limit to a number > 1 requires it to be bitdepth aware (<1 is a percentage). Which I suppose is what you figured out?

[mattkhan]

Yes exactly, surprised it actually a) accepts such values and b) produces generally sane results but which still can differ just by varying those values (from 1 absurd value to another)

[mattkhan]

corrected my use of limit & reran same analysis, now with 16px tolerance

461 are ok
129 are not

sensitivity counts

limit: 34
skip: 68
both: 18
neither: 45

go down to 8px and you get

317 are ok
273 are not

go up to 32px and it barely changes from 16px

need to spend a bit of time looking at specific differences to see if there is a recommendation to make

[Hendrik]

Note that we use "round=4" as a parameter (used to be even round=2 before .25, but i bumped it up one notch to reduce some variance), while the default is round=16, which is likely why you are seeing many small-pixel differences, as it rounds the rectangle differently (and also why 16px difference is the sweet spot, as the difference should in many cases be at most 12px)

[mattkhan]

ok makes sense

looking at my stats, going up to skip=24 looks like a positive move though not perfect

out of ~600 films, I'm left with ~140 that don't match (once some tolerance is allowed)

of those, there are ~70 films which produce a different result with skip=24 vs skip=8 and which is different to MC (by more than 0.01 in AR terms)

of those 70, I think most (~60) look more correct with skip=24 though there are <10 which have questionable results (would need to look at them manually to know what is going on I think)

there are also ~14 where changing black levels makes a major difference, there are definitely a few cases where this is correct but it's wrong more often than not and when it's wrong, it's way way wrong (like producing an AR of 4.1). I would think this could be completely automated (run it with both, see if it produces a different but still "normal" AR, if so it's likely to be correct).

what parameters are you using for the next build?

[Hendrik]

Parameters are likely not changing much, but rather adding filtering and heuristics on top of the results, rather then taking every single result rectangle into account.

This should definitely fix issues where skip is too low, because the frames at the beginning only make up a tiny percentage that would just be ignored, even if it slips by the skip parameter.
Not sure about level being too high. I would think if level is too high, it would only show up in some very dark scenes and overall fix itself if some other scene is brighter, but alas. I'll have to dig into some samples of that being a problem.

[mattkhan]

my examples of black level making a difference attached (in case you have any of those films), 18 & 25 refer to the black level (18/255 and 25/255 for actual limit values)

[Hendrik]

The next build will have a simple heuristic and a report with a scoring system (1-5).

Example report:

Code: [Select]

Score: 5
Result: 0x278x3840x1882 (2.39)

0x278x3840x1882 (2.39), 89% (1109 frames sampled), first seen at 00:14:14
0x278x3512x1882 (2.19), 1% (16 frames sampled), first seen at 00:42:48
2x278x2910x1882 (1.81), 1% (10 frames sampled), first seen at 00:42:48
138x278x3678x1882 (2.21), 1% (10 frames sampled), first seen at 00:57:00

Hits <1% have been skipped due to a high accuracy score.

It takes into account the percentages of the various detection, as well as the aspect ratio to make a determination. Lets see how the result of that is going to be. 5 and 4 should be relatively safe detections, unless something fundamentally is going wrong, any lower might warrant some investigation. At least thats the idea. Lets see if thats how it works on larger samples!

I thought about also looking at the values in the rects, but didn't do it yet. For example, pillar boxing is relatively rare, so if thats detected it might be weird, or equal bars on both sides are better matches then uneven ones. I'll add those if needed, but it seemed to just make everything overly complicated when most videos might just fall within these heuristics already.

For easier smartlist to detect things, I could add an expression function to get the score directly, if that seems helpful.

[Hendrik]

Testing my library, so far the only low scores i'm getting are from badly defined borders in broadcast. So you have stuff like the dump below. I might add some more logic to try to unify rects that are very close together like this again, without just blindly unifying all rects like the old code did.
Going to be a bit until my entire library ran again. I should probably only do movies, TV shows rarely come with black bars. But anyway its going now!

Code: [Select]

Score: 3
Result: 4x4x1920x1080 (1.78)

4x4x1920x1080 (1.78), 60% (889 samples), first at 00:00:45
2x4x1918x1080 (1.78), 16% (236 samples), first at 00:03:44
2x0x1918x1080 (1.77), 13% (184 samples), first at 00:11:12
0x4x1920x1080 (1.78), 11% (163 samples), first at 00:18:40

Hits <1% have been skipped due to a high accuracy score.

I was using round=4 to minimize those, but clearly they are still showing up. Although it does make me wonder why there is 2px differences between them... (edit: its applying the rounding to the width, not the crop itself. Makes sense. So you can have 0 -> 1920 and 2 -> 1918, but not 2 -> 1920).

[mattkhan]

Great, I will redo my library tonight

[Hendrik]

After some more progress analysing all my library, I'm seeing a lot more cases with low scores. Luckily for most of them the heuristic seems to still pick the right one, but the detections are all over the place. Unclear if those are mostly very dark videos, where the limit is catching stuff too aggressively, or something else. I'll be investigating some of those tomorrow.

On a quick glance adjusting the limit helps with those, but its hard to track if other files stop detecting their black bars if I lower it. Hard to find a compromise here. Running it twice with different values has some problems with the way we track values, but 25 seems pretty high, as shadowy scenes easily detect with that. Even going to 22 helps a lot already.

How common are dirty/elevated black bars, anyway? Did you have any where lower black level was a problem?

[mattkhan]

I think this is a significant step forwards, well done!

I now see just 28 tracks that disagree with skip=24,limit=25/255

of those:

* VAR is as big a pain here as it is in football

https://www.blu-ray.com/movies/Aquaman-Blu-ray/227124/ is a good example, my analysis happens to look at the 1st hour, yours is more comprehensive which means you detect the 2nd half of the film being IMAX whereas mine sees the scope part

https://www.reddit.com/r/movies/comments/21jqpc/hunger_games_catching_fire_bluray_slowly_changes/ is another one
https://www.blu-ray.com/movies/TRON-Legacy-Blu-ray/18434/
https://www.blu-ray.com/movies/Top-Gun-Maverick-4K-Blu-ray/317486/

would be nice to add a VAR flag and hence allow for a profile driven choice on how to react

[mattkhan]

here's one why I don't understand why it picked the wrong choice, it does say the score is low but not sure why

https://www.blu-ray.com/movies/American-Graffiti-Blu-ray/22698/

Code: [Select]

Score: 1
Result: 0x0x1920x1080 (1.78)

0x136x1920x948 (2.36), 46% (538 samples), first at 00:00:46
0x0x1920x1080 (1.78), 4% (48 samples), first at 00:14:04
2x136x1186x948 (1.46), 2% (22 samples), first at 00:42:10
674x432x1306x948 (1.22), 2% (21 samples), first at 01:24:24
0x136x1156x948 (1.42), 1% (17 samples), first at 00:42:16
672x432x1308x948 (1.23), 1% (16 samples), first at 01:24:25
0x136x1836x948 (2.26), 1% (15 samples), first at 00:28:11
456x304x1920x948 (2.27), 1% (15 samples), first at 00:28:08
0x136x1192x948 (1.47), 1% (15 samples), first at 00:42:13
610x428x1298x948 (1.32), 1% (15 samples), first at 01:24:19
2x136x1182x948 (1.45), 1% (15 samples), first at 00:42:11
672x436x1304x948 (1.23), 1% (15 samples), first at 01:24:23
2x196x1306x948 (1.73), 1% (14 samples), first at 00:28:09
600x424x1304x948 (1.34), 1% (14 samples), first at 01:24:21
534x136x1830x948 (1.6), 1% (13 samples), first at 00:56:15
0x196x1312x948 (1.74), 1% (13 samples), first at 00:28:10
604x424x1304x948 (1.34), 1% (13 samples), first at 01:24:21
444x300x1920x948 (2.28), 1% (12 samples), first at 00:28:07
464x304x1920x948 (2.26), 1% (12 samples), first at 00:28:09
610x424x1302x948 (1.32), 1% (11 samples), first at 01:24:20
560x136x1868x948 (1.61), 1% (11 samples), first at 00:56:13
0x136x1624x948 (2), 1% (9 samples), first at 00:42:15
412x304x1920x948 (2.34), 1% (9 samples), first at 00:28:08
616x420x1304x948 (1.3), 1% (9 samples), first at 01:24:23
610x428x1302x948 (1.33), 1% (9 samples), first at 01:24:22
2x136x1162x948 (1.43), 1% (8 samples), first at 00:42:10
672x436x1308x948 (1.24), 1% (8 samples), first at 01:24:25
2x136x1838x948 (2.26), 1% (8 samples), first at 00:28:12
252x136x964x948 (0.88), 1% (8 samples), first at 00:42:16
616x424x1304x948 (1.31), 1% (7 samples), first at 01:24:23
2x136x1834x948 (2.26), 1% (7 samples), first at 00:28:11
2x136x1174x948 (1.44), 1% (7 samples), first at 00:42:11
606x424x1302x948 (1.33), 1% (6 samples), first at 01:24:20
544x134x1840x946 (1.6), 1% (6 samples), first at 00:56:14
102x136x1190x948 (1.34), 1% (6 samples), first at 00:42:14
102x136x1182x948 (1.33), 1% (6 samples), first at 00:42:14
384x136x1920x948 (1.89), 1% (6 samples), first at 00:56:17
2x136x1178x948 (1.45), 0% (5 samples), first at 00:42:13
552x136x1920x948 (1.68), 0% (5 samples), first at 00:56:17
608x424x1300x948 (1.32), 0% (5 samples), first at 01:24:20
0x196x1304x948 (1.73), 0% (5 samples), first at 00:28:09
528x136x1836x948 (1.61), 0% (5 samples), first at 00:56:16
608x424x1304x948 (1.33), 0% (5 samples), first at 01:24:21
350x136x1186x948 (1.03), 0% (5 samples), first at 00:42:10
240x136x1096x948 (1.05), 0% (5 samples), first at 00:42:15
0x136x1840x948 (2.27), 0% (5 samples), first at 00:28:12
572x136x1920x948 (1.66), 0% (4 samples), first at 00:56:20
672x440x1304x948 (1.24), 0% (4 samples), first at 01:24:23
452x304x1920x948 (2.28), 0% (4 samples), first at 00:28:07
142x136x1834x948 (2.08), 0% (4 samples), first at 00:56:19
560x134x1872x946 (1.62), 0% (4 samples), first at 00:56:13
612x424x1304x948 (1.32), 0% (4 samples), first at 01:24:22
354x136x1186x948 (1.02), 0% (4 samples), first at 00:42:10
254x136x978x948 (0.89), 0% (4 samples), first at 00:42:17
606x428x1298x948 (1.33), 0% (4 samples), first at 01:24:19
0x136x1112x948 (1.37), 0% (4 samples), first at 00:42:10
676x436x1304x948 (1.23), 0% (3 samples), first at 01:24:24
550x136x1858x948 (1.61), 0% (3 samples), first at 00:56:16
168x136x1920x948 (2.16), 0% (3 samples), first at 00:14:07
560x134x1868x946 (1.61), 0% (3 samples), first at 00:56:13
548x134x1840x946 (1.59), 0% (3 samples), first at 00:56:14
544x134x1844x946 (1.6), 0% (3 samples), first at 00:56:14
0x196x1920x948 (2.55), 0% (3 samples), first at 00:28:09
564x136x1868x948 (1.61), 0% (3 samples), first at 00:56:13
610x424x1298x948 (1.31), 0% (3 samples), first at 01:24:20
394x136x1918x948 (1.88), 0% (3 samples), first at 00:56:18
394x134x1706x798 (1.98), 0% (3 samples), first at 00:56:17
0x196x1308x948 (1.74), 0% (3 samples), first at 00:28:10
576x136x1920x948 (1.66), 0% (3 samples), first at 00:56:20
12x136x1920x948 (2.35), 0% (3 samples), first at 00:14:08
2x136x1766x948 (2.17), 0% (3 samples), first at 00:28:11
2x136x1798x948 (2.21), 0% (3 samples), first at 00:28:12
380x136x1920x948 (1.9), 0% (3 samples), first at 00:56:17
448x136x1920x948 (1.81), 0% (2 samples), first at 00:56:18
476x304x1920x948 (2.24), 0% (2 samples), first at 00:28:07
320x136x1096x948 (0.96), 0% (2 samples), first at 00:42:15
562x134x1870x946 (1.61), 0% (2 samples), first at 00:56:13
550x136x1834x948 (1.58), 0% (2 samples), first at 00:56:15
296x136x1096x948 (0.99), 0% (2 samples), first at 00:42:15
548x136x1860x948 (1.62), 0% (2 samples), first at 00:56:16
548x136x1836x948 (1.59), 0% (2 samples), first at 00:56:15
674x436x1306x948 (1.23), 0% (2 samples), first at 01:24:24
420x136x1920x948 (1.85), 0% (2 samples), first at 00:56:18
670x436x1306x948 (1.24), 0% (2 samples), first at 01:24:25
542x134x1846x946 (1.61), 0% (2 samples), first at 00:56:14
32x136x1172x948 (1.4), 0% (2 samples), first at 00:42:12
532x134x1824x946 (1.59), 0% (2 samples), first at 00:56:15
4x136x1920x948 (2.36), 0% (2 samples), first at 00:14:08
612x428x1300x948 (1.32), 0% (2 samples), first at 01:24:19
612x424x1300x948 (1.31), 0% (2 samples), first at 01:24:19
2x196x1314x948 (1.74), 0% (2 samples), first at 00:28:11
360x136x1164x948 (0.99), 0% (2 samples), first at 00:42:17
360x136x1168x948 (1), 0% (2 samples), first at 00:42:17
360x136x1172x948 (1), 0% (2 samples), first at 00:42:17
2x136x1794x948 (2.21), 0% (2 samples), first at 00:28:12
254x136x974x948 (0.89), 0% (2 samples), first at 00:42:17
0x136x1176x948 (1.45), 0% (2 samples), first at 00:42:12
244x136x1096x948 (1.05), 0% (2 samples), first at 00:42:15
0x136x1168x948 (1.44), 0% (2 samples), first at 00:42:11
460x304x1920x948 (2.27), 0% (2 samples), first at 00:28:07


[mattkhan]

another minor problem area is multiple streams and it playing the wrong one

https://www.blu-ray.com/movies/Harry-Potter-and-the-Deathly-Hallows-Part-2-Blu-ray/156262/

is an example

it picks some sort of interactive movie here whereas the film itself is in a different title, if I select that title as the one to play and then run black bars again, it still uses the default track. Shouldn't it respect my manual override in this case?