Topic: "Swag of Tools" : Collection of Utilities for MC (Read 118777 times)

lello · « **Reply #550 on:** August 24, 2021, 05:11:08 am »

Thanks

zybex · « **Reply #551 on:** August 24, 2021, 05:32:21 am »

Try fixing the apostrophe in the "Miglia all'ora" name, it's possible that you have an invalid character there. It's producing an invalid UTF8 encoding, either from MC, SOT, or the filename itself (looks like it's both the name and filename).

Some possible explanation involving Unicode apostrophes, UFT8, PostgreSQL, copy/pastes and some latin

https://stackoverflow.com/questions/60940414/apostrophe-in-postgresql-query-comes-out-32-bit-c3-a2-c2-80

jmone · « **Reply #552 on:** August 24, 2021, 06:19:39 am »

....also what is name of the MKV file on the HDD? Is it exactly the same as that in the string starting with "<Field Name="Filename">......"? Also, what is the string for the side car ".txt" file that is created, is it the same as the MKV?

What SOT does is it uses the "filename" returned from MC and passes that string to MediaInfo / ffmpeg to then read the file and then create the TXT file.

jmone · « **Reply #553 on:** August 24, 2021, 05:59:34 pm »

Here is a quick and dirty test version where I've changed how I handle reading UTF-8. Let me know if this is any better?

lello · « **Reply #554 on:** August 25, 2021, 02:51:41 am »

Everything works perfectly! Again thanks for your willingness to solve even small problems

jmone · « **Reply #555 on:** August 25, 2021, 03:56:53 am »

Great! I'll clean it up and post a Beta Version for wider testing (in case the change breaks some other part of SOT).

jmone · « **Reply #556 on:** August 25, 2021, 04:21:55 am »

Beta Version: 210825
- Added the ability to enter in the desired point in the video to start measuring the Crop Factor
- Simple check (and warning in the log) if the desired point in the video to start measuring the Crop Factor is greater than the duration of the video itself (and SOT will instead measure from 1 sec so processing over lots of files can continue)
- Your preferred value will be remembered between runs (stored in the INI file)
- If the value is blank or 0 then the Crop Factor will not run (so as to not slow down those who just want the Media Info meta data but don't want to check for the crop info)
- Changed the Language parsing to remove duplicates, padding + it will not call MCWS if there is nothing to change
- Changed how UTF-8 Strings from MCWS calls are processed

As usual, let me know what I broke! If all looks OK, I'll update GitHub.

lello · « **Reply #557 on:** August 26, 2021, 12:11:34 am »

Everything works perfect ..... too bad

jmone · « **Reply #558 on:** August 26, 2021, 12:14:07 am »

Nice! Always surprises me (given I don't really know what I'm doing)

lello · « **Reply #559 on:** October 23, 2021, 11:31:02 am »

Excuse me jmone, when you have time could you take a look at the attached log file? An error message has been appearing for some time but it is not clear to me why.

Thanks

jmone · « **Reply #560 on:** October 23, 2021, 04:30:09 pm »

According to the log:
- "<Response Status="Failure" Information="No changes."/>" is that the same values are already in the MC Database and hence did not try to update them (eg it looks like it's already been processed)
- "<Response Status="Failure" Information="Field not found."/>" is that SOT could not find the Field called "Codec Id"

I take it however that is not what you expect? Can you also post a screen shot of your SOT GUI with the values you are using.

lello · « **Reply #561 on:** October 24, 2021, 10:16:30 am »

The screenshot

https://ibb.co/x19w8Xz

the values:

-Audio Track Count RAW;Audio Language;Audio Tracks RAW;Extras;Subtitles;Video Codec;HDR Format RAW;Aspect Ratio;Codec Id
-ID;Language;Format;Title;Language;Format;HDR format;Display aspect ratio;Codec ID
-Audio;Audio;Audio;Audio;Text;Video;Video;Video;Video

jmone · « **Reply #562 on:** October 24, 2021, 03:59:28 pm »

Thanks - I'll use these to test. What the particular issue, is the "could not find the Field" and/or "No changes" needed error?

lello · « **Reply #563 on:** October 24, 2021, 04:07:23 pm »

"could not find the Field"

jmone · « **Reply #564 on:** October 24, 2021, 04:09:26 pm »

Right, can you post a screen shot of the Field's Details (Manage Library Fields) in MC so I can set it up the same way?

lello · « **Reply #565 on:** October 24, 2021, 04:15:06 pm »

I don't have the htpc near now, so tomorrow.

But do you want a screenshot of all the fields?

jmone · « **Reply #566 on:** October 24, 2021, 04:16:04 pm »

Nope, just the one causing the issue.

lepa · « **Reply #567 on:** October 25, 2021, 03:58:26 am »

I don't think there is nothing wrong in SoT. Probably lello just don't have user field "Codec Id" created.

@lello either remove last items from each string ";Codec Id", ";Codec ID" and "Video"
or add Codec Id field in MC (see picture)

lello · « **Reply #568 on:** October 25, 2021, 07:36:03 am »

In fact, the Codec Id field is not there: is it important?

I thought the problem was the Video Codec field, but I was wrong

lepa · « **Reply #569 on:** October 25, 2021, 09:53:22 am »

I can't remember. If everything looks ok to you in the theaterview then it is probably not used currently

jmone · « **Reply #570 on:** October 25, 2021, 03:47:45 pm »

The "Video Codec ID" must be a user created field as it is not part of the STD MC Library fields (and hence will not be used by MC directly). I guess someone wanted it for reference.

BY96 · « **Reply #571 on:** October 29, 2021, 03:35:10 pm »

Hello!

I am very new to SOT, having just come upon it. I am mainly wanting to use it to collect metadata. I have one question (so far): MC already seems to populate the audio bit depth field. However, I wanted to extra the video bit depth too. So, I created a new MC field for "Bit Depth (video)," which worked. However, I then realized that the MediaInfo file that is used, there is Bit Depth for both audio and video. Is there anyway to distinguish which one I want to populate into my new "Bit Depth (video)" field? When I do it, both the audio and video results populate in that field.

Thank you for your help and for such a great tool!!

jmone · « **Reply #572 on:** October 29, 2021, 03:58:36 pm »

Yup - in the "Enter MediaInfo Section to restrict to" put in "Video"

BY96 · « **Reply #573 on:** October 29, 2021, 06:43:56 pm »

That's great! Thanks!

Do I need to put "video" for each field, or if I just enter it once, it will cover all new fields?

Edit - Just answered my own question. Since I was adding five new fields (all for video data), I added "video" five times and it worked.

But new question... In MC, can custom fields be categorized under Audio, Video, etc., or do they all just fall under Advanced?

Thanks!!

jmone · « **Reply #574 on:** October 29, 2021, 08:06:55 pm »

Great! Glad you worked it out.

You can modify what tags show up for Audio, Image, Video etc (see pic).

I don't really use the Tag Window so can't say if you can move your fields to another part other than advanced. I do all my tag editing in the main windows with the columns added for the fields I want (and remove the one I don't want to see).

BY96 · « **Reply #575 on:** October 29, 2021, 09:11:28 pm »

Thank you!

Would you mind putting up a screen shot of where you edit tags. I seem to struggle figuring out all the different windows and views in JRiver!

Thanks!

EnglishTiger · « **Reply #576 on:** October 29, 2021, 10:22:53 pm »

Quote from: BY96 on October 29, 2021, 06:43:56 pm

But new question... In MC, can custom fields be categorized under Audio, Video, etc., or do they all just fall under Advanced?

Thanks!!

Yes if you use the Tag Windows Customize Option you can add any Tag/Field that appears in the Advanced Section to any other Section, which removes it from the Advanced Section.

In some respects the Advanced Section is misnamed as any Tag/Field that is not in Any Other Section automatically gets added to the Advanced Section.

jmone · « **Reply #577 on:** October 29, 2021, 10:28:43 pm »

If you are in Std View you can edit any "cell" or group of cells by highlighting them and pressing F2 or a slow 2nd click on the item(s) you want to change (double click will play the item). You can also add and remove visible cols.

BY96 · « **Reply #578 on:** October 29, 2021, 11:36:19 pm »

Quote from: EnglishTiger on October 29, 2021, 10:22:53 pm

Yes if you use the Tag Windows Customize Option you can add any Tag/Field that appears in the Advanced Section to any other Section, which removes it from the Advanced Section.

In some respects the Advanced Section is misnamed as any Tag/Field that is not in Any Other Section automatically gets added to the Advanced Section.

Wow! Thank you! That is so helpful!! I never realized I could customize the tags so much. Thanks again!

lepa · « **Reply #579 on:** October 31, 2021, 09:37:39 am »

Hi jmone,
Would it be possible for you to make SoT to work memory only if one doesn't want to save mediainfo log into disk. I think it would speed up processing large collections considerably when you don't have to create and delete file for every item. Also saves SSD cycles if writing is done to SSD

The other I noticed that SoT doesn't overwrite old data if new field value is empty. I was replacing some old DVDs with bluray version while keeping the same DB entry. DVD had subtitles and Bluray didn't so when I did SoT Media Info parsing to get new data it kept old data (subtitles) even though I now don't have those with the BD.

jmone · « **Reply #580 on:** October 31, 2021, 10:34:53 pm »

01 Nov 2021 Beta: I made a bunch of changes around what Lepa suggest but it will need testing due to logic changes to make sure I did not stuff anything up (and you know that is very possible!)
- Memory Buffer for Media Info: SOT will use Memory instead of Disk to pipe the information from Media Info (note: You may see SOT saying "Not Responding" during longer Media Info scans but it is fine). I'm not sure this is any "faster" that the old method and SOT will still save this information to a file unless you have "Don't save MediaInfo output as an Extra File" Checked, but it will save you some SSD writes.
- "Use existing MediaInfo Extra File if it already exists": This is the fastest and is great if you want to go back and add new fields to MC as it does not run MediaInfo again, just pulls data out of the existing "Extra" file that was saved on a previous run
- Updated Log Messages: to reflect these changes
- Updates MC with a NULL Value: SOT will now make a MC Field Blank (instead of skipping it) if a specified MediaInfo Data Field is blank.

TBA
- I've not yet changed the use of a tmp file for the Measuring of the Crop Factor but if all goes well with the above, I'll make that change as well

lepa · « **Reply #581 on:** November 04, 2021, 10:16:58 am »

Thanks for the new version. Haven't done any specific testing but just using it everything seems to work

jmone · « **Reply #582 on:** November 05, 2021, 12:29:37 am »

Well that's odd! I always break something

Do you want the ffmpeg Crop Factor detection to also use Memory instead of a Temp File?

lepa · « **Reply #583 on:** November 06, 2021, 02:31:42 pm »

That would be nice

jmone · « **Reply #584 on:** January 02, 2022, 02:17:30 pm »

FYI - Lello and I have been discussing Dolby Vision detection. SOT uses MediaInfo that from what I can tell, only detects DV in MKV (and I think MP4/MOV) but not from a Blu-ray disc or even the M2TS files (though makeMKV can). I suspect that the work being done by Hendrik to add DV Tonemapping to JRVR means that he has someway of detecting DV though I'm unsure if it is through ffprob (what MC uses for meta data retrieval) or the underlying libraries used for JRVR. Anyway, with luck the type of HDR (HDR, HDR10, HDR10+, DV etc) may be able to be read and stored in a MC Library Field.

jmone · « **Reply #585 on:** January 03, 2022, 11:48:14 pm »

Quote from: lello on January 03, 2022, 11:28:23 am

It seems to me that mediainfo, in addition to not detecting DV, cannot detect all subtitles and audio languages either.

Also, the AR-calculated sometimes works and sometimes it doesn't with the same film.

I know that mediainfo has some issues with chapters over compound Blu Rays (had to write my own parser for that one in SOT), but I've not seen the issues with Subs and Audio missing (unless it is the same issue with compound BDs ?) Do you have an example or two?

The calculated AR is actually done by ffmpeg not mediainfo so it could be another issue, even a SOT problem. Is there any pattern to it? ...and when it does not work what is the error etc or does it report back nothing???

Anyone else seeing any issues?

Thanks
Nathan

lello · « **Reply #586 on:** January 04, 2022, 01:37:54 am »

I found the problem with the BD UHD of the Lord of the Rings trilogy (the only one I have with DV) and in all of them it gives me only English audio and no subtitles.

With Tenet's BD UHD, however, it detects all audio and subtitles for me.

As for the calculated AR, on a first pass it gave me 1.78 for one disc and 2.39 for the other while on the following passes it gave me 1.78 for both.

jmone · « **Reply #587 on:** January 04, 2022, 02:19:50 pm »

Thanks - I've had a look at LOTR and I too only get English Audio and No Subs being reported. I've seen MediaInfo miss pick what MPLS to scan before and hence under report the meta data.... so I also have SOT scan the main M2TS as well which normally works in such cases but not for LOTR. It seems that the files I scan are just reporting an ID but not a Language for these tracks.

Here is a test build that additionally scans all the MPLS files in the Playlist directory (instead of the largest M2TS) and this should pick up everything..... but it will be slower on some discs as there is now more to scan. The good news it is not just LOTR that was missing details but other BD's as well and this test version picks up the extra meta data on these discs as well.

Let me know what you think.

Thanks
Nathan

PS - I've not looked at the AR part

lello · « **Reply #588 on:** January 05, 2022, 01:41:51 am »

Downloaded the new version. Audio and subtitles all detected correctly. The calculated AR instead is now empty

jmone · « **Reply #589 on:** January 05, 2022, 03:02:33 am »

Woops - fixed one thing.... broke another. Try this one.

lello · « **Reply #590 on:** January 05, 2022, 04:00:19 am »

Tried. With The Two Towers measured 2.39 while with The Fellowship of the Ring 1.78

arcspin · « **Reply #591 on:** January 05, 2022, 04:14:08 am »

I have had strange aspect ratios from time to time, I solved those issues by setting 400 sec into video to measure crop factor.

jmone · « **Reply #592 on:** January 05, 2022, 05:03:57 am »

Crop detect is hit and miss as it tries to measure what the black parts of the image is, and plenty of movies like LOTR have a lot of black frames. I've tried lots of ffmpeg crop detect parameters and if there are any suggestions, I'm all ears! eg:
- Current > -t 10 -vf cropdetect -f
- Also tried stuff like > -t 10 -vf cropdetect=0.1:16:2 -f

Thanks
Nathan

lello · « **Reply #593 on:** January 05, 2022, 05:32:34 am »

Quote from: arcspin on January 05, 2022, 04:14:08 am

I have had strange aspect ratios from time to time, I solved those issues by setting 400 sec into video to measure crop factor.

I have tried 120, 600 and 800. Later I try 400

lello · « **Reply #594 on:** January 05, 2022, 07:22:47 am »

At 400 ok disc1 and not disc2. At 120 ok disc1

jmone · « **Reply #595 on:** January 05, 2022, 03:21:37 pm »

After more testing, there is not "one" set of ffmpeg cropdetect variables (Limit:Round:Skip) that works on every video, but some combinations are better than others and I've now made them customisable.

Quote

It calculates the necessary cropping parameters and prints the recommended parameters via the logging system. The detected dimensions correspond to the non-black area of the input video.

It accepts the following parameters:

limit - Set higher black value threshold, which can be optionally specified from nothing (0) to everything (255 for 8-bit based formats). An intensity value greater to the set value is considered non-black. It defaults to 24. You can also specify a value between 0.0 and 1.0 which will be scaled depending on the bitdepth of the pixel format.

round -
The value which the width/height should be divisible by. It defaults to 16. The offset is automatically adjusted to center the video. Use 2 to get only even dimensions (needed for 4:2:2 video). 16 is best when encoding to most video codecs.

skip -
Set the number of initial frames for which evaluation is skipped. Default is 2. Range is 0 to INT_MAX.

Default: 24:16:2 (or leaving the Custom Settings blank) has been what SOT has been using
Better?: 0.1:16:2 (I find this more reliable than the default so it is worth a try)

Let me know what you think. The settings below worked on all the LOTR UHD discs for example.
Thanks
Nathan

mattkhan · « **Reply #596 on:** January 05, 2022, 03:50:16 pm »

does ffmpeg take 24 literally for 10bit video? if so, is that why it is more reliable?

I'm slightly surprised black bars are that bright. Not to say it's wrong, just that it seems greyer than I would have thought them to be

jmone · « **Reply #597 on:** January 05, 2022, 04:01:46 pm »

Well that is the default that ffmpeg use. I too would have thought that they would be 16 or 15 but I don't know how the studios encode their letterboxs... and they may not even be consistent. Anyway, using the 0-1 range (eg 0.1) seems to work much better for me, but you can now try any of the values as I've exposed these parameters.

mattkhan · « **Reply #598 on:** January 05, 2022, 04:09:13 pm »

just thinking that if it takes it literally then the default of 24 just won't work for UHD hence using 0.1 is a better default all the time

jmone · « **Reply #599 on:** January 06, 2022, 03:30:12 am »

I'm happy to hard code a default, but for now it is an easy way to test.

INTERACT FORUM

Author Topic: "Swag of Tools" : Collection of Utilities for MC (Read 118777 times)