Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[jmone]

You had the answer - Album had to be set for the top file.   So now I'm pushing down File Type, Media Type, Media Sub Type, and pushing up Album.  I've attached an updated Particlizer - unzip and place it in Scriptlets.  It also now warns of missing fields in the file list.

New Particlizer sort of worked:
- The push down of File Type, Media Type, Media Sub Type is great!
- The push up of Album is also good buy you may need to also push up/down Artist (as to me they are still different groupings),
- Filename was auto changed again by MC....(could be the above?)

[jmone]

...I think this issue will go away if the particles are created from the MediaInfo call as the meta data will be the same

[jmone]

I pushed down "Artist" so that it became one album in MC and the ran Particlizer.  No problems with file renaming.

[jmone]

Strike that - they just renamed (tested a couple of times)

[jmone]

More Testing.  It is just the "Name" Field that causes this file renaming behaviour and it just may be the "order" in when the name is changed, eg

If I change the name of all of the dummy files to "The Rolling Stones Eoms" then run Particlizer it is fine.  I can then change the "Name" once they are particles to anything I want and the filename does not change.

I'm guessing that the particles have to be created first and must have the same "Name".  Post he creation of they can be renamed.

[MrC]

Sorry, how do I tell MC which MPLS to use?  The ripped BD isn't playing (Something went wrong with playback).  Maybe its a copy protection thing.  MC ripped the BD, but it doesn't seem to play.  I know I've used AnyDVD in the past, but that was on a trial basis, so no license here.

Edit: installed AnyDVD and started re-ripping.  Sigh.

Edit: Now we're BD'ing.  Yeah.

[jmone]

Yeah - so what I'm thinking for the workflow once the BD is ripped (   )

1) Use MC's "Auto Create BD/DVD Title Particles" to create an enties in MC with the format of "...The Rolling Stones Eoms\BDMV\PLAYLIST\00000.mpls" instead of "...The Rolling Stones Eoms\BDMV\index.bluray;1"  This is important as the Chapter Information pulled from Media Info is associated with each of the "mpls" not the main Bluray (and some disks have multiple MPLS with different content in the other MPLS though most have a single MPLS for all songs).

2) We then select the MPLS based particle as the source to create the Dummy Files using MediaInfo for each chapter

3) We then update the meta data from one of the scrapers.

Thanks
Nathan

[jmone]

So it looks something like this

[MrC]

So it should look something like this?  Download and play:

    https://dl.dropboxusercontent.com/u/87189402/Particles%20from%20BD.mp4

 

There is a new scriptlet called GetBDChapters.  It is a two stage scriptlet.  The first time you call it, it does the mediainfo lookup on the mpls, creates the dummy files and imports them for you into MC.  For the next stage, you select the mpls and the dummy files, and it calls Particlize to create the chapter particles.  You may have noticed in the movie that the dummy files and mpls file are not sorted in the view.  I made a change to pscriptor to allow scriptlets to sort their inputs, so the order in the view no longer matters.

It also has some code to get all the chapter information for an entire BD rip, but this part is not complete yet.  I started working on that yesterday before I read your desired workflow.  It was going to figure out the MPLS for you including how many chapters are in each so you can know which one to select.

[jmone]

Yes - exactly like that!

[jmone]

It also has some code to get all the chapter information for an entire BD rip, but this part is not complete yet.  I started working on that yesterday before I read your desired workflow.  It was going to figure out the MPLS for you including how many chapters are in each so you can know which one to select.

I'm not sure how well this would work in practice as the authoring on some BD's are just bizarre (such as multiple MPLS for the same content so you would end up with lots of dups, or a big list of entries from "junk" MPLS that are full of copyright, directors comments etc etc).

I'd suggest that this could be a phase 2 to see how well it could be automated.  I'd propose that I'll run the "MC Auto Create BD/DVD Title Particles" (to create MPLS) -->  "MrC GetBDChapters Autoworkflow" (to create Chapter Particles on the MPLS) --> "MrC MetaData Taging" on my collection and report back on issues and any ideas to combine, simplify the workflow.

Thanks for doing this.  I take it the calls to Media Info where fine?

Nathan

[MrC]

Here is a pre-release or pscriptor you can start working with.  It will take a bit to setup.  Read the comments in the MakeBDChapters scriptlet, and those in the Particlize scriptlet (you're probably setup already with this part).

You'll want to copy the config variables on the config file for the MakeBDChapters section into your custom config file, and customize them appropriately.

[MrC]

I just added some more comments to the scriptlet which you'll need:

# An example command line is:
#
#     perl pscriptor.pl -E MakeBDChapters -f "tmp field" --filekey
#
# The field 'tmp field' is used in phase 1 to store the chapter playback range information
# in the mpls file for later retrieval in phase 2.  You can use any field you like, and
# the field must be in the file list.

[jmone]

I must be doing something wrong.

I'm running "perl pscriptor.pl -E makeBDChapters -f dummyfield --filekey" over an MPLS and it run without any errors but I don't see any resultant files created in Windows.  I see in your movie you run a slightly different cmd.

Here is my custom config

Code: [Select]

# Config file for pscriptor.pl

MCWS_server:   localhost:52199
MCWS_username: xxxx
MCWS_password: xxxxx


# MakeBDChapters
MakeBDChapters::Var: exe_path_win = C:\Users\Nathan\Downloads\MediaInfo_CLI\mediainfo.exe # path to mediainfo executable for Windows

# MediaInfo options
GetMediaInfo::Var: exe_path_win = C:\Users\Nathan\Downloads\MediaInfo_CLI\mediainfo.exe # path to mediainfo executable for Windows

[MrC]

Sorry, I changed the name of the scriptlet last minute, but forgot to change the variable names both inside the scriptlet and in the config file.  You'll need the new MakeBDChapters scriptlet and update the variables (one which had a typo) in the config file (copy to your custom config and modify), removing the old GetBDChapter variables.

The update is in the attachment above.

[jmone]

Mmmm same thing (and the vars look the same).  There are two attachments in the post above with the same name but I'm trying the bigger of the two (I think it is the latest as it has a larger file #....)

[MrC]

Sorry. Out right now. Edit the script an configuration, replacing GetBDChapters with MakeBDChapters, and correct the spelling typo in basedir.

[jmone]

Scanned through the config and scriptlet files and they look OK to me, so I'll wait till your back!

[jmone]

...I did also notice that the Scriptlet folder has a ".MakeBDChapters.pm.swp" file

[MrC]

I'm back.  Went out for some Spanish Paella.

I downloaded the large of the two archives from above, and it works fine.  Add the -v -d options to see what's going wrong for you.  I suspect your basedir is not set.  Did you fix my typo in the config file.  This is what you should have:

MakeBDChapters::Var: basedir                    = $DESKTOP              # basedir for dummy files folder ('$DESKTOP' => Desktop folder)

Also, I assume you replaced your entire pscriptor folder with the new one above, and then just moved your custom config file into the new folder.  To make this work, I had to change the main pscriptor code, some MCUtils modules code, and about 1/2 of the scriptlets.

The .swp file is just a temporary file my editor makes while I'm editing - you can toss/ignore it.

[jmone]

Mmm - Good dinner?

Code: [Select]

C:\Users\Nathan\Desktop\pscriptor>perl pscriptor.pl -E makeBDChapters -d -v -f d
ummyfield --filekey
main                : Command Line: -E makeBDChapters -d -v -f dummyfield --file
key
do_mcc_command      : 21000: Sending: http://localhost:52199/MCWS/v1/Control/MCC
?Command=21000

C:\Users\Nathan\Desktop\pscriptor>

Code: [Select]

# MakeBDChapters
MakeBDChapters::Var: exe_path_win = C:\Users\Nathan\Downloads\MediaInfo_CLI\mediainfo.exe # path to mediainfo executable for Windows
MakeBDChapters::Var: basedir = $DESKTOP\DummpFiles # basedir for dummy files folder ('$DESKTOP' => Desktop folder)

# MediaInfo options
GetMediaInfo::Var: exe_path_win = C:\Users\Nathan\Downloads\MediaInfo_CLI\mediainfo.exe # path to mediainfo executable for Windows


[jmone]

So I deleted everthing (but my custom config file pasted above) and added the files that I had downloaded from the bigger file.  Other scripts work.  I'm wondering is it is an issue calling MediaInfo, as in looking at the scripts I was wondering if the path was hard coded (so tried copying Media info to various folders but no change).

Edit - it is actually the seperate discogs that works fine in creating the dummy files.

[MrC]

Typo:

   perl pscriptor.pl -E makeBDChapters -d -v -f dummyfield --filekey

I'll get some code in there error out on this.

[jmone]

woops!  no idea it was case sensitive - stuff being created!

[MrC]

Yup, they are.  Perl finds them case insensitively, but I have to convert the scriptlet's name literally into what are called package variable names, so I can lookup the scriptlet's callbacks.  These *are* case sensitive, so this error slid by the statement that loads the module.  I've added code to warn and die when this happens.

[jmone]

      Success with Part 1 and Part2!  I've been called to Lunch so will test after that but what View are you using to keep the temp files grouped with the main particle (eg nothing is being flowed down at that point?) so you can run Part 2 easilly?  (I just manually tagged the temp files)

Thanks
Nathan

[MrC]

You don't need to group or sort the files anymore.  The MakeBDChapters scriplet sorts its file list for you.  Just select the 1.mp3 - N.mp3 and the mpls.

Yes, dinner was very good.

[jmone]

I'm not sure I understand (what news!).  In you clip - the dummy files are being added in the view you have (not sure what it is), but for me this does not happen (I had created a view previously by Discogs) - see pic.  The files are added to MC just fine but not to the view I launched the script from.  Does that make sence?

[jmone]

Yes, dinner was very good.

I can't remember what lunch just tasted like I bolted it down so quickly! 

[MrC]

See the attached screenshot.  I include files that are in either of those two folders.

[jmone]

First test looks very very good (see pic):
- Particles created properly (no renaming problem),
- Playback range is spot on (and correct number of tracks)
- Pulled the names from discogs just fine using "perl discogs.pl -c discogs-config.txt -i name --mediatype Video --filekey"

The only comment I'd have at this stage on the workflow is around the Stage 1 --> Stage 2 views.  During the first pass of MakeBDChapters scriplet would it be possible to roll down the Meta Data as most people would be using a std view to see their BD and the dummyfiles will not appear.  It is unlikely they would want to create a view as yours for each disk to process.

Off to test some tougher BD!

Well done MrC - I need to send you a beer somehow!

[MrC]

Thanks.

As for the view, there has to be a customized view to include the FileKey, and Stack * fields anyway.  While you can just add those to an existing view, it seems reasonable to just include the Filename (path) rule that also includes the dummy file folder.

I could certainly push more fields downward if they exist.  Which fields should be included?

[MrC]

One of the things that the other mode will eventually do (the one where you select the main index.bluray;1 file) is to not only create the title particles like MC does, but it will also return the number of chapters in these too so that it might help guide towards the correct mpls to use.

Its still going to be a two stage operation, because I have no way to select files for you, and and that's the entire basis for getting the track data into pscriptor.

[jmone]

Here is a second test.  This was is much harder as it is Adele's Live at the Royal Albert Hall and on the BD they had many more tracks than what is listed on Discog as they tried to put much of her between song dialogue as separate chapters to the actual songs.  This is an example of where the "old" method of using track lengths pulled from Disogs failed miserably.  So For this one, I had to play each of the particles to see what each track actually was.

PS - I don't like this album much as it is a conversation for much of the album and she just swears all the way through it.

[jmone]

Thanks.

As for the view, there has to be a customized view to include the FileKey, and Stack * fields anyway.  While you can just add those to an existing view, it seems reasonable to just include the Filename (path) rule that also includes the dummy file folder.

I could certainly push more fields downward if they exist.  Which fields should be included?

The problem is I have 500 BD's rips (+ particles) all in one folder called "Disks" of which some are Music Videos so even if I added the dummy folder it will be a very long list to pick the individual items out for the second pass.  Can you push down the Meta Data in the first pass instead of the 2nd?  Most should be OK to push down at this point with potentially the exception of Media Type to Video (from memory I don't think MC cares if the Media Type is wrong as long as you don't try to play it).

Thanks
Nathan

[jmone]

One of the things that the other mode will eventually do (the one where you select the main index.bluray;1 file) is to not only create the title particles like MC does, but it will also return the number of chapters in these too so that it might help guide towards the correct mpls to use.

Its still going to be a two stage operation, because I have no way to select files for you, and and that's the entire basis for getting the track data into pscriptor.

Understand.  Some of the MPLS are rubbish (and are not presented in MC during normal playback), eg:
* Exile on Main Street has 2: 00000 contains all the songs, 00001 is just for the Menu Screen
* Adele Live has 4: 00000 contains all the songs from the concert, 00002 is an 8min backstage video, 00003 is some backgroud music and still clips of Adele,  00004 seems to be same as 00003.  There is no 00001

I'll keep reporting back on what I find as it may simplify the logic if there is no worthwhile content on anything other than 00000

[jmone]

...also is all this particles stuff polluting your pscriptor thread (eg do you want a new thread)?

[MrC]

Can you push down the Meta Data in the first pass instead of the 2nd?

No, because the files are not yet in MC, and are not yet selected.  The only way would be to get metadata into the files before they are imported, and I don't want to get into tag writing.

Or maybe I can create an MPL and import that.

[MrC]

The problem is I have 500 BD's rips (+ particles) all in one folder called "Disks" of which some are Music Videos so even if I added the dummy folder it will be a very long list to pick the individual items out for the second pass. 

Would it work to use Search to select only the files you are working on?  I just typed "exile" in Search, and that limits the # of files! since I include the album name in the temporary files directory.  That's why I include dead it, by the way.

[jmone]

Yeah that will work for now (Exile cuts it down to about 50 files).  Let me try a couple.

[jmone]

Here is the next one (sorry for the gap - dinner guests!).  Bit more complicated, Pink Funhouse Tour (another disc not worth the money IMO except we where at this concert... the concert was good by the recording is not great).  Wikipedia has the right track info (except one of the specials is in the wrong order), but both Discogs and AllMusic are wrong so I just updated the "Name" by hand.  It is more complicated as:
- 00000.MPLS contains 22 chapters that corresponds to the songs (the 23rd is and odd 1 sec chapter and I'll delete it) which I ran MakeBDChapters over
- 00001&00002 contain 1 bonus track each so no need to run MakeBDChapters
- 00003&00004 contain 1 bonus extra so again no need to run MakeBDChapters
- 00005 contain the "On Tour With Pink" with a heap of chapters but you would not want to run MakeBDChapters over this
- 00006,7,8,9 contain Logo's, Warnings, the Menu etc so you don't want these at all.  In MC these are not even presented as options but I don't know what criteria LAV users to hide these ones as some are 10mins long (eg the Menues).

Overall, working well!

[jmone]

Got one that fails - Amy Winehouse Back to Black BD Pure Audio.  It is authored differently with a separate M2TS file for each of the 11 Songs.  While MC/LAV present them as chapters, Media Info presents then as different Videos (#1 to 11) with only 1 chapter in it's output.

Code: [Select]

General
Complete name                            : \\MAIN\Media\Discs\Amy Winehouse Back To Black\BDMV\PLAYLIST\00000.mpls
Format                                   : Blu-ray Playlist
File size                                : 1.66 KiB
Duration                                 : 34mn 50s
Overall bit rate mode                    : Variable
Overall bit rate                         : 6 bps

Video #1
ID                                       : 4113 (0x1011)
Menu ID                                  : 1 (0x1)
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4.1
Format settings, CABAC                   : Yes
Format settings, GOP                     : N=1
Codec ID                                 : 27
Duration                                 : 3mn 33s
Bit rate mode                            : Variable
Bit rate                                 : 36.6 Mbps
Maximum bit rate                         : 40.0 Mbps
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 24.000 fps
Standard                                 : NTSC
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.735
Stream size                              : 932 MiB (81%)
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
format_identifier                        : HDMV
Source                                   : 00000.m2ts

Video #2
ID                                       : 4113 (0x1011)
Menu ID                                  : 1 (0x1)
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4.1
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 4 frames
Format settings, GOP                     : M=3, N=18
Codec ID                                 : 27
Duration                                 : 4mn 18s
Bit rate mode                            : Constant
Bit rate                                 : 21.4 Mbps
Nominal bit rate                         : 15.0 Mbps
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 24.000 fps
Standard                                 : NTSC
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.430
Stream size                              : 659 MiB (73%)
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
format_identifier                        : HDMV
Source                                   : 00002.m2ts

Video #3
ID                                       : 4113 (0x1011)
Menu ID                                  : 1 (0x1)
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4.1
Format settings, CABAC                   : Yes
Format settings, GOP                     : N=1
Codec ID                                 : 27
Duration                                 : 2mn 32s
Bit rate mode                            : Variable
Bit rate                                 : 36.1 Mbps
Maximum bit rate                         : 40.0 Mbps
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 24.000 fps
Standard                                 : NTSC
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.726
Stream size                              : 656 MiB (81%)
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
format_identifier                        : HDMV
Source                                   : 00003.m2ts

Video #4
ID                                       : 4113 (0x1011)
Menu ID                                  : 1 (0x1)
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4.1
Format settings, CABAC                   : Yes
Format settings, GOP                     : N=1
Codec ID                                 : 27
Duration                                 : 3mn 11s
Bit rate mode                            : Variable
Bit rate                                 : 36.2 Mbps
Maximum bit rate                         : 40.0 Mbps
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 24.000 fps
Standard                                 : NTSC
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.726
Stream size                              : 824 MiB (81%)
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
format_identifier                        : HDMV
Source                                   : 00004.m2ts

....etc....

Menu
00:00:00.000                             : Chapter 1



[jmone]

Another fail.  Sex Pistols - Never Mind the Bollocks.  MC/LAV shows all 30 chapters but Media Info is only pulling the first 16 related to the first M2TS.  The other chapters are in the second M2TS.  This is a limitation of Media Info.

[jmone]

This one is epic weird.  Green Day - Awesome as F**K.  
- 00000/01000.mpls: Background Menu
- 00001.mpls: 1 Chapter for the whole concert
- 00002/01002.mpls: 1 "Bonus Song"
- 00003/01003.mpls: FBI Warning (hidden by MC/LAV)
- 01001.mpls: 19 Chapter for the Concert (this is the one we want)

The really weird bit is MakeBDChapters worked fine in 01001 but, as you can see in the pic, as I started naming them MC started changing the Filename!

I've tried 3 times, and it is the same.

Of course this breaks playback and I have no idea why MC would do this.  Bug?

Edit - I'll leave this one till tomorrow and try again as it stopped change the file name after a certain point on the 3rd test.

[jmone]

So after today's testing I'd summarise it as:
- Basically works well in generating the correct number of Particles and their Playback Range for a given Playlist (mpls) and this alone saves heaps of time and effort
- MC/LAV is more reliable in reporting Chapters for a Playlist than Media Info (in cases where the content is split over multiple M2TS files)
- Meta data from Discorg and AllMedia is generally unreliable (so you need to enter your own names)
- BD Authoring is all over the place, you can not rely on a particular Playlist, you have to play them and see what is the correct one (note: MC/LAV tends to pick the correct one)

Thanks again MrC.

[jmone]

As an idea... Instead of calling Media Info, Hendrik may be able to advise on how to progamatically call LAV for the Chapter information + it is already installed and would be consistent with what we see in MC.

[MrC]

If there is a command line interface to LAV to get this info, I'm happy to use it.

When you get a fail, since I don't have your disc material, I'd need an archive of separate mediainfo --full runs on each mpls, and a recursive directory listing of the BDMV folder.  If you can create that and attach (or email me) the zip file, I'll see what can be done.

It occurred to me last night, and I have not tried this, that you might be able to save an MPL with the FileKey column instead of the Filename column, and MC might be able set your data from those.  If MC can use the FileKey in MPL, then you might have a way to save your particle data.

[jmone]

I guess Hendrik can advise on LAV and I assume there is a cli (but ....?).

I'll package up a few later today and send them though, but it was more looking for any common patterns / use cases.  The hard bit is going to be finding how to get mediainfo to report all the chapter on those authored with one MPLS for multiple M2TS.

Thanks
Nathan

[jmone]

It occurred to me last night, and I have not tried this, that you might be able to save an MPL with the FileKey column instead of the Filename column, and MC might be able set your data from those.  If MC can use the FileKey in MPL, then you might have a way to save your particle data.

That would also be very good!

[jmone]

When you get a fail, since I don't have your disc material, I'd need an archive of separate mediainfo --full runs on each mpls, and a recursive directory listing of the BDMV folder.  If you can create that and attach (or email me) the zip file, I'll see what can be done.

Better still here are zip files for the two fails that contain a copy of the MPLS + a directory listing (note: you can use Media Info on the MPLS without needing the actual media files).  

FYI - madshi's eac3to also list the correct chapter names and durations for these disks (but you need the whole disk with media files for this to work not just the mpls)

Edit: I've also posted a Q on the syntax to extract chapter information using ffmpeg over on the Zeranoe build forum as this "should" return the same result we see from MC.