Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[glynor]

MCAutoQueue is a standalone application that can be used to fully automate things like Video Conversions, File Replacements, Post-Processing, Ad Skip Processing, and much more.  It uses the power of MC's Smartlists, Database Fields, and Tag-On-Import settings to accomplish its magic.  Here's a look:


Click to embiggen.

Download Here
Current Version: 0.9.9.2

Requirements:

1. JRiver Media Center: MCAutoQueue was developed on MC19. Prior versions are not supported (though it may be functional). Note: Confirmed still functional on MC27.

2. Microsoft .NET 4.5 Framework: This is the .NET framework from Microsoft.  You probably already have it, but if not it is available here:  Microsoft .NET 4.5 Framework Download

3. Windows 10, Windows 8, Windows 7.  The .NET 4.5 Framework is not available for Windows XP and older versions of Windows.  Sorry, guys, no XP support.

Please Note: This is a BETA.  However, this can be considered a release candidate.  No further substantial changes should be expected before the 1.0.0 release.  I've been using it in production now for quite some time.

Change Log:

Change logs for the current release:

* glynor.common-ChangeLog.txt
* MCAutoQueue-ChangeLog.txt
* MCFileIngester-ChangeLog.txt
* MCVideoRedoer-ChangeLog.txt

Included:

I've also included a few quite handy processors that can be used with MCAutoQueue (or completely standalone), or as tools to build your own scripts.  In particular, check out MCFileIngester and MCVideoRedoer details in the thread below.

Getting Started:

You need three main things set up to use MCAutoQueue:  a Field and Smartlist in MC, and some Processors (command line tools) to run on the files with this field filled in.

1. Field:
MCAutoQueue uses a Field in Media Center as a key to pick which processor will be used for a particular file.  The best option is to use a String Type field in Media Center, and then add an "Acceptable Values" restriction to that field.  Here's an example:


(I suggest you use [NeedsProcessing] but you can use anything you'd like.)

To use the examples in the included example configuration file, you'd use this in the Acceptable Values restriction box when configuring the field:
Completed;Failed;Echo Normal;Echo Series Argument;Echo Causes Timeout;Echo No Update;Echo MC Style;AdScan;VideoReDo-iPhone;VideoReDo-Auto

Then, add this Field to your Views in Media Center (as a column, or choose Also Show in the Tag Action Window).  It is also convenient to make a view where you have the [NeedsProcessing] field added as a category, so you can see them in one place and filter by it.  In any case, get it into your Views so you can tag some files with the field.

2. Smartlist:
Then, you create a Smartlist in Media Center that shows any file that has an entry in this field, except for the special values "Failed" and "Completed".  Here's an example Smartlist:

Code: [Select]

-[NeedsProcessing]=[],[Failed],[Completed] ~sort=[Date Imported]
You can use any filters on the Smartlist you want, including things like adding limits, a sort order (to process things in the preferred order), or anything else. You could even, if you want, use a manual Playlist (though a smartlist is almost always going to be preferable).

3. Processors:
In MCAutoQueue itself, under the Settings button, you can define what command line tools you want to run on files that "match" this key.   When it runs, MCAutoQueue will get the list of files from the playlist you made in Step 2.  Then, for each of those files, it will read the contents of the field you defined in Step 1.  And lastly, it will look at the names of the Processors you've defined looking for a match to decide which "Processor" (or command line tool) will be run on the file in question.


Click to embiggen.

Each Processor can take these settings:
Key: The name of the processor, and the value that will trigger it in the [NeedsProcessing] field.
Command to Run: This must be a path to an executable file on disk (exe, script, etc).
Argument: The argument you'd like to use for the command.  This setting is parsed, and works much like using MC's Rename, Move, and Copy tool.  It can be as simple as the [Filename] of the file from MC, or as complex as using Media Center's expression language.
Options:  Each Processor has a couple of options, which will be explained below.
Timeout: When they run, each processor is limited to a maximum run-time, after which MCAutoQueue kills the process and marks the file as failed.  You can modify the timeout setting for each Processor independently here (in minutes).

Running MCAutoQueue

When you run MCAutoQueue, it will pull the contents of the Smartlist, look at the Field to Process, decide which Processor to use, and then execute the commands one at a time in sequence (following the sort order defined in your Playlist in MC).  As each Processor completes execution, MCAutoQueue will then mark the field "Completed" (and mark them as Failed if they fail).

When it finishes with all of the files in the Smartlist, it is done, and can exit.  MCAutoQueue logs all of its activity, and can be controlled by command line options itself.

For best results, you would add Tag On Import rules to your Auto Import settings in MC, that automatically set this [NeedsProcessing] field for you.  Then, you schedule MCAutoQueue in the Windows Task Scheduler to run automatically in the middle of the night.  It runs through and processes your files for you.

You can, of course, use it completely manually, though.  Perhaps you just want to be able to queue up batches of files for processing manually?  It doesn't matter, as it will work either way.

Command Line Options:

Code: [Select]

MCAutoQueue (by glynor) v0.9.5 Command Line Help

Usage:
Process Media Center files using settings provided in the AQConfig file.
If no --config option is provided, the most-recently used automatic config file will be used.

Options:
  -c, --config=AQCONFIG      the AQCONFIG file to use for processing.
      --notimeout            Disable all processor timeouts (not recommended).
  -e, --autoexit             automatically exit after running.  This option
                               is ignored if an error occurs and the relevant
                               preference is set.
      --AutoExit             automatically exit after running (alternate
                               syntax).
  -a, --autorun              automatically run the processor.
      --AutoRun              automatically run the processor (alternate
                               syntax).
  -m, --runmin               start up minimized.  This has no effect if
                               autorun is not also set.
      --AutoMinimize         start up minimized (alternate syntax).
  -h, --help                 show this message and exit

So, download it, give it a whirl, and tell me what you think!

[glynor]

Processor Options:

Argument Styles:
The argument can be of two types: Normal, or MC Style.

Normal:
Normal Style uses simple Field replacement, using the [FieldName] notation we're all used to here.  The nice thing about this mode, is that it automatically adds quotes around results from fields that need them (because they contain spaces), making it simple to use.  So, for example, if you wanted to send files to a script that takes a few options and a filename identified by a -i= option, you'd just put this in the Argument box for that Processor
-a --MakeCool --fast -i=[Filename]

And MCAutoQueue will fill in the [Filename] field using the information from MC, adding quotes as needed.  You can use any named Field in MC, but not expressions in this mode.  If you include text that looks like a Field, but which isn't "tagged" on the file in question, then the field name will be passed through untouched (so if the options to your command line tool use square brackets for something, that isn't a problem).

MC-Style:
This sends the text exactly as it is to MC, and asks MC to return the results of the expression.  This allows you to do full expression logic, using all of MC's language's bells and whistles.  It also, incidentally, is faster than my Regular Expression based "normal" mode.  However, it does force you to manually quote any fields that might have spaces, or else no quotation marks will be included (which will probably keep your command line tool from understanding the command line arguments you sent it).

Update MC:

Normally, when MCAutoQueue finishes processing a particular file, it will change the entry of the Field to Process for that file ([NeedsProcessing] by default), to Completed for files that "succeed" and Failed for those that "fail".

However, if the command line tool or script you are running is "MC aware" then it could conceivably set this field for you, perhaps to reassign it to another Processor.  This would allow you to "chain" Processors together.  For example, with a script, you could have it Quick Stream Fix video files in VideoReDo, and then (if that succeeds) compress them using HandbrakeCLI on the next "round".

Unchecking the Update MC box for a particular processor tells MCAutoQueue not to update that field at all, and to leave it alone.

Please note: If you use this, and you don't clear that field somehow (manually or by script) then MCAutoQueue will reprocess the file over and over again each time you run it.

Saving and Loading Settings:

MCAutoQueue can save out the set of loaded Processors, along with the specific Playlist and Field to Process, to an XML settings file on disk.  Then, you can use the --config command line option to load these settings when you are running.  This can allow you to have multiple different setups in MCAutoQueue all simultaneously.

Perhaps you want to have one "setup" for music files, and another for video files?  Or a set of processors just for Classical music, or TV Shows, and a different set for movies (which run on a different schedule)?  Or you want to compare the results of compression engines with different options?  Save the settings out and you can do this!

Here is an example of an aqconfig settings file (this is the "real world.aqconfig" file included in the Example AQConfigs subdirectory of the installation folder):

Code: [Select]

<?xml version="1.0" encoding="utf-16"?>
<AQSettings xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <QueuePlaylistPath>MCAutoQueue</QueuePlaylistPath>
  <FieldToProcess>NeedsProcessing</FieldToProcess>
  <ProcessorConfigs>
    <ProcessorSetting>
      <Name>AdScan</Name>
      <Command>Processors\MCVideoRedoer.exe</Command>
      <Argument>--adscan --source=[Filename] -a -e</Argument>
      <MCStyle>false</MCStyle>
      <UpdateField>true</UpdateField>
      <Timeout>120</Timeout>
    </ProcessorSetting>
    <ProcessorSetting>
      <Name>VideoReDo-Auto</Name>
      <Command>Processors\MCVideoRedoer.exe</Command>
      <Argument>--profile="Auto Profile" --type=Replace --deletesource --source=[Filename] -a -e</Argument>
      <MCStyle>false</MCStyle>
      <UpdateField>true</UpdateField>
      <Timeout>45</Timeout>
    </ProcessorSetting>
    <ProcessorSetting>
      <Name>VideoReDo-iPhone</Name>
      <Command>Processors\MCVideoRedoer.exe</Command>
      <Argument>--profile="iPad / iPhone 4G" --type=Replace --deletesource --source=[Filename] -a -e</Argument>
      <MCStyle>false</MCStyle>
      <UpdateField>true</UpdateField>
      <Timeout>360</Timeout>
    </ProcessorSetting>
  </ProcessorConfigs>
  <DisableProcessorTimeouts>false</DisableProcessorTimeouts>
  <MinimizeProcessors>false</MinimizeProcessors>
</AQSettings>
Global Options and Advanced Settings:

Minimize Processors:  Select this if you want MCAutoQueue to automatically start any Processors up minimized.  Note: This option is saved to the .aqconfig file, so you can have different choices for this per-settings-file.

Disable Processor Timeouts:  This is basically "Safety Off mode".  If a given processor lasts three weeks, months, years, or millenia, MCAutoQueue won't stop it or force it to time-out.  It'll just patiently wait until it is done, and then run the next item in the queue.  This, obviously, could spell trouble if you have a crashed Processor.

Default Timeout:  When you make a new Processor, it gets this Timeout setting by default.  This is an automatically-saved global option, just there for convenience if you don't use the default I preset very often.

Logging: You can rotate the log file, using a variety of choices, or disable logging altogether.  Pretty self-explanatory.

[glynor]

0.8.0 (12/04/2012) -- BETA

1. Initial Release

Known Issues:

1. The Settings button does not properly disable while the Queue is processing, which would allow you to go in and muck about with the Settings mid-processing, a decidedly bad idea.
2. Validation of the aqconfig files is early.  Manually edit them with care.  If you run into trouble, delete your MCAutoQueue.exe.config file in the installation directory, and it will reload the defaults.
3. Error reporting is weak in the UI.  The log is pretty good though.  If you get any general exceptions or other unexplained errors, please retain the logs.

[rjm]

Very nice work.

I am trying to think of possible applications for myself. One idea is that I am trying to replace as much video as possible in my library with IOS friendly mp4.

Does anyone know if there is an mkv to mp4 converter that works in unattended mode and produces excellent results without having to tweak lots of conversion variables?

[glynor]

One idea is that I am trying to replace as much video as possible in my library with IOS friendly mp4.

Does anyone know if there is an mkv to mp4 converter that works in unattended mode and produces excellent results without having to tweak lots of conversion variables?

This is certainly one of my goals as well (in my case, I want to be able to create automatic stacked MP4 versions of certain videos).  The easiest way?  Handbrake.  It is quite scriptable:

https://trac.handbrake.fr/wiki/CLIGuide

There are nice presets built into Handbrake that can do things like this quite easily.  It can be (and usually is for simple Apple-compatible exporting) as simple as running:

HandBrakeCLI -i <input_filename> -o <output_filename> --preset="Universal"

This is one of the example scripts I'm going to create in the coming weeks.  Combined with my CopySidecar.vbs script (already included), you could have this workflow:

1. Tag files with [NeedsProcessing] = "Handbrake Universal"
2. MCAutoQueue runs, and fires off a Handbrake Conversion script.
3. This script first copies the existing sidecar file over to a new copy with the extension changed to .mp4 (that way, when Handbrake creates the new destination MP4 file, it gets auto-imported with all the existing metadata intact) -- this is my included CopySidecar.vbs script.
4. Then, it calls HandbrakeCLI to do the conversion, putting the output file right next to the input file.
5. Then, it runs a final cleanup script/utility that (optionally) removes the original source file from MC, or maybe stacks the new one with the original.

Handbrake isn't perfect in every way, and there are cases where FFMPEG might do a better job, but it is easy to use, reliable, and produces quite good quality with the current builds.

Another script that I'm going to provide (even sooner) is one that does almost the same exact steps as above, but instead of #4 (the video transcoding step), it will simply run VideoReDo's QuickStreamFix on the file and then pump it through MKVMerge to enable a non-transcoded (but fixed) conversion from TS to MKV.  Since it won't transcode, it will be very quick.

Lastly, I have some ideas for doing Re-Ripping replacements (so if you have old MP3s you ripped eons ago, and you re-rip to FLAC, it will go into MC's Library and replace the existing MP3 file entry with the new FLAC copy).

[glynor]

One thing I'm mulling changing with this already is the setup of the Processors, and I'd love some feedback, as I'm waffling back and forth on how I want to do it.

Right now, MCAutoQueue is pretty simple.  You have two "pieces" to use to construct the command line that will execute for that Processor: the Command (application) to run, and the Argument (the field from MC to use as the command line arguments for the command).  MCAutoQueue checks that the Command given for a particular processor exists on disk before it runs it, and therefore you can't add extra "static" command line arguments to the Command box when you are setting up a Processor.

My original thinking was that you could accomplish anything you want using a Calculated Field in MC and use that as your Argument for that particular command.  Or, if you want something very simple that never changes depending on the input file, you could just wrap your tool in a simple VBS Script or BAT file that provides the other command line arguments needed.  That is true for both things, but it might be too much work for a simple conversion task like the Handbrake one above.

So, I see three possible options (ranging from easiest to code to hardest).

1. Leave it as is.

2. Change it so that MCAutoQueue no longer sanity checks the command line you provide for the Processors to see if they exist.  This would be the simplest option, of course, and then you could add whatever static command line arguments you want to a particular command line tool, and the [Filename] (or whatever field is chosen) will be added as the "last" argument in the chain.

3. Change it so that MCAutoQueue has two Arguments for each Processor.  The first would be a string you type in like this:
-i <$MCfield> -o <$AutoOutput> --preset="Universal"

Then, you could define a second Argument based on a field.  When the Processor runs, it would replace the <$MCField> part with whatever you provided in this second Argument Field, and replace the <$AutoOutput> item with a filename generated based on the source filename.

I don't know how much I like this third option, though.  First of all, it would be clunky to use, and make the UI more cluttered and confusing (and there are already some esoteric steps to use this if you aren't comfortable making new Smartlists and Fields in MC).  More importantly, I like that MCAutoQueue doesn't assume that the Argument given will be the [Filename] tag, but allows you to use any tag you want.  In order to do the <$AutoOutput> magic, it would need to ensure that the field given was the filename (maybe not though...)

And, frankly, MC has all the tools built-in to generate this string with an existing Calculated Field.  You could easily make a [Handbrake Universal Argument] field, which generates the same thing, and pass that to your "Handbrake Universal" Processor in MCAutoQueue.  Perhaps all of this would be much better served with a separate "generic processor" application that could wrap other command line tools.  Hmmm... I don't know exactly what to do.  Opinions would be appreciated...

[rjm]

This is certainly one of my goals as well (in my case, I want to be able to create automatic stacked MP4 versions of certain videos).  The easiest way?  Handbrake.

Thanks. I will play a bit with Handbrake to see if I can get my confidence high enough to run it on batches of video with your app.

[MrC]

Some suggestions:

1) Use a command line option such as -f or -c to indicate the config file will follow.  This allows you to reserve and use trailing arguments as one or more filenames.

2) It seems you can assume the first word in the command line is the command - thus you can always check for its validity as a processor/executable.

3) I'm not sure you'll be successful trying to cover all cases of argument passing, or abstract it to a convention which suits all needs.   Why not consider just passing the entire argument chain Args in an MC field.  E.g. -i [Filename] -o [Filename].out -x -y -t 20, etc.  If a processor needs static arguments in all cases, define those in your configs as some static argument token, and construct your command as <cmd> <static args> <MC supplied argument chain>.  You can have processor variants, such as Convert, Convert-Large, etc. which include the relevant static arguments so that uses don't have to include these in the argument chain.

4) For ease in documentation, debugging, and support, consider using predefined fields that your utility requires, such as _MCAutoQueue and _MCAutoQueueArgs.  I'm not sure you gain much by generalizing this to be any configured variable.  (This adds a layer of support questions - "Did you configure the variable name in the config?"  "Does it match the variable you created in MC?")

[glynor]

2) It seems you can assume the first word in the command line is the command - thus you can always check for its validity as a processor/executable.

This is what I'm thinking of doing.  I'll have to play around with it a bit to see what really makes sense.  I really might not need to do the check at all, depending on what status the Process.Start() command gets back if the command doesn't exist/work (if it throws an exception, then I need to deal with it, since it is done in a Background Worker, if it just fails, then I can just report it as failed and not worry about it).

1) Use a command line option such as -f or -c to indicate the config file will follow.  This allows you to reserve and use trailing arguments as one or more filenames.

I'm confused about most of the rest of your comments, though... In particular this one.

Do you mean in the Command to Process field?  If so, this would have to match exactly what is being sent to the external processor (which is why I was thinking of using some kind of string-replacement scheme).

If not, and you mean MCAutoQueue's command line options, then no, it doesn't apply.  MCAutoQueue doesn't care what order you give the command line options.  It just parses each argument, and anything it gets that isn't a pre-defined command line argument, it assumes is an aqconfig filename (it checks this for sanity as well, though).  You cannot pass multiple aqconfig filenames, and wouldn't ever need to.  If you look at the structure, you'll see why.

The reason I want to keep the fields and playlists flexible is because I can envision people having multiple playlists and fields that they use for different purposes.  So, say you have one Queue List that handles all video post-processing and is run automatically, but you also want a separate "set" that does audio-file replacement on demand.

With this scheme, you can just make two different aqconfig files, and launch them with:
MCAutoQueue.exe --AutoRun --AutoExit "my video processors.aqconfig"
MCAutoQueue.exe "my audio processors.aqconfig"

And these two things would look at and process two completely separate Playlists and Fields in MC.  They could be scheduled separately in the Windows Task Manager (or schedule one, leave the other manual, or whatever).  It does, of course, provide default examples.  And, if all goes well, my future setup wizard would walk you through the process of creating these Library fields and Playlists.

So.... I'm a little confused.  But I'm still thinking about it.

I'm leaning towards a modified version of #2 above, where it sanity checks the first item in the command list, but then just lets you put whatever else you want in there.  For complex things, just use a BAT file or a script (and I'll probably make some nice ones for things like Handbrake myself).

[glynor]

Re-reading here...

3) I'm not sure you'll be successful trying to cover all cases of argument passing, or abstract it to a convention which suits all needs.   Why not consider just passing the entire argument chain Args in an MC field.  E.g. -i [Filename] -o [Filename].out -x -y -t 20, etc.  If a processor needs static arguments in all cases, define those in your configs as some static argument token, and construct your command as <cmd> <static args> <MC supplied argument chain>.  You can have processor variants, such as Convert, Convert-Large, etc. which include the relevant static arguments so that uses don't have to include these in the argument chain.

This was exactly my plan already, except without the separate static arguments field in MCAutoQueue.  You can do that in your calculated field that you send to that Processor.  So the way it is set up now is that you define two items per-processor.

<command> <argument_field>

The Argument field is a field in MC.  Any field you want, and each processor can have their own.  Command is an executable file (exe, vbs, bat, whatever).  When you run the queue, it retrieves the contents of the field specified for that file from MC, does essentially Shell Run:

<command> <arguments>

The issue is only that if you want to run a command that takes complex arguments (more than just a source filename), you have to make a custom field in MC, even if the arguments you want to add don't change from file to file.  That seems silly.  I think I'm going to change it so that the <command> box in MCAutoQueue will let you put whatever arguments you want in addition to the EXE (and like you said, the EXE will be the first thing in the box, so I can still verify it exists if I want).  Then, if all you want to do is something like:  mycommand.exe -mp4 --noresize --foo -o:<filename> you'd just put:

mycommand.exe -mp4 --noresize --foo -o: in the Command box, and
Filename in the Argument box.

The only time you'd need to do a custom calculated field would be if you needed complex arguments where order matters, or if you needed to send multiple fields from MC into the processor (in which case you'd also want to consider wrapping your command in a script or BAT file to make life simpler).

Does this make sense?

If more complex options are needed, I'm willing to add them, but my goal is to keep MCAutoQueue simple.  If complex edge-cases can be adequately handled by simply using a calculated field in MC as the argument for that particular processor, I'm thinking that is the way to go.  I just don't want it to end up that you need to make a calculated field for everything you use it for...

[MrC]

Why not just use field placeholders in your config files, such that the user doesn't have to pass arguments, and instead your app queries the values from MC?  This way, its sure to always get the correct values just before the command executes.

   mycommand.exe -mp4 --noresize --foo -i:[Filename] -x [Media type] -n [Name] -s [Series] --debug

where each value in brackets is looked up from MC for a given file.

[glynor]

Why not just use field placeholders in your config files, such that the user doesn't have to pass arguments, and instead your app queries the values from MC?  This way, its sure to always get the correct values just before the command executes.

   mycommand.exe -mp4 --noresize --foo -i:[Filename] -x [Media type] -n [Name] -s [Series] --debug

where each value in brackets is looked up from MC for a given file.

Ahhh.... Now I see what you meant.  I considered that way back when I first started this project.  You'll have to excuse me a bit, but take note what time I posted this up above (EST, for reference).  My brain not work right today.

The answer is this:

my goal is to keep MCAutoQueue simple.

Basically... Why reinvent the wheel?  MC already has a powerful expression language with a complex parser.  No matter what I implement, if isn't going to be as powerful as that, and I'll have to invent my own system (and maintain a more complex parser, which might be re-run with thousands of different files at a time and passed all kinds of weird things I don't expect).

JRiver has already done that work.  If you want something like your example, regularly, to go with a particular processor, make a calculated field in MC and pass that!

So, in other words, I'd consider it (and I could implement some kind of simple find-and-replace scheme).  But...

What if a command needs brackets?  Then I need to find a way to escape them, and... I don't know.  Seems like a rabbit hole to me, when the answer is right over here already built in, and it can use complex Regular Expressions and If-Else and the whole bit.

[MrC]

How will you implement a pipeline command chain should you desire?

   cmdA argA | cmdB argB1 argB2

If you're going to place arguments in an MC field, then you have to place the pipe and subsequent commands there too.

There's nothing complex about using replacement tokens.  You don't have to use brackets, you can use any number of ways to indicate in
the command syntax that something is to be looked up in MC.  And supporting an escape character is pretty trivial.  There's no need to replace MCs expression parser - the query to obtain a field's value will do the right thing and just return a text value which becomes your argument parameter.  You're going to have to handle quoting anyway.

Anyway, just some thoughts.  See how it works out.

[glynor]

Actually, no...  You're right.

Well, with this:

How will you implement a pipeline command chain should you desire?

   cmdA argA | cmdB argB1 argB2

How I'd implement it would be with a wrapper script.  But you're right about this:

There's nothing complex about using replacement tokens.  You don't have to use brackets, you can use any number of ways to indicate in
the command syntax that something is to be looked up in MC.  And supporting an escape character is pretty trivial.  There's no need to replace MCs expression parser - the query to obtain a field's value will do the right thing and just return a text value which becomes your argument parameter.  You're going to have to handle quoting anyway.

And, I'm not right now.

Or not really.  I tried to implement a special case for the [Filename] (as that's really all I needed right now), but it doesn't even seem to be working correctly.  I hadn't noticed because almost all the files I've used it with happen to have no spaces in their names!  My bad (SageTV, which doesn't use them, and my work systems, which are all space-character-free because they're used in URLs and RTMP addresses).

You're right.  That's the simplest solution.  I won't be able to tell if the user is passing me a complex, parsed expression as the argument (where they don't want it quoted) and the simple fields like Filename and Artist, where they probably do.

It needs a parser and replacement fields.  Okay, that isn't impossible.

MrC, I'm going to be hitting you up for suggestions (and feel free to fire away).  My thinking is this:

Allow any MC Field to be inserted into an otherwise string literal.  I'd like to use [] to denote them, if it doesn't cause major problems, since that's our lingo anyway and it is used in MC.  But no other complex parsing, as that can be done outside in MC itself.  I'll quote all fields that contain spaces, and pass through any that do not.  The rest is a string literal sent to the command line (essentially).

How's that sound?

[MrC]

Pretty much that's the idea!

[glynor]

Pretty much that's the idea!

Kept me up half the night reading about RegEx.  I know now that this was really just your ploy to force me to learn it.  But...

My brain hurts, and I'm still not 100% sure what to do.  I'm going to play with it a bit, but I think I'm going to be asking for help.

EDIT:

Okay... My main questions are:

1.  I want to do a first pass with a Regex and do a parse for any non-printing characters (stuff like Control-C and NewLine), and just strip them all out.  That's mainly in case people copy-pasta in from a text editor or whatever.  The textbox itself is a single-line job in a DataGridView, so...  I'm just worried about weird non-printing characters messing up the command lines sent, without being immediately obvious what is going wrong.  So, I figure, they're useless anyway, and I should strip them out.

But do I just have to do some kind of crazy "all the keyboard characters" match to do this?  Suggestions?

2. Then I can do a more traditional RegEx.Match run to grab the values inside the brackets.  I need, obviously, something along the lines of: "\[.*\]", but including grouping to capture the .* in the middle, and some way to avoid capturing the ones marked by whatever I use as an escape character (sorry, still learning, I only spent a couple hours reading).  Then I should be able to iterate over the groupings with Match.Groups[n].Value to retrieve my replacement values from MC, and then all it would be is a simple substring replace to write the new values back in (once I decide whether to add "" or not).

Also, I'm thinking of using [] to denote fields (as discussed above) with the ^ as the escape character.  Since ^ is an escape character at the cmd prompt anyway, I figure it should be relatively rare (particularly when combined with brackets).  So, the rules would be simple:

\[.*\] = Field
\^\[ | \^\] = should be ignored (not a field).  However, I need to also strip out the extra ^ characters when they're next to a [ or a ].  Probably at the end of the parsing, I'd assume?  (And then, if someone does need to do ^[ or ^] for some reason, they can just double up on the ^ character, since it'd only replace the one character.

Strategies?  Suggestions?  Does that seem like a good choice for an escape character?

MrC, RegEx help for a hopeless newbie?  I'm working on it, and I do want to understand whatever I put in there (for maintenance reasons), but... RegEx is hard, as we've discussed before.  I know it is mostly a mental thing in that the syntax makes my eyes bug out (too many weird escapes and slashes and brackets and whatnot).  I wish there was a "more verbose" version with less shorthand (like a VB of RegEx).... Sigh.

[glynor]

\[.*\] = Field
\^\[ | \^\] = should be ignored (not a field).  However, I need to also strip out the extra ^ characters when they're next to a [ or a ].  Probably at the end of the parsing, I'd assume?  (And then, if someone does need to do ^[ or ^] for some reason, they can just double up on the ^ character, since it'd only replace the one character.

Right, so I need a RegEx that does:

\[.*\] but NOT \^[.*\^\] and then gets a group on the .* in the middle of the matching set, right?  I'd been thinking last night (late) that I could just ignore the first part ^[ and look for a space before the [, but that doesn't work if the user wants to do something like --inputfile:[Field].  So, it can't rely on spaces, it has to look for both patterns.

I need to get one of those RegEx tester thingies...

EDIT:  Wait... Why use a separate character for the escape?  I can just use double-brackets to escape, right?

So, [Artist] would output "Pink Floyd"
But, [[Artist]] would output "[Artist]"

That seems easier.

[glynor]

One other thing to consider...

I can easily add a GetFields() method to my existing MC Singleton class that returns a List<string> of all of the valid Field names in MC.  Would it be useful/valuable or hurtful to check any results from the RegEx match against this list to see if they're actually valid Field Names (if not, I can treat them as literals).

I'm wondering if this could cause more problems than it would solve, though... Perhaps the expected behavior would be to return null for unmatched Fields?  That is, I believe, what it would do if I just ignored it and tried to Get the value of them all from MC anyway (the IMJFileAutomation.Get(strField, bFormatted) function should return null if it the strField you send a field it can't find, so I can just replace those in-line and call it good).

And, it looks like want I want is something to do with the Regex.Replace() with a MatchEvaluator.  But that is currently opaque to me, unfortunately.  I need to read about Lambda expressions and the MatchEvaluator class....  Either that or something like this, but that looks like it is way more complex than I need.

[MrC]

Before you spend much more time on this... consider:

   https://www.google.com/search?q=argument%20processing%20.net

[glynor]

Before you spend much more time on this... consider:

   https://www.google.com/search?q=argument%20processing%20.net

Hmmm... I looked into that stuff when I implemented the Command Line parsing for this application itself (but decided my needs were so simple, for now, that it wasn't worth the effort of building something more flexible).

But I'm not sure how that applies to this.  I don't need to parse the command line in any way, just execute it (which just means I assign Process.StartInfo.Argument a string).  I just need to do Token Replacement in the string first.

[MrC]

But I'm not sure how that applies to this.  I don't need to parse the command line in any way, just execute it (which just means I assign Process.StartInfo.Argument a string).  I just need to do Token Replacement in the string first.

Yes, you do.  Otherwise you've just opened up a massive security hole.

Consider what happens when values contain metacharacters that you'll simply interpret.  Redirects, pipes, commands that remove files, etc.

Anytime you're going to execute a command line from unfiltered data, you must untaint that data.  Therefore, you need to examine each component.

[glynor]

I responded in a PM.

[glynor]

Yeah, I looked into it more, and I'm pretty sure this is not needed for the way I'm launching the external process.

The Argument field is NOT processed by cmd.exe, unless the user selects cmd.exe as the executable (file) to launch (which would probably be a bad idea).

In other words, you can't pipe commands like in your example above, MrC.  The file that is "executed" is specified separately, and is executed directly (not by cmd.exe).  If you really wanted to pipe, you could tell MCAutoQueue to use cmd.exe as the command, and then pass your "real command" as the Argument.  All Arguments are sent only as parameters to the executable specified as the "Command".

Using cmd.exe would be a bad idea though because the fields in MC could contain dangerous characters as you mentioned.  In fact, I may block it and require an explicit config override or something.  I'm not going to protect against this by parsing all the command arguments (that would be terrible).

I'm struggling to get a RegEx working the way I want though...  I'll post more later.

[rjm]

Thanks. I will play a bit with Handbrake to see if I can get my confidence high enough to run it on batches of video with your app.

I tried Handbrake and it's superb U/I replacement VidCoder. Neither seem to do what I want.

I want an app that will take as input an x264 mkv, pass through the video unchanged, transcode the audio to aac if not already aac, otherwise pass the aac through unchanged, pass through any imbedded subtitles or chapters unchanged, and change the container to mp4.

Anyone know of such an animal?

[glynor]

If the goal is to get Apple Device support, remuxing it to MP4 isn't going to work most of the time.  The iDevices (as is true of all mobile devices) only support a subset of the H.264 spec, and most MKVs you probably have lying about don't have the right data rates, profiles, and whatnot.

The iPhone 4+ (and modern iOS devices of the same era) support more than Base Profile MP4 (finally), but you'd have very hit-or-miss support if you just remux them.

But....

To answer the question, not really.  Not for a reasonable price.  Telestream Episode can do it, but it can't take MKVs as source files (supposedly they were adding it, though, and I'm still on an older version).  But that's a pretty penny!  It would also be worth it to check out VideoReDo (which is a handy tool anyway).  They have a free trial, but it still isn't cheap (and is really more of an editor, and is kind-of a pain to script).

You could use MP4Box and BeSweet to script it, if you're feeling ambitious.  But I don't know of much free.  Maybe HDConvertToX.

EDIT:

This is an older article (it mentions YouTube's videos don't work, which they now do, for example), but it covers many of the basics:
http://www.niallkennedy.com/blog/2010/07/h264-video.html

The iPhone 5 now supports High profile level 4.1, so it has more comprehensive H.264 support, but that's not always been the case, so unless you have a latest-gen device, you're going to have mixed results.

[rjm]

Thanks. I tried my old swiss army knife MediaCoder and it works except does not handle subtitles.

I think I found the perfect solution: MkvToMp4.
http://forum.videohelp.com/threads/340675-Mkv%D0%A2%D0%BEMp4-v0-222-rapid-tool-for-repack-Mkv-to-Mp4
http://forum.doom9.org/showthread.php?t=163050

Sorry to hijack this thread. Will shut up now.

[glynor]

Nice find.  Hadn't seen that one.  Does it take files from the command line?

PS.  The dude that posted in the thread in all caps was a riot.  I don't know that I'd have been able to respond intelligently to him like the author did.

[rjm]

Does it take files from the command line?

Unfortunately not. Author in thread explicitly states no plans to support command line operation.

[glynor]

So, I figured out two ways to do the discussed multi-field replacement in the Argument.

Option 1:  I should be able to get replacement working using a RegEx and essentially a modified version of this approach: http://blogs.msdn.com/b/akirsman/archive/2011/09/13/c-token-replacement-using-regex.aspx

I have a RegEx that does what I want, but I'm reasonably sure it isn't "correct".  MrC, care to comment?  My RegEx is:

Code: [Select]

\[([^\]]+[^\t\r\n])\]
I'm trying to ignore line breaks and other special characters, but capture what I want.  It seems to be working fairly well, but I need help.  It won't properly handle "double-braces".  I had a version earlier today that I think worked better, but it is saved at the office, and I don't think I can recreate it easily tonight (I've tried).  In any case, assuming I get the proper RegEx pattern figured out, it works.

The nice thing about this method, though, is that I won't have to worry about escapes at all.  Any "match" that is not a valid field for that particular file doesn't get replaced.  I already wrote a quick function for my MC wrapper that returns a list of strings that contains every field MC knows about (on my system around ~280).  Then, I can pass a file in and do a mcFile.Get() for each possible field, and build a string dictionary, just like he uses in that example.  It parses plenty quickly (for my purposes and for the types and lengths of text we can expect to have to parse), which I was worried about initially with this method.

But there are problems with this approach.  So, take this example Argument (this would be a dumb argument example to use, but bear with me):

[Series]-[Season]e[Episode] -io:[Name] : [Cheetos] [Filename] -aqv /xx

Using this system, and feeding it a test file (an episode of Dexter), it spits out this:
Dexter-7e2 -io:"Sunshine and Frosty Swirl" : [Cheetos] "M:\Video\TV Show\Dexter\07\s07e02 - Sunshine and Frosty Swirl.mkv" -aqv /xx

So, that looks pretty good, but the problem is that there doesn't seem to be a way to pad the numbers like the way MC automatically formats them (normally) via the Automation interface.  The Field Get() method has a formatted parameter, but nothing changes with this (and other) examples I tried when I turn it on or off.  But, it would work.  My function handles adding parenthesis properly automatically, and it ignores anything not recognized (hence the Cheetos example).

I can implement that pretty easily.  However...

Option 2:

I just discovered the wonders of IMJFileAutomation.GetFilledTemplate().  I'd seen that in there before, but forgot about it.  MC can do all the work.  You can feed MC a line of text, using the full Expression language, and it'll spit out the results.  It still doesn't do the automatic [Episode] and [Season] padding, but it doesn't much matter because you can use expressions.

The code is a heck of a lot simpler too.  Literally, all I have to do is call .GetFilledTemplate(inputString) on the file object (which I already have), and it spits out the same exact thing.  The beauty part is that you can add things like PadNumber([Season], 2) and whatever else you want, and it'll evaluate it the same way you already expect MC to evaluate it.

Here's the rub, though...  It doesn't handle quoting at all.  Of course not, MC's engine doesn't quote anything.

So, in order to produce identical results (including the quotes) to what I produced above, you have to use something more like this as the argument:
\"[Series]\"-PadNumber([Season],2)ePadNumber([Episode], 2) -io\"[Name]\" : [Cheetos] \"[Filename]\" -aqv /xx

It works perfectly, it is just that you have to (obnoxiously) add the quotes manually, and escape them with the \ character.

I'm leaning (strongly) towards going with the second method, as it is far more powerful and flexible.  But it seems a bit harder to use.  I could, of course, offer both, with a checkbox to enable "MC-style parsing" versus "default parsing".  One would allow things like Expression logic and number padding, and the other would be simpler and auto-quote the results.

What do you think?  Is the MC-style parsing all you'd need?  Is the escaped manual quoting too difficult to deal with?  Considering it would make it more complex and confusing, is it worth adding both styles (and which do you think should be the default)?

[glynor]

For option one above, I think with this I really do need two RegEx patterns instead.

I need one that I can use as a replace pattern to strip out any stupid non-printable characters (return, control-c, etc) and just allow "normal text" (alphanumerics, whitespace, and punctuation).  Filter all that crap out of the input Argument first (in case people copy-pasta them in from Word or something equally dumb).

And then the second one just captures any text between two braces.  One problem I encountered with this approach was that the RegEx patterns I kept coming up with wouldn't properly handle things like [[Episode]] (where you want the result to be [2], or something like that).  Help from someone who is better at RegEx (which means, pretty much, anyone but me) would be greatly appreciated.

If I don't get help here, I'll ask at work.  There are a bunch of database programming warriors I know I can ask, so...

[MrC]

Maybe this is more what you want:

   \[([^\][]+)\]

This matches a bracket [, followed by anything not an opening or closing bracket [ ], followed by a closing bracket ].  This forces innermost pairs of brackets, and doesn't care what is outside.

[glynor]

Thanks, MrC!  You're the best!  It was, of course, simpler than I was making it.  I really need to get a book or something.

I'd still love a comment if you can come up with an example of an argument string that would lead to trouble if sent unparsed for craziness so that I can test it, but I do think I'm okay going this route.

I've also implemented a function that strips out invalid (non-printing) characters from the Argument input string, without using RegEx actually, so it was easy for me to do.  I did it the "cheap way" but it should work.  I'm just basically replacing all characters below ASCII 32 and above (whatever it is) with null, without even reporting any issue to the user, just doing it when the property is set.

So, it is working well.

I'll have a new build in a day or two.  I'm making these changes (among a few others):

1. The argument parsing scheme discussed above has been implemented (mostly, it just needs a tweak or two).  It will be available in two modes:
1a. Default - replaces any [Fields] given, and valid for the file in question, with the contents of the field from MC.  Wraps the contents in quotes if it contains spaces.
1b. MC-Style - as described above, gets a Filled Template from MC, which allows you to use expressions in your Argument and have them evaluated, but requires you to manually add quotes as needed.

2. The Settings DataGridView will have a Button Column with a browse button next to the Command field, allowing you to browse the filesystem and select your processor that way.  You'll still be able to type it in manually.

3. If you select cmd.exe as your Processor Command, it will warn you the first time that this may set your house on fire, erase your hard drive, or kill your dog.  The same thing, of course would be true if you pick other complex parsers like Perl or something, but if you're getting into that, we're going to assume you know what you're doing.  This isn't implemented yet, but I'm going to do it.

4. And a few other odds and ends.

Thanks for the help, MrC!  This method is WAY, WAY better.  That's why I wanted to get this out there.  I knew people would find things that I was doing dumbly (in my quest to solve my particular problem).

[MrC]

Sorry about my poor response time re: the quoting PM's etc.  My wife is home sick this week after a mad New York work trip with too many late nights, and I was reworking our home network (added a ProSafe GS716T Network switch to provide Gb Ethernet to all rooms).

Anyway, it seems like you have the process launching stuff managed.  The only issue that is possible at this point is if any of your processors call cmd.exe with your arguments.  While this technically wouldn't be your fault, users will be ticked when issues occur.

So, to avoid that, you might consider stripping-out any dangerous shell meta-characters to be safe.  Sure this will remove nice pipe characters, or redirects from some titles, etc., but better than the alternative.

[MrC]

You might consider that MC supports Unicode, so your token replacements may contain characters that are valid, but triggered by your current ASCII range checks.  You can use regular expression character classes to be more robust:

    http://msdn.microsoft.com/en-us/library/20bw873z.aspx

if you want.

[glynor]

Sorry about my poor response time re: the quoting PM's etc.  My wife is home sick this week after a mad New York work trip with too many late nights, and I was reworking our home network (added a ProSafe GS716T Network switch to provide Gb Ethernet to all rooms).

No worries at all.  I've been terribly sick the past two weeks as well, which has slowed my progress a bit on this.  Children are horrible disease vectors.  My daughter is lucky she's adorable.

I'd add that I love, love, love my ProSafe switches.

While this technically wouldn't be your fault, users will be ticked when issues occur.

Agreed, and if it becomes a major hassle that's what I'll do.  For now, I'm hoping the warning will suffice.  It was a very good point to bring up.

Honestly, it shouldn't be that much of an issue.  The main danger would be if there was a | or something in a MC field, which you didn't realize, and then sent it to cmd.exe and it hoses your system.  But, that's unlikely to occur because it will quote any field that has spaces, so unless your field contains both no spaces and a poorly placed metacharacter, it should be fine.

That's a pretty small problem window for something that would clearly be a quite advanced usage model, I think.  I could be wrong, but I'm going to go with the warn but let the user act crazy if they want system.  If it blows up in my face, well, then you can say I told you so.   

You might consider that MC supports Unicode, so your token replacements may contain characters that are valid, but triggered by your current ASCII range checks.

I'm only doing the filtering on the Argument string the user puts into MCAutoQueue, not the result from MC, so this shouldn't be an issue.  I'd considered doing that (it was my original plan), but for these purposes I think this is sufficient.  If it becomes an issue for some reason, I'll change it.

The only thing I can imagine it impacting is if someone needs non-ASCII characters in a Field Name (or otherwise sent to the command line).  That seems like it would be rare, but if it happens and someone complains, I'll revisit it.

[glynor]

On the last thing, I'm already reconsidering...

We'll see.  I'll take a look.  I implemented the filtering with a method call, so I can change the method internally without much effort.  I'll see if I can get the RegEx character filtering to work, or if it makes my brain explode.

[glynor]

I should note, this does not work with the latest public experimental 18.0.90 build (nor do pretty much any .NET applications or VBS scripts that try to grab the COM Automation object, unless they have some other tricky way to do it).

Matt's on it.

[Ekpen]

Greetings:

Thanks for writing this :

In the "Acceptable values" field, the content is truncated, can you write out the complete words?

Also, I still have the file, not yet installed.
Do I put this in its own folder or does it go to a specific folder.?

If there is going to be a final version, can this be placed in the Interact utility area.?
Thanks.
Ekpen.

[glynor]

In the "Acceptable values" field, the content is truncated, can you write out the complete words?

The Acceptable Values field when you are making a custom field in Media Center can be used to limit the choices in MC when setting the field to a pre-defined list.  So, for example, if you made a field called [Colors], but you wanted the only choices possible to be Red, Blue, Green, Pink, and Black, then you'd put this in that field's Acceptable Values list:

Red;Blue;Green;Pink;Black

From there on, anywhere you can set the field in MC, these will be the only "choices" possible (it won't let you type in the box to add new values anymore).

So, the idea would be with this, you'd set it to whatever Processors you have defined in MCAutoQueue (probably plus Failed, though this really isn't necessary).  The string to use with my example set of Processors is:

Failed;Example Normal;Example Series Argument;Example Timeouts;Example No Update

Thanks for writing this

No problem.  I wanted it, badly, and it seemed like a good candidate for something that didn't have to be built into MC itself.

Also, I still have the file, not yet installed.
Do I put this in its own folder or does it go to a specific folder.?

I'd maybe hold off right now.  The currently published version doesn't work with files that have spaces in their names correctly (which is pretty bad), unless you make a custom field and add the quotes yourself.

I have a new version almost finished that fixes that problem, and adds a much better way to define the Processor Argument (the command line sent to the application you have set up as the Processor).  But I've been stalled by futzing around with the new builds trying to help them get their COM Automation stuff working again.  Oh yeah, of course, it also doesn't work with build 18.0.90 and later (though JRiver is working on it).

I'm planning to have this new version out by the end of this weekend.  It just needs a bit more work.

Then, assuming that works well, the next step will be to build a couple of wrapper Processors for HandbrakeCLI and for MKVMerge.  For these, I'm not sure if I'll be writing them as VBScripts or as actual applications.  I'm now leaning towards writing a couple small command line applications.  For any of these that I release, though, they'll be open source, so that people can write their own easily (and, hopefully, make them way better).

[glynor]

0.8.1 -- Internal Build

0.8.2 (12/09/2012) -- BETA

http://glynor.com/files/MCAutoQueue/MCAutoQueue-0820.zip

1. Changed: Various tweaks to the interface (button alignment, window sizes, etc).
2. Fixed: Settings button properly enables and disables when appropriate.
3. NEW: Argument can be complex string containing MC fields that are evaluated and replaced with the specifics from each file (see below).
4. NEW: Argument formatting has two possible modes: default and MC-Style.  In MC-Style, the argument is processed by Media Center, and allows you to use full Expression logic.  However, you must manually insert quotes where appropriate when using this mode.
5. Fixed: Argument properly quotes fields when they contain spaces (in the default mode).

Known Issues:

1. Incompatible with MC 18.0.90 due to problems with MC's COM Automation interface (JRiver is working on this issue).
2. Validation of the aqconfig files is early.  Manually edit them with care.  If you run into trouble, delete your MCAutoQueue.exe.config file in the installation directory, and it will reload the defaults.
3. Error reporting is weak in the UI.  The log is pretty good though.  If you get any general exceptions or other unexplained errors, please retain the logs.

[glynor]

MCAutoQueue now uses a parsed Argument for each processor.  The Default configuration gives examples of usage, but it is essentially as described above.

The Argument can be built via two different modes now:

Default:

Fields are denoted via the regular [Field Name] notation.  Any valid Fields in MC are processed and converted to their results from the file in question.  Fields from MC that contain spaces are automatically wrapped in quotes.  If the file doesn't have a corresponding tag, the [Field Name] token is removed, but the rest of the argument is left intact.  Any items in brackets that do not correspond to a valid field in MC, are ignored and passed through.

Example:

Code: [Select]

--someOption --series:[Series] --inputfile:[Filename]
MC-Style:

Media Center is used to perform the field replacement, using normal Expression rules.  This includes full expression support, so it can be used to format values appropriately (or anything else you can build with an expression).  However, because MC doesn't automatically quote fields, this means you'll need to manually add quotes where they may be needed.

MC-Style arguments also load substantially more quickly.

Example:

Code: [Select]

--prefix:"[Series]"-PadNumber([Season],2)ePadNumber([Episode],2) --name:"[Name]"

[glynor]

I have a pretty nice wrapper for VideoReDo started up now.  I can easily take an input TS file, quickfix the stream, and remux it to whatever format I want (and it'll recode if needed, but leave the stream untouched if it can, depending on the output format).

I'm working on connecting this to MC now.  The idea will be that you can add files to MCAutoQueue (probably via a Tag On Import rule), and then it will call this application which will do the conversion, and then find the original source file in MC, and redirect the file record to the new "version" automatically (optionally deleting the source file from disk when done).  This stuff is all working in my tester application, so now I just have to build the functions into something real, and design it in some kind of rational fashion.  One nice thing is that I've been able to do all of this from within the automation interface directly, rather than relying on the Sidecar files like my original VBScript did, so it'll work even if Sidecar support is disabled.

I'm hoping to also figure out how to add the new file to MC and stack it with its "parent" file, but I haven't gotten there yet.

The goal will be to build the "MC handling" part of this all in separate classes, which call a generic "processor class" of some kind, so that it would be easy to extend to make it work with other converters/processors like MKVMerge, Handbrake, MeGUI, and whatever (and audio-only related things too, which I haven't even begun to think about).

I thought about trying to write something truly "generic" that would just handle the MC part of the equation, and then let you specify the "processor" yourself as the user.  And, I might take a crack at that in the future, but for now... I think an integrated approach makes more sense.  The alternative means a lot of complexity for limited gain (for me), mainly because I don't want to do the "MC handling" part until I'm sure the conversion worked.  That makes it clumsy to make it truly generic (other than in the code itself).

However... If anyone has any concrete suggestions for how to work around this, I'm all ears.

[glynor]

I thought about trying to write something truly "generic" that would just handle the MC part of the equation, and then let you specify the "processor" yourself as the user.  And, I might take a crack at that in the future, but for now...

I think I came up with a good plan for this.  It almost certainly won't be "first", but it should be soon.

[glynor]

I made awesome progress on this over the weekend...

Good things are coming.

[glynor]

An update on this... I have all the back-end work done now, and I'm just wiring it up to an interface now.  I'm hoping to have something ready by next week sometime.

Basically, I've built a general purpose file ingester for MC.  It can ingest new files via a variety of means, including copying metadata from an existing file in the library (clone) and replace (which swaps the new file in as a stand-in replacement in the library).  I think it'll come in pretty handy for many people.

Then, the idea is to make a processor that does the file conversion, and sends the results to the ingester.  But, you could just use the ingester directly to accomplish other handy tasks (like re-ripping tracks and replacing them).

[glynor]

0.8.3 -- Internal Build

1. Changed: Extended beta timeout through the summer (expires on or after 9/1/2013).
2. Changed: Settings dialog resizes properly.
3. Changed: Logging improvements.
4. Changed: Internal changes to enable the MCFileIngester processor.

0.8.4 (02/02/2013) -- BETA

http://glynor.com/files/MCAutoQueue/MCAutoQueue-0840.zip

1. NEW:  MCFileIngester Processor included (see below).

[glynor]

This build includes a brand new processor that can be used to ingest newly created files into MC, while "cloning" or "replacing" an existing source file.  I'm going to call it currently an Alpha-quality release, but it is fully functional in all core ways, and the Ingestion process has been well-tested in all of the various modes.

Here's a screenshot of the current version:


The new ingester system is quite powerful.  It can be driven from the command line fully, and it has a number of flexible modes.

Basics:

1. You give it a source file to work with from MC's Database.  The Source File can be thought of as the "existing file" (typically the old one).  It will be the source for any metadata cloning done.
2. You give it a New File to ingest.  This can be set manually in Standard Mode with a path, but can also be calculated automatically using one of the various modes of operation.  Note that it doesn't matter if the New File has already been imported into MC or not (by Auto-Import, for example).  MCFileIngester will handle it properly either way.
3. You set the Ingest Mode:
     a.  Standard:  Normal mode.  The new file is selected manually.
     b.  Change Extension:  The new filename is generated by changing the extension of the Source Filename.
     c.  Stack Swap:  The new file has already been imported, was stacked with the source file, and set as the stack top (see below).
     d.  Using MC Fields:  All other settings are retrieved via special pre-defined fields in MC's Database.
4. If applicable to the mode selected, you choose an Ingest Type and set any relevant options.

Ingest Types:

1. Clone:  This ingests the New File as a new, separate entry in MC, and clones all of the existing metadata over from the (old) Source File.  The Source File is left untouched.
2. Stack With Source:  This ingests the New File as a new, separate entry in MC, and then Stacks it with the given Source File.  You can optionally clone the source metadata and add the new file as the Stack Top.
3. Replace:  This ingests the New File as a replacement for the given Source File.  All metadata (including [Date Imported], [Number of Plays], etc) is cloned over to the new file, and then the old file is removed.  You can optionally also remove the (old) Source File from disk when complete.

For the Clone Operations, you can optionally limit the metadata clone only to "Play Stats".  This includes the following fields:
[Date Imported], [Skip Count], [Last Skipped], [Number Plays], [Last Played], [Rating]

Using The UI:

The UI is there primarily to allow you to set defaults and global options, and to learn the Command Line Interface.  As you change options in the GUI, the Command Line Example generator updates to provide you with the equivalent command line parameters.  In addition to learning the CLI, you can also use the GUI to actually process new files, and to set the program defaults.

Using the GUI to set defaults will make it much easier to use the application without complex command lines if you simply want to do one kind of Ingestion over and over (or primarily).  You can simply open the GUI, set whatever settings you want to be used as the defaults, and then close the application.  It will ask you to confirm that you'd like to set new defaults.  Say Yes, and from then on, it will default to those settings, and you can omit the relevant command line options (except to override them).  The only thing it will not save is the Source Filename (which must be set for each file, obviously), and the New Filename for Standard Mode.

There are also a number of Global Options that can be set from within the GUI.  These are mostly self-explanatory, but I should explain the Show Progress behavior.  There are three Show Progress choices:

Always: Shows the Ingestion progress dialog whenever an Ingestion is running.
If Errors Occur:  If the Ingestion progress dialog is minimized (either manually or with the --runmin flag), it will be restored and shown if any errors occur.
Never: Always keep the Ingestion progress dialog minimized, regardless of errors.

These apply primarily when using the --autorun --runmin and --autoexit options.   Also note that the AutoExit option is only triggered if Never is selected as the Show Progress behavior if error occur during ingestion.

Using the Command Line:

Here is the basic information on the command line formatting.  The best way to learn the various command line options is to play with the GUI and observe the Command Line Example generator.  However, this is a listing of all of the command line options:

m|mode= - the Ingest Mode (Standard, ChangeExtension, StackSwap, or UsingMCFields) to use.
t|type= - the Ingest Type (Clone, Stack, or Replace) to use.
stackclone - Clone source metadata (for Stack IngestType).
cloneonlystats - Clone only stats when cloning metadata.
stacktop - Add new file as stack top (for Stack IngestType).
deletesource - Delete the original source file from disk when finished.
s|source= - the source filename (must be a full filename) to process.
n|new= - the new filename (must be a full filename) to process.
x|newext= - the extension to use to generate new filenames (for ChangeExt IngestMode).
e|autoexit - automatically exit after running.  This option is ignored if an error occurs and the relevant preference is set.

a|autorun - automatically run the ingester.
runmin - start up minimized.  This has no effect if autorun is not also set.


These last two command-line options are not accessible via the GUI.  They are best used when you have a "fully prepared" command line configuration, and where you want to run directly into the Ingestion process.  The Command Line Example generator adds these two flags when the command line is "ready" to be processed (it has enough valid options set to run).

The order that options are given on the command line is not important, though the options are case sensitive.

Options can be specified using any of the following notations:
-opt
--opt
/opt

Options that take a parameter (like --source or --newext) can be specified like any of these styles:
--source:"M:\Temp\script test\TheBigBangTheory-S06E10-TheFishGutsDisplacement-12851987-0.ts"
-x=mkv
/new "M:\Temp\script test\TheBigBangTheory-S06E10-TheFishGutsDisplacement-12851987-0.mkv"

The parameter must immediately follow the related option flag if the = or : notation is not used.

If any unknown command line options are passed, then AutoRun is disabled and the UI will alert you of the mistake.  However, note that any known flags set that are set, which are not relevant to the current Ingest Mode and Type are ignored.  For example:  If you provide --new:"c:\path\to\some\file.foo" but the current mode is set to ChangeExtension, then the given new filename path will be ignored and not used.

Change Log:

Change logs for the current release:

* glynor.common-ChangeLog.txt
* MCFileIngester-ChangeLog.txt

[glynor]

Additional details about the Ingest Modes:

StackSwap Mode:

StackSwap mode is a clever way of selecting through MC itself which file you want to use to "replace" another file in the Library.  To use MCFileIngester to replace files, it needs to know essentially these two important filenames:

Source File: This is the file you want to replace.  It has valuable metadata that you want to retain, but you want to replace this file with a new version or copy.
New File:  This is the new copy.  It has no valuable metadata.

StackSwap allows you to specify the New File visually, from within MC, so that you can make sure the proper files are matched up before running them through the replacement system.  You do this by adding the New File to a stack with the specified Source File, and then making it the Stack Top.

So, for example, if you have this file in your MC Library for some time:
M:\Audio\Music\F\Foals\Antidotes\2\01 - Hummer.mp3

And then one-day, you decide to re-rip this album to FLAC files.  Now you have new FLAC Files that might be somewhere like this:
M:\Incoming\Foals\Antidotes (Disc 2)\01 - Hummer.flac

And you want to swap in the FLAC version for the MP3 version in the Library.  You don't have to worry about the metadata or filename of the new file, you just get it imported into MC (which it already will be if you used MC to re-rip the file) and then you can do this:

1. Stack the two files together, and set the new FLAC version as the Stack Top.  It doesn't matter what metadata the FLAC version has, or what the filename is, you can just stack them together and make the one you want to "keep" (the New File) as the Stack Top.

2. Call MCFileIngester.exe in StackSwap mode, and give it the old Source File (the one with the good metadata that you want to keep, but get rid of the file) as the --source: parameter, like this:
MCFileIngester.exe --autorun --autoexit -m:StackSwap -s:"M:\Audio\Music\F\Foals\Antidotes\2\01 - Hummer.mp3"

This will find the file given, and then replace itself with the file currently listed as its Stack Top.  If any other files are currently Stacked with the given Source File, they remain in the stack (and the New File remains the Stack Top).

UsingMCFields Mode:

The special UsingMCFields mode allows you to specify all of the needed parameters for MCFileIngester to run, except the Source filename, from within the MC Library itself.  You specify these parameters by adding a set of special fields to your MC Library, and then setting these fields appropriately on the given Source File(s) to achieve the desired results.

This is best understood using the Command Line Examples generator in Field-Based style.  If you set the Command Line Example generator to Field-Based style, then it will list the proper command line to initiate UsingMCFields mode, and then provide the correct Fields and values that would need to be filled in MC to accomplish the same thing that is currently shown in the GUI.  Play around with it a bit and it should become clear what fields you need to add to MC and fill in.

You do not need to add the "full set" of possible fields to MC.  If you'll never use a particular option, such as [MCnet-DeleteSourceFile], then you don't need to create that field in MC.  The field names are hard-coded to keep me sane.

Here is the full list of special Fields, and what they may contain:

[MCnet-IngestMode] - String type field specifying the Ingest Mode: Standard, ChangeExtension, or StackSwap
[MCnet-IngestType] - String type field specifying the Ingest Type: Clone, Stack, or Replace
[MCnet-NewFilename] - String type field specifying the full New Filename for Standard Mode
[MCnet-NewExtension] - String type field specifying the New Extension for ChangeExtension mode
[MCnet-CloneOnlyStats] - Integer type field, 0 for false, 1 for true.
[MCnet-StackAddAsTop] - Integer type field, 0 for false, 1 for true.
[MCnet-StackCloneSource] - Integer type field, 0 for false, 1 for true.
[MCnet-DeleteSourceFile] - Integer type field, 0 for false, 1 for true.

Excluding Fields from Metadata Cloning:

When you use MCFileIngester to clone metadata from a source file to another file, it copies all fields (including custom ones) except the following stock fields:

[Filename], [File Type], [Media Type], [Volume Name], [Filename (name)], [Filename (path)], [Stack Top], [Stack Files], [Stack View], [Bitrate], [File Size], [Duration], [Width], [Height], [Date (day)], [Date (month)], [Date (year)], [Date (filename friendly)], [Dimensions], [Sample Rate], [Channels], [Bit Depth], [Compression], [Aspect Ratio], [FPS]

It also will not clone any read-only fields, even if they aren't listed above.  If you need to exclude any additional fields, you can add these to the IngesterExceptionFields.xml configuration file to manually exclude them from MCVideoRedoer and any other similar Ingesting Processor.  Please see this post for additional details.

Demonstration Video:

This demonstrates how to use MCFileIngester to replace a set of existing files in your Library with newly ripped duplicates, while preserving all of the original metadata (including Playlists, Date Imported, Play Stats, etc).


MCFileIngester - Standard Replace Playlist Demo
Click to play video.

[glynor]

0.8.5 (02/03/2013) -- BETA

http://glynor.com/files/MCAutoQueue/MCAutoQueue-0850.zip

1. Updated MCFileIngester to 0.7.1

MCFileIngester 0.7.1 Changes:

1. NEW: Implemented Command Line Examples generator.
2. Fixed: Tray Icon Ingest and Cancel Ingest Context Menu items work now.
3. Changed: Back end improvements for Playlist Source Type (not yet implemented)
4. Fixed: Source and New Filename browse buttons properly remember the last-used directories.
5. Fixed: Minor improvements to the defaults setting system.

[glynor]

0.8.5.1 (02/04/2013) -- BETA

http://glynor.com/files/MCAutoQueue/MCAutoQueue-0851.zip

1. Updated MCFileIngester to 0.7.2

MCFileIngester 0.7.2 (02/04/2013) -- ALPHA

1. NEW: Implemented Logging extensions (enables Log rotation and purging).
2. NEW: Implemented Playlist Source type.  Source can now be either a single file, or a Playlist of files in MC (for batch processing outside of MCAutoQueue).
3. Fixed: Row counter column on Ingest Progress dialog was not working properly.
4. Fixed: Improved performance of loading the Ingest Progress dialog.
5. NEW: With Source Type set to Playlist, the Source and New Comboboxes pre-fill with a list of all of the available Playlists in MC.
6. Fixed: Logging errors in Standard Replace mode.