Okay... Here goes. These are my typical settings for encoding with AutoMKV. When I use more than one setting, I've tried to explain as best I can why I use one or the other. If I don't mention it, I leave it alone.
AutoMKV 0.74e (current build) settings.
Basic Settings tab:
Audio:
Nero AAC - Stereo
Qual: 0.5 (for main track), and 0.4 (for an additional track, if any)
This generally works out to roughly 130kbit AAC VBR.
-or-
Copy Original: to keep AC3 audio from a rip
Container, Codec, and Profile:
Container: MP4 - I generally use this as it is more widely compatible than MKV. (MKV has some extra fancy features, but I don't use them).
Codec: x264 - That's the reason I use AutoMKV. Other good options available in it are VC-1 WMV, but I haven't used it. XviD is also available, but I'd probably just use AutoGK for that.
Profile: depends on purpose...
CE-Highprofile.xml - General purpose high quality encodes for playback on my computer or HTPC.
CE-Quicktime.xml - I only use if I need to support Quicktime player, which I need to do sometimes for work stuff.
PD-iPod.xml - I don't actually have an iPod, but some of my friends do. If I needed to do an encode for them, I'd use this. Took me a while to get it working but I figured it out finally...
Generally, the profiles listed as HQ will give better quality than Highprofile, but they take an ABSURD amount of time to complete for a small gain in quality. CE-Mainprofile.xml is slightly more compatible with players, but really most anything that can handle x264 at all can usually handle Highprofile too (which gives better results) so you might as well just use that.
Resizers:
Resize Filters: Spline36Resize (neutral sharp) - the new default and works well. I used to use LanczosResize.
Width Resolution: Auto Crop Only almost always - I generally want full resolution encodes. If you set this manually, it must be a number evenly divisible by 8.
Filters: RemoveGrain(mode=2) looks great for most things. It does tend to make video look a little "cellulosy" (sorry, don't know how to describe it really), which is great for stuff that's supposed to look "oldish" or "gritty" anyway, like Band of Brothers or HBO's The Wire. For other types of movies I sometimes use Convolution3D (movieHQ). For cartoon sources (anime and south park style stuff) I use Convolution3D (animeHQ).
I haven't spent a lot of time experimenting here. I haven't tried much other than those I've listed above (and the different versions of those filters). Asking what's "best" on Doom9.org get's you yelled at, so I'm mostly flying blind.
Output File Size Options:
Generally I like to give it 1024 MB for a regular 90-ish minute movie.
For "special" movies, or longer films, I bump it up to either 1500MB or 2048MB. These are usually beautiful encodes.
For TV Shows, I usually give it 250MB for a 1/2 hour show and 350-450MB for an hour show. These too are beautiful encodes, so the movie sizes could probably stand to come down a bit. Just to explain, the number to the left of the disk size box will let you split the final output file into one or more separate hunks.
Advanced Settings Tab:
The vast majority of this stuff I leave alone.
Deinterlacer: Always Auto unless I happen to know it's a progressive source. Then I change it to None which saves some time in the encode process (it doesn't have to test it then). I generally do not check HQ Deint, which makes the deinterlacing a little nicer but takes way longer. Compare it yourself if you want.
Custom Matrix: These are special ways to improve the color fidelity of the encodes. They are built specifically for a particular codec. The user on Doom9 called "Sharktooth" created the eqm_avc_hr.cfg matrix which works very well for high-resolution (640x480 and bigger) encodes to x264. I almost always use it. However, as is noted above I just discovered that this breaks compatibility with Quicktime and iPod, so if you need to support them, you must set this to NONE.
Del Temp File: Check this to have it automatically delete the Temp directory after the encode completes. This happens automatically when you encode in batch mode.
Specify Bitrate: sometimes it's nice to shoot for a specific bitrate instead of a file size target (same goes for specify quality). This is a target, since it is a VBR encode and it won't be hit exactly.
Language: eng because I'm an American.
Generally, I leave everything else to the defaults. You can save and load different configuration files on the Advanced Settings tab as well, which is a good way to create different quick "setups" for different media types.
I have not yet used the Episodic DVD/IFO option, as I haven't had many episodic DVDs to do recently. I'll get into that rotten stuff soon enough though...