Red October fully supports all common media containers and codecs (those without DRM, anyway), certainly including H.264 MKVs.
I'm going to guess that you did the conversion of the file directly in a folder watched by MC's Auto Import system. When you do this, MC will try to import the file "before it is done", and eventually MC will decide the file is just bad and ignore it. It is supposed to recheck the files if they change, and mostly it does, but generally it is a bad idea to do conversions directly in a watched directory. Handbrake conversions work fine if the temp files are stored elsewhere and then it moves the completed file to a watched folder when it is done, though.
I've
explained before in detail how to get files out of the Bad Files list, once they're in there (that first post is related to the deleted items database, but read down further and I get to the rest of them, including the Bad Files database).
