Absolutely correct : only way to get full menu support unfortunately. if and once MC has this covered as well (maybe in a pro version) we can get rid of PDVD's and TMT's of this world...
So to your initial question :
You need to point TMT5 to the containing folder and not to the bdmv folder or the index file in the folder. If you have this structure for example : my bd/movie test/disc1/bdmv you need to point to disc1. A way to do so is to :
1. select tools>options>library&folders>manage library fields
2. add new field (I named it TMTLauncher)
3. set the new field to calculated data and construct your structure line. Here you could enter expression filevolume([filename])/FileFolder([Filename],2)/ ... (it will depend if you want to use relative paths or not - (display this new field in your library and check the result) remember you'll have to end up with the path to the folder holding your bdmv and certificate folders.
for expression building consult :
http://wiki.jriver.com/index.php/Media_Center_expression_language.
4. once the result of your expression is OK return under file types (in the MC17 options menu)
5. select video bdmv
6. choose external player (custom) under playback method and point to the tmt5 player (you can also try to point to the uMCETMT player which opens fine and is better when using theaterview but you will freeze MC17/TMT5 when returning....so better use the normal TMT5
7. under arguments enter "[TMTLauncher]" (thats the field you created under point 2)
That's it !!! works perfectly. Seems complicated but in reality it isn't. Just make sure you got the path naming right.