JRiver needs to know what it is to play it. I guess they could add support for a ".h264" file extension, but that isn't a real container type. JRiver is doing the right thing.
VLC might be looking at the compression type to play it I guess.
If the instructions say you can record in avi or h264, then the manufacturers don't know what they are talking about, or are hiding the fact that they can only record in avi format and no other format. Because avi is a container type. H264 is a compression type. Two completely different things. In fact, an avi file can contain media in the H264 compression format.
https://en.wikipedia.org/wiki/Comparison_of_video_container_formatsTry using "Media Info" to see what it says about the container type.
Or just try changing the extension to avi and see if it plays in MC. If that doesn't work, change the extension to ".mov" (Or less likely, .mkv) and see if it plays. (It would make much more sense that the camera could record in either avi or mov file format, rather than avi or h264.)
Or post a small sample for us to look at.