This is a relatively new feature (implemented in MC20 or maybe late in MC19), and as is their custom, they implement new features with limited scope and improve them over time.
Why it wasn't done so that you can pick any arbitrary relational "key field"? I dunno. There may be an architectural limit in how the database works? I'm guessing here, but...
Perhaps the issue is that Fields in MC can be very complex beasts that are calculated and can reference other fields, which do their own expressions and reference playlists. Doing this and then iterating across all of the files in the Library (which may be 8 million rows) could cause all sorts of untold performance problems. Not sure. But, in any case, it is currently limited to just those three possibilities.
I would also like to see this extended to arbitrary fields, including user-created ones. If it has to be limited such that it can only be a "standard" (and not calculated) field, so be it. But I don't know if this is possible.