I had the same issue loading a Library on my Sony TV. It is an Android TV navigation issue. I figured I was just the only person that didn't know how to navigate Android TV using an Infrared Remote. ?
As I was only using the TV's infrared remote control because I don't have a keyboard or mouse connected to the TV. The solution for me was to press the Back button on the remote once I had entered the password using the onscreen keyboard. That left the focus on the Cancel button, but I could then press the right arrow on the remote to select the Confirm button, then OK on the remote to confirm.
Note also that pressing the Up arrow on the remote after using the onscreen keyboard to enter the password acted in the same way as the Done button. It cancelled the loading of the Library. Although it did look like it was trying to load the Library as the spinning icon sat there spinning until I backed out of the load function altogether. I suspect that could be improved.