For light use like music playback, you would probably get by fine with just sticking on some heat sinks. Its really sad that they don't include those yet since that would make a much better product (and on the 4 they are almost mandatory to reach anything close to specified performance), but you can get decent ones for a buck or two on the web anyway.
I agree, I have the Pi4 4GB model with the firmware revision, currently it idles at 54C using stick on passive heat sinks for the CPU, USB controller, and Ethernet controller chips.
Certainly heavy lifting tasks such as 4K video will require a fan to avoid throttling, but for music server purposes I think the passive cooling will do just fine. While 54C is significantly warmer than the 3 or 3B+ boards idle temps, it isn't a problem, throttling occurs at about 81C.
Once IdPi is released for the 4B board, I plan to use a case that is itself a heatsink, the KKSB aluminum, though I believe it only touches the CPU itself, so the other passive heat sinks will remain in place.
The bulk of the heating issue was initially due to the USB controller running downright HOT even with no USB devices attached. That was corrected via firmware update early on.
Still, there are no free lunches, the faster CPU and separate USB and Ethernet controllers along with additional RAM on the 4B will result in more heat, but it's manageable without a fan in most cases unless the ambient room temperature is already quite high to begin with.