The memory limit is 2 GB on 32bit and 16 GB on 64 bit (23.0.61 release notes), depending on how much memory is available on your system. I am not sure what it does if it exceeds those limits. There is also a limit of 30 tracks. If it exceeds the track limit, it drops back to normal playback. All of these limits changed when the new options were added and the Wiki, unfortunately, does not document the new limits and fallback positions.
You can watch the load with the task manager and see if it goes up and then comes back down before playing. If so, you reached a limit and it defaults to normal playback.
As I remember, when loading the full album into memory it is suppose to load the whole album before playing, so it should not play anything before loading. When loading a single track, it can start before the full track is loaded. Unfortunately, I do not currently have access to the system that I changed the buffering on, but that should not matter on loading a full album.
I would try loading single tracks into memory and watching task manager when doing that. And, watch task manager when loading a full album. That might give you some insight into the problem.