I assume that you are connecting the phone to your PC to do that transfer?
The Handheld Sync function checks the phone internal memory/SD card for changes in files before it starts the sync process, so that it only syncs changes, and not all files all over again. MC is actually managing the synchronisation process, rather than just copying files. That includes deleting files no longer in the settings (via Playlists), updating files that have changed, and adding new files, but not re-copying existing files. Using Windows Explorer just copies over the files you choose to copy over, and you would have to do all the files/version/updates management yourself.
Because you have a lot of files on the phone, it takes a while to do the check, and update the database it stores on the phone. Back and forwards communication with a phone isn't all that fast.
You could speed up the process by removing the SD card from your phone, plugging it into a fast memory card reader and connecting it to you PC, and then using Handheld Sync to update the SD card.
Or you could sync to a location on your hard disk, which would be fast, and then copy the files to your phone, which would be fast, but unless you only copy files recently updated, may take longer than just connecting the phone and letting MC do its thing.