Remember that energy cannot be destroyed, only converted. So let's take a look at that bullet hitting a 100 kg target (in simplified terms).
The bullet eventually comes to rest, so all of the bullet's kinetic energy is converted into some other form of energy. Only some of that energy is used to transfer the (conserved) momentum to the target if it's not 'fixed'. So immediately you have your answer as to which is more important. The rest of the kinetic energy causes a lot of damage due to changing to heat, some sound etc., PLUS creation of momentum in all directions of parts of the target.
Also, do not forget that momentum is a vector, and kinetic energy is not. The energy of the impact of the bullet will cause some of the target material to disperse in all directions. These particles/fragments will conserve momentum too, so that a bigger chunk going off the right, might be balanced with smaller, multiple chunks with the same speed flying off to the left, or a single smaller chunk going to the left at higher speed, or any combination that conserves momentum along all axes. The part of the target that stays intact, itself, may not move that far in the direction of the bullet's trajectory, because of its mass. But the faster the bullet, for any given bullet mass, the more damage it will do in transferring kinetic energy (and infinite pairs of equal and opposite momentum vectors) to parts of the target.
In other words, the size of the impact 'crater', and hence 'damage', is going to depend a lot more on the kinetic energy (scalar) than on the momentum (vector), in part because the kinetic energy is not constrained in direction as it transfers and converts into other forms of energy.
I used to be a physicist, even went to Cambridge to do physics. I changed to focus more on Chemistry, and the biology, however, when I discovered that many of the Cambridge physicists emitted foul odours from their bodies, and I would never get a girlfriend if I continued to hang out with them.