If you have already installed MC26 as you suggest, then your library will have already been ported from v25 to v26 by the installer. You have already upgraded. You no longer require the The library, or anything else to do with older versions in the appdata folder hilighted by Jamil. So, yes, remove the library
The library used by MC26 will remain untouched.
The reason I suggest reinstalling MC26 afterwards is that sometmes, especially with things like 3rd party plugins, uninstalling the older versions can break things in MC26, which are corrected by a reinstall.
-marko