dont use device manager
what you need to do is
The windows 10 have a policy to update automatically all the drivers, you need to disable this in gpedit.msc
this link will help
https://www.howtogeek.com/302595/how-to-stop-windows-10-from-automatically-updating-hardware-drivers/
Ok, windows will...
check your temperature while gaming, use gpu-z or msi afterburner (but dont activate any options for overclock, just monitoring)
if the system overheats it will decrease the gpu core clock and gpu memory clock and thats send your fps way down