That sounds very odd, only thing I would guess is the sound drivers are somehow causing some sort of high resource consumption upon disconnecting the headphones, have you already tried reinstalling them?
https://www.techpowerup.com/forums/threads/handy-software-for-reverse-stereo-in-windows.226167/
If your on Windows you can use this software. This will also works for anyone preferring to wear them backwards.