Done some more monitoring, take a look at this:
https://gyazo.com/307dcd96ea7a79cd0d54b12d96e85a2c
When the graph is dropping is when I get the fps-drops. Graph BEFORE is when fps is normal, graph AFTER is when I quit the game.
I am not that into graphic cards, so what does this mean...