Steam have a VR test program you can install and run that checks your hardware for VR ability.
You may be right on the margin, I was using a GTX980 and 6700K with my oculus rift and that worked really well but the recommended specs for both is about a 1060 or better and the 970 is a little under that recommendation.
I was (briefly) running my HTC Vive with a GTX 970 and an i7-5820K and it seemed to perform fine in most titles, Skyrim VR, however, may present some issues. Currently, I'm using an i5-4690K paired with a GTX 1080 and even that struggles to keep up with Fallout 4 VR at times. Keep in mind, played FO4VR on the 970 and haven't played Skyrim VR at all but I assume the graphical demands are roughly the same given the open world nature of the games.
These are likely, some of the most demanding VR games on the market at the moment and as such it's no surprise that there may be issues running them on lower end hardware, many other games like Gorn, Job Simulator, Budget Cuts, and Beat Saber are likely to run with relatively few problems, provided you lower the graphical settings a little.