What games are you trying to record?
Did you see this:
"Not every game will work with NVIDIA ShadowPlay by default. ShadowPlay only directly supports with games that use Direct3D, and not OpenGL. While most games do use Direct3D, there are a few that use OpenGL instead. For example, DOOM, which we used as an example above, uses OpenGL, as does Minecraft.
To record OpenGL games that don’t work with ShadowPlay, head to NVIDIA GeForce Experience > Preferences > ShadowPlay and activate the “Allow Desktop Capture” option. ShadowPlay will now be able to record your Windows desktop, including any OpenGL games running in a window on your desktop."
From this link:
http
/www.howtogeek.com/259573/how-to-record-your-pc-gameplay-with-nvidia-shadowplay/