I'm surprised you even got this to work to the extent you did. VHS uses the old analog NTSC broadcast standard. Nearly all content today is broadcast in digital. The two are fundamentally incompatible. To get digital broadcasts to display on an analog NTSC device like a VHS VCR, you need one of those converter boxes that were initially sold when we switched TV broadcasts to digital.
It's possible your cable box or DVD/VCR has one built in. However, since your playback device is a digital TV (connected via HDMI), I doubt recording will work. An HDMI signal isn't just a straight video signal. It also has something called HDCP - high-bandwidth digital content protection. It's basically Hollywood's method of making sure you aren't recording the digital TV signal. For HDMI to work, the video source and display device
and everything in between must have HDCP. If something doesn't, then HDCP fails, and the video source will cut the picture. That's what your RDK-03060 message is saying.
https/www.xfinity.com/support/articles/x1-hdcp-devices
Basically, the DVD/VCR is somehow telling the cable box that you're trying to record. The cable box checks to see if you're allowed to record that show, determines you can't, and cuts out the picture. I wasn't aware HDCP signals could be sent over coax, but based on your description that's the only explanation that makes sense (you being able to view cable box -> DVD/VCR -> HDMI -> TV). I suspect even an old analog VCR wouldn't help, since it's the
lack of HDCP which is causing the failure. Maybe if you connected the coax out to a splitter, stuck one coax in into your DVD/VCR (since you know it supports HDCP), and the other coax in into an old VCR, you might be able to record on the old VCR.
This is one of the real shames of the conversion to digital TV. For decades Hollywood has been trying to kill the VCR. And with the transition to digital TV and HDCP, they succeeded. Yes you can still use a DVR, but those prevent you from making copies of the shows you've recorded. Heck, if the box fails, you can't even transfer the HDD to the new box. All the recordings are encrypted with a key unique to each box, so the new box will just see garbage.
https/www.forbes.com/sites/joshbarro/2012/01/18/thirty-years-before-sopa-mpaa-feared-the-vcr/#1450e1b85c8d