I had almost the same problem with one of my televisions which only had optical out. Some do not have RCA audio out. Does yours? Because it sounds like you have the wires in place, so I am unclear.
If not, the cheapest solution would be to get a headphone to RCA adapter which would be annoying...