I wouldn't do that as speaker outputs can push a lot of power through that speaker wire, where as the RCA's are much more controlled. Turning things up could damage internal components.
I just noticed you said on both ends you'd do this. Power would no longer be a problem, so it shouldn't be...