I'm not an audiophile, but I like listening to the TV audio through my AV system. On my receiver, all roads lead to the RCA inputs. I currently am connecting from the headphone jack on the TV. I works, but I have to do a lot of tweaking with bass and treble to get it where I want. Would my other options be a better choice?
TV optical output >> converter >> Receiver RCA input
I don't believe it matters, because you're doing the same thing with both formats: converting digital signals to analog signals. What does matter is the quality of the DAC in the adapter/ converter. Compare the reviews on the various adapters/ converters, concerning sound quality, because they're not all created equal.
If the bass and treble adjustments are only needed when you use the TV then try an optical to analog converter.
If you use them all the time then it's likely you won't get a big improvement using the DAC.
If your AVR is analog only then replacing it with a used AVR that has HDMI inputs will probably be a better route. Even if it's 5-6 years old it will likely sound better than your old one.
Are we talking red & white RCA connectors or orange? If it's the analog red & white, it probably doesn't make a difference. If it's the digital orange connector, you'll want to use the optical and a non-destructive converter.