XLR and RCA are both analog signals, transferring the full waveform. Optical and coaxial are digital and only transmit the sound as 1s and 0s and require the signal to be decoded at the other end to waveform (analog) format. They are not directly comparable as they are different transmission types, but digital is less prone to errors/interference, and optical is not affected at all.
Personally I am not a fan of studio monitors. Mounting the amp on the speakers is just another source of cabinet resonant noise. I like to keep my amps out of sight where I can't hear them, heh.
The kind of subwoofers that monitoring companies sell are designed mainly for rock/pop music and have very little output below 40hz. If you listening to movie scores, orchestra music, etc, you are missing out on a large part of the subsonic range. To me those aren't even true subwoofers, just lower mid-bass modules.
Also, I prefer to listen to speakers at a distance, where the left/right channels will intersect (this is what they call a soundstage). Listening to monitors nearfield is basically like listening to headphones--it's more like "dual mono" rather than stereo. The one advantage to nearfield monitors (like headphones) is that the treble is aggressively rolled off to reduce listening strain. But meh, I'd rather listen to all the music personally, all the treble (and bass included) even if it is a little more fatiguing.
My personal setup is a Rythmik F15 subwoofer with a Paradigm Signature S2v2. Gives me flat response from 14hz to 45KHz +/- 2db. I have the bookshelves on a pair of sturdy 40lb steel stands and sit around 5.5 feet away from them. Studio monitors would start aggressive dropping output above 16KHz, and the subs take a dive around 40hz...not my cup of tea--it's great for most music, but I like a lot of movie scores (and movies...), and the impact isn't there without a extended, linear frequency response.
My Rythmik replaced an Infinity Cascade 15...the Cascade was no slouch, it weighed 100lbs, was a sealed sub with an 800W amp and a 4x6x6 array of drivers. But it had no output below 32hz and I just didn't feel the impact. (Just fyi, the cascade retailed for $1,500...the rythmik was much cheaper so it's not like more $$ = better)...I'm not trying to push the Rythmiks either. Elemental Designs, Hsu Research, and SVS have just as good of a pedigree and I would recommend any one of them over a "monitoring sub".