generally there are 3 ways to go about it:
optical, coax and hdmi.
hdmi is the only way you can get uncompressed audio.
optical can support compresed 5.1 or uncompressed 2.0
if your motherboard supports optical out it would also need to support DDL (dolby digital live), DTS-Connect or similar which essentially "compresses" the audio for use with optical. otherwise you'd be stuck with stereo sound only on optical and scratching your head wondering why it wont work. also, the receiver/amp you are using also has to be able to support 5.1 over optical. you'd need to read the manual to see if it does or not.
the best way is via hdmi if the set supports it. you can either passthrough the hdmi video to your monitor (on some models, this is the cleanest way) or just run another hdmi from your graphics card for audio (would show up as an extended monitor in this way but wont draw any additional gpu power)
everything depends on what connection types you have to work with. you'd need to specify.