Could be one of several things really.
First and formost, make sure you are plugging into an optical input; very few motherboards have an input slot, although most soundcards do.
[Note: if you are only sending 2.0 PCM, this section does not apply]
Secondly, you can only get 5.1 over optical by using Dolby Digital/DTS (I think DD is the only supported mode on the 360), which needs to be decoded prior to being sent to speakers. Usually a receiver is used for this task, although various software methods (FFDshow) can be used instead.
Thirdly, you're PC needs to be set (Via Windows Control Panel) to use the sound input instead of the sounds nativly comming from the PC.