A lot of PC sources, such as games and music, are not pre-encoded, as would be the case with content on movie discs. That's probably where the issue lies. If you know that you are using pre-encoded source material, this would indicate that your audio device is not simply passing the bit stream through to your receiver but instead modifying it by down-mixing to the 2-channel stereo you're seeing.
Optical TOSLINK is only a 2-channel standard. That's simply the way it was designed "way back when." To achieve greater than 2-channel sound, engineers came up with clever encoding schemes that transmit all of your speaker data through those 2 channels, and it's then decoded by a compatible receiver on the other end. That you are getting 5.1 when you use your optical means some form of encoding has either been done on the source material and it's simply passing through your audio device on your PC, such as movies or TV broadcasts, or you have live encoding taking place on your optical output from your PC.
When it comes to a PC, hardware manufacturers can license DDL (Dolby Digital Live) or DTS-Connect, both of which perform live encoding on audio so that generated sounds, such as those in a computer game, standard music, or even Windows Desktop sounds, can be encoded into 5.1 channel audio as the computer plays them.
As your GPU has it's own audio device that is separate from the audio device on your motherboard, which also will be separate from any possible add-in-board soundcards, your audio hardware in your GPU would need the licensing for the live encoding software if you want to output using such a method. I have no idea how many games, or even if any, can output multichannel, raw PCM over HDMI, which would be necessary for multichannel if not using some form of encoding. Quite a few Realtek codec chips on motherboards come with a license for DDL, which is a nice selling point, so it's always possible that as part of the software installation for your motherboard, you have live encoding for your optical output there.
In Windows, you can open the Playback devices dialog box by right mouse-clicking the volume icon in the System Notification area, next to the clock. In the Playback devices dialog, open the properties for your particular device you wish to test, such as your GPU's HDMI audio output, and go to the tab page which contains tests for formats and bit rates. I would recommend testing Dolby Digital and DTS and seeing what happens. You can also test the various bit rates, but that's not really the focus here, unless of course somebody has chosen a bitrate that your receiver doesn't support. These tests are pre-encoded, and are the only pre-encoded in-built Windows tests, so should be the only ones you focus on to determine if you can bit-stream pre-encoded sources correctly.
One more thing to look at, have you configured Windows to know it's supposed to output 5.1 channel sound? By default, the speaker configuration in Windows will be stereo. I would expect most software which properly outputs multichannel sound will adhere to the Windows speaker configuration.