there are far simpler ways to get what you want.
if you're using a stereo headset and want 7.1 then just use virtual surround from a soundcard. dolby headphone or cmss3d/sbx would work fine.
if you're using a true 5.1 or 7.1 headset with the appropriate multiple 3.5mm plugs then using a soundcard with 5.1/7.1 capability is also the best solution.
why doesnt your hdmi->optical->mixer->headphones work? likely because your pc is not recognizing the destination as a 5.1 capable device. as far as forcing the output to 5.1 if the option is greyed out? besides writing custom drivers or using modified drivers someone else made.. i would say no. you could try to shorten the chain of devices to see which is causing the issue with your pc but...