Note: Unidirectional HDMI in to 5 RCA out. This cable don't own the chip to convert digital signals to analog signals, so it only fits for the devices which itself can convert digital signals to analog signals, for example SET TOP BOX (STB)
If both input / output device do not support signal conversion, a signal conversion box is required. HDMI to RCA cable is designed to connect and transfer signals from HDMI port to RCA port. This cable functions as a signal transmitter, but NOT a signal converter. As HDMI carries digital signals while RCA carries analog signals. In order for these two formats of signal to communicate with each other, both input and output devices needs to support the signal conversion function (code and decode), please kindly check with your user manual.
Because it doesn't convert digital signals to analog. So unless your ser-top box does that for the HDMI output, it won't work. If you get a good display from an XBox360 using the component connectors, the question is why you don't get a similarly good display with your set-top box. That sounds like a limit of the box or something that you need to configure correctly.
This is just my opinion - with no knowledge whatsoever of what hardware you have. You are welcome to buy the cable and try it.
It's possible that someone could give a more informed answer to your question if you told us the make and model of your set-top box and your TV. In the interim, I suggest that you read the manuals for each to see what configuration options there are.