it's very hard to say for sure what is causing the noise, however you could try changing a couple of things.
Try turning off microphone boost, this can be found in your 'recording devices', under your mics 'properties'.
From the same menu, you could also try a different volume, try turning your mic down a little from this menu.
the software you are using might be using an 'automatic voice gain detection' setting, turn this off to avoid the microphone increasing its sensitivity automatically.
Even trying a different USB port could be useful, as some of the noise is very possible electronic interference fro your USB port, this will be almost impossible to eliminate.
and finally, and probably the hardest one to test. Try a different...