Hard disk .

Definitely, depending on the version you want you can buy either a disc or usb. The only one that ships natively on a USB is Windows 10 but you can also download the image files online for other versions (7, 8, 8.1), image them to a USB, install the OS and then buy the registration key for activation later.
 


it may do

or depending on the motherboard involved you may need to either go in the bios and set usb as 1st boot device or press the appropriate key at start up for a one off boot order change

 
You should just be able to plug it in. Since you dont have any other OS on your HDD or the USB, so your PC should recognize that and just boot off the USB without you having to give it boot priority.