Usually, my philosophy is "bigger is better". However, smaller USB drives still serve a purpose. If you are using a flash drive for work and nothing else (spreadsheets, backing up emails, docs, etc.), a 16 or 32 GB+ flash drive is overkill. Only when you get into transferring large media files and backing up things like .isos, etc does a larger USB drive really serve a purpose.
Personally, I have a few different small USB drives that I consider to be "disposable", meaning if they are lost, damaged, or god forbid stolen, I wouldn't completely freak out. All of these disposable drives are 2 and 4 GB models, and are easily replaceable (as in they don't hit the wallet too hard).