Answered my own question, here it is for any of you Truecrypt users in a similar position.
Copy ALL your data off of the drive, we're about to completely erase it! (I use SuperDuper to clone it to another drive so I can just clone it back once I've removed the encryption, but use whatever...