I'm trying to re-encode a movie that is about 6GB down to 4GB, I want to keep the video bitrate the same, but knock down the audio from 5.1 to 2.0, and I went to use Handbrake to do this, but I wound up with a file than was larger (13GB) than what I started with? How did this happen? I can't use a program to enhance what already is there, right?
Lossless will basically make the file size larger (from an already encoded video) with no actual quality gain. Hence roughly doubling in bitrate and file size. Try skipping the re-encoding of the video and just change the audio settings.
Most common video codecs use lossy compression, meaning you lose some info permanently when it's compressed. Lossy compression typically allows a greater degree of compression (reduction in file size) compared to lossless. Basically you decoded/decompressed the video to original size (although whatever info that was lost during compression is still gone), and then re-encoded it lossless resulting in little to no compression. As said above, lossless encoding a file that has already undergone lossy encoding is sort of pointless.
Well, instead of Handbrake this time I used Super to re-encode. I set the video bit rate to approximately the same, and cut the audio bitrate in half (it was actually 2.0 to being with, not DD5.1). Super didn't re-encode it at exactly the same bit rate, it came in at about 1500 kbps less, enough to come in at 4GB DVD size, but I can tell a subtle difference in I guess what they call compression artifacts, so I think I'm going to stick with my original policy of retaining the best possible bitrate and just plunk down the extra 75 cents to burn it over to a BD-25.
You can tell Handbrake to encode to a target average bitrate (typically not that useful) or target file size. It will not match it exactly, but it's usually pretty close. A two pass encode will give you a better result, as will using an encoder preset that takes longer. SUPER is probably similar. They both use the x264 encoder under the hood (though SUPER has other encoders too).