Gen 2 (Sandy Bridge) is based on the older 32 nm process, while Gen 3 (Ivy Bridge) is based on the 22nm process. Basically, the wires got smaller so they could fit a bit more circuitry in there.
Comparing similar processors, you get a better integrated graphics chipset in the newer processors (4000 or 2500 vs. 3000 or 2000). Also, the power consumption has been improved a bit.
Bottom line, unless you're getting a really great deal on the older generation I would go for the newer Gen 3 chip.