Yes. they make a big difference from each other. A dual core with hyperthreading (2 cores, 4 threads) will be slower than a quad core with hyperthreading (4 cores, 8 threads).
Honestly, if you're trying to cut cost of that rig, then go for an i5-6600K. For games, there's not a gross performance advantage of having the i7 over the i5, but your i3 will last you a lot less time, especially for a rig that you're throwing a GTX 980 into.