It's not quite that simple. Did your note one receive a carrier update to 4.1 or did you try to add it yourself? 4.0 is actually no less taxing on hardware than 4.1.
Its more than likely the additional bloatware Samsung installed through touch-whiz. I would suggest going to something like Cyanogenmod for a much smoother experience.