The issue isn't that it tracks where you go (all modern cell phones do this, smartphone and dumbphone alike). The issue is that it stores all of this information in a plain text file, and not only that, but also spreads it all over the place.
If you ever lose your iPhone, or your iTunes gets hacked, or your MacBook gets stolen/hacked, all a would-be thief has to do is open that file and he'll know exactly when you'll be out of your house, and for how long, perfect for a home invasion.