The only way your phone can be tracked if is if there is software on it (app, malware, etc.,) for tracking, you are accessing, open WiFi, you are signed up for a service that allows tracking, or there is a piece of hardware that was placed in the device that is for tracking.
You can't be tracked by GPS by anyone without the above. So now if your location is showing on say your Google, well then you chose to allow that one. However, without the above, just having GPS on can't track your phone.