I don't believe he can use your "location history" to track you from your own phone. At least not by itself. That information is attached to your google account that you use on that phone. He could only access your account if you left the account unlocked/open. However his moving it on your phone (making your app not where it was) can't make him have access. If that is all he did, you just need to locate the app again.
If, however, you gave him access to your account, or he acquired the password, then he could make changes.
Or, if you allowed him access to the phone, and he put something on it he shouldn't have, then he could track it.
But, you can't 'move' where location history data is. It is always attached to your google account...