You say you did a "Factory Reset" which usually allows for keeping your stuff, but have you tried a full "Hard Reset"? That will wipe the entire device of everything, but if the problem is software related it should fix it.
NOTE: Doing this will clear 'all' the data on your phone, so it is suggested you not choose this option unless you have to, or aren't worried about losing anything on the device.
Be sure you back up any data you want to keep, to a computer or cloud account, before doing the reset.
To "Hard Reset" the phone...
1. Make sure that the phone is actually powered off.
2. Remove your SD card, if there is one in the device.
3. Press the "Power" and the "Volume Up" buttons, together, until you see the start-up logo on the screen.
4. Once you can see the logo, release the buttons and allow the device to enter the "System Recovery Mode".
5. Next use "Volume Down" button to get to the "wipe data/factory reset" option.
6. When in recovery mode the "Volume Up" and "Volume Down" buttons are how to can navigate and highlight what option you need.
7. Once you have selected the "wipe data/factory reset", press the "Power" button to select that option.
8. On the following screen, repeatedly press the "Volume Down" button until the "Yes - delete all user data" option is what is selected.
9. Then press the "Power" button to hard reset the phone.
10. If you took your SD card out in step 2, when the device is up and running again, reinsert the SD card.
If you have done all this and you are still stuck in "Safe Mode" then you likely have a button that is stuck/getting stuck, which is keeping the phone from staying out of "Safe Mode".