Unless there is a flaw in the device, the charger or the plug you have it connected to, it is highly unlikely that it would cause a fire or damage the device.
Most devices do not keep charging, even if you keep them on the charger, unless they are also draining the battery. Now if you left it turned on with apps running in the background, it would go between charging and discharging due to the running apps. If you left it turned off, then it should do the same/charge/discharge but at a much slower pace.
And unless there is a flaw in the phone, it is not likely it would be damaged. There is know way to know ahead of time, however, if there are any flaws in the device.
You could always go back or get someone to go disconnect it if you are that worried, but it would be more likely the battery would be damaged before anything else would.
Still, in the future, I suggest not leaving such devices connected when they don't need to be.