It is possible the charger just isn't aligned correctly when first put it. It is also possible that the problem lies in the cord. They can easily be damaged.
The only way to know for sure if it is the charger or the port is to try a different charger. Granted it would need to be an actual OEM charger, as Samsung devices can be a bit moody with other chargers (and other chargers can be cheap and problematic). If the new one works fine, then the charger was the problem. However, if it has the same problem, then it is the port.
Additionally, just because you see no damage in the port doesn't mean that there isn't damage inside the phone where the port connects. Which you would not see. A loose connection perhaps. If the change in...