What you describe sounds like the mp4 has a multichannel audio stream and you are listening with 2 speakers or headphones.
You need to set your audio output to 2 channel if available. If you have 5.1, and still don't hear it, then you need to figure out what codecs are used and install them.
Without further info as far as player used, speaker setup ect. this is about the extent anyone can give you for suggestion.