First off, what player are you using for your videos? Have you tried another player and checked if it does the same thing? It might just be the software :)
Second, do other movies (or even audio) experience the same behaviour? (e.g. CD's, DVD's, Bluray)