provided your portable devices are driving them just fine as is, no there is not a big difference between using a strong amplifier with them and using the device. the only gains would be equalizer functions or bass boost although on most portable devices like phones, ipod or similar you can do this right on the device which makes it not worthwhile.
the fiio e6 has a 3 setting equalizer built in and is cheap and can certainly power them well enough. is it worthwhile? i would say only if your devices are somehow struggling to power the m40x or if you're finding you wanted an 3 choice eq.
source? i own the m50s (previous version of the now m50x which is the better version of the m40x) and have used it for years both amped and on multiple...