The SD card itself likely isn't the problem. However what you are running on the phone could be. Going by the info in your title, I am assuming you made the SD card "Internal" storage so you can run apps from it? If that is the case, and you are running more/installing more apps, then this would cause your drainage problem. Especially with some apps that are power hogs, and if you are like many who don't actually close an app when done, that will drain the power faster.
Try closing apps (rather than just going to another one or screen) and also restart (power off and then back) the phone regularly and see if that resolves the battery issue. If not, then I would try starting up in "Safe Mode" and see if the phone drains when not using the apps installed (most apps won't work in that mode) if it doesn't drain as fast in there, then you have something on the phone that is the cause. Bad app, virus, malware, etc., that you will need to find and remove.