You should be able to go into "Settings" and then "Bluetooth" in there tap the device you are trying to disconnect from and then tap "Forget".
That should disconnect it. If it gives you trouble doing so then try turning off the speaker so that it isn't able to connect, turn restart the phone (turn it off and back on) then try turning on the speaker again. It may just connect. If not, then use the "Forget" option.