You can trying going into "Settings" and then "Bluetooth" and either turn off the scanning (some phones allow it some don't) while leaving the Bluetooth itself on, or even try renaming your Bluetooth. If you do the second you will likely have to reconnect the devices you have connected, but it should stop the problem.
If by some chance it doesn't stop, then you would have to turn off Bluetooth when not in use.
NOTE: This usually happens when there is a device nearby that the phone is trying to connect to. So I would suggest checking that as well.