Does the switch actually power down? That sounds like a defective switch. Try changing any video settings in the Netflix app.
If it doesn't happen when you connect the Xbox directly to the TV, with the same cable that the switch output is connected to the TV, then it's the switch. You can try turning off CEC in the TV menu.