The manual seems to show the mic is on the remote and is only active when you press search or google assistant. You could put a piece of tape over the mic to block it.
If this is not the case then check the TV setup menu. Tape will block the mic on the TV.