If you turn on the "Hey Siri" function that probably isn't true. My wife enabled that function but it kept trying to answer the TV so turned it off.
LOL!
"Hey Siri" is only supposed to work if you say the words when you are connected to a power source. Is your TV saying those words. . . Without power source Siri is activated when you hold the home button for an extra long time.
I don't particularly need something to respond to my voice, so I have not activated it. I may give it a try to test and see if I have a similar issue as you describe, hoot.
Just turned it on. . . so we shall find out.