If I recall correctly, the State of the Union speech has not always been a prime time TV spectacle. If I recall correctly some presidents have delivered a hand written letter to Congress, but didn’t show up in person to deliver a speech.
This is something which has evolved into a prime time speech each year, but there’s no reason it has to be that way. Considering who the current president is, I won’t be watching anyway.
Since we’ve been in biz for 230 years or so, you can bet it wasn’t always broadcast in prime time.