The reality show thing began around 1970 with a series on PBS. It was later parodied by Albert Brooks in Real Life.
I saw reruns of the PBS series in the 1990s.
Lets just say people have always enjoyed watching others in the act on “being themselves” Sound familiar? Candid Camera kicked off the medium years ago first on the radio(candid microphone) in 1947 and later on TV (1948) as we know it today. The documentary you are thinking of is called “An American Family”