The Smart's were OUT OF THEIR MINDS with worry. He seemed like the most natural suspect. However, after awhile, even they knew that maybe he wasn't the one and begged the police to follow other leads. I mean really, do you blame them? They were only trying to find their little girl.
Plenty of other people have been "out of their minds with worry" but they didn't take the path that the Smarts took. Wiser people know their limitations.
If they didn't trust LE to find the perp, they certainly could have hired private detectives, or taken on Dr. Henry Lee much earlier than they did. (and btw, where is the esteemed doctor?) They could have allowed a composite drawing. Uncle Tom nixed that idea, pronto.
We'll never know the entire truth in this case.