Knowing and suspecting are two different things. He had no way of knowing but was suspicious That’s enough!
Don;t think he suspected either, at time of FIRST confrontation. Though, Brown didn’t know that.