Refresh my memory. Had you told the pastor not to bring this up with your husband?
Stopping by unannounced and uninvited, and questioning him like that, when he had never even met your husband, is pretty cheeky. For whatever my opinion is worth, I'd be looking for a different church.