So he's supposed to be a guardian for this virgin, and she turns up pregnant? Wouldn't you think this would offend his sense of propriety? Isn't it a tribute to his uprightness that he just says, "She's obviously going her own way. OK, she goes on with her life, and I'll go on with mine. No need to punish her, I'll divorce her quietly."
I could understand and respect that response. It's not til he realizes that the child is from God, that the way forward with Mary seems clear to him again.