Your point is valid concerning married couples. It is not valid concerning Mary and Joseph, however.
In Scripture, it is often the seemingly insignificant word that has the most meaning. And at the end of Matthew there is a little greek word "eos" which is translated "until." In this particular case, it is eos ou, which according to the Greek Lexicon, when it is with the neutral relative pronoun ou or otou, it gets the force of a conjunction, until, till (the time when);...http://www.blueletterbible.org/cgi-bin/words.pl?book=Mat&chapter=1&verse=25&strongs=2193&page=1&flag_full=1
Thus, when one looks at the actual greek, there is not a question that Joseph knew Mary. Rather, the Bible states that he witheld from marital relations with her UNTIL (or up to the time when) Jesus was born. Mat 1:25
Incidentally, it also refers to Jesus as Mary's firstborn (prototokon) son. The implication is that there was also a second born. Since Jesus's brother's and sisters are mentioned in Scripture as well, it is within reason to take this to mean that at least some of these were Mary's biological children (when one accompanies it with Jesus being her firstborn).