Actually, that passage says that sin entered the world through one man (Adam) and death through sin and that death came to all men because all men had sinned.
My NIV says this for example:
Rom 5:15-19 : 15 But the gift is not like the trespass. For if the many died by the trespass of the one man, how much more did God's grace and the gift that came by the grace of the one man, Jesus Christ, overflow to the many! 16 Again, the gift of God is not like the result of the one man's sin: The judgment followed one sin and brought condemnation, but the gift followed many trespasses and brought justification. 17 For if, by the trespass of the one man, death reigned through that one man, how much more will those who receive God's abundant provision of grace and of the gift of righteousness reign in life through the one man, Jesus Christ.
18 Consequently, just as the result of one trespass was condemnation for all men, so also the result of one act of righteousness was justification that brings life for all men. 19 For just as through the disobedience of the one man the many were made sinners, so also through the obedience of the one man the many will be made righteous.
I just don't see any reasonable reading of this other than Adam's sin had real repercussions for the rest of humanity. The distinction of Adam's sin was WAY MORE than that it was simply the first. Do you believe in the concept of original sin as being bigger than that it was simply first?
The whole "it's a genetic condition passed down through the male" is a more or less parascriptural gloss. It was taken to an extreme about 30 years ago by the Institute for Creation Research. Henry Morris reasoned 1. since the sin nature was passed along through the father necessitating a virgin birth, 2, since the Lamb of God had to be completely perfect in every way, and 3. since Mary herself, as part of the human race, took part in the corrupted physical nature of fallen flesh and couldn't be the physical antecedent of the perfect physical nature of Jesus any more than Joseph could, the physical, human body of Jesus, therefore, had to be an ex nihilo creation of a human embryo within Mary.
But #1 cancels out #3. If the sin nature is passed down through the male, as I believe, then Mary's sin nature would be irrelevant to Christ, just as my wife's sin nature is irrelevant to our children. So I say that Jesus had Mary's DNA flowing all through Him. He was fully human and did not have the sin nature.
Well, it says here clearly that "one act of righteousness was justification that brings life to all men." That is exactly the Apostolic view. God made it possible for all men to come to Him, as He desires all men to be saved. The only reason this doesn't happen with all men is because God gave man freedom to choose.