That's true when the changes are just allele changes. It's also true when genes are inserted or copied within existing chromosomes. But somewhere along the line, chromosomes are duplicated or fused or whatever, changing the count. Otherwise, common descent is invalid.
There are (at least) two possible scenerios when this happens: one is that it happens frequently enough within a population that the "new" types find mates; the second possibility is that sometimes this just isn't an absolute barrier to reproduction.
If a chromosome split in half and had the same genetic material, you are probably right. There would not be much of a barrier to reproduction.