Math aside, why then don't we see a piece of the progenitors of either? We do. Here, for example.
There are a lot more "pieces" of DNA which can be traced than just the mitochondrial or Y-chromosome DNA, and each one gives different views into the big picture.
I worked with a guy in California who swore that he saw half-men, half-apes in Guyana just after WWII when he was being brought back to the States.