There is something beautifully wonderful and holy about a man devoting the entirety of himself to the path of Christ. Some think it is too much to ask. I don't, but I could be wrong.
No, you are very right. The point is that the true calling to the celibacy is for the some, many men can be very good husbands, fathers and priests at the same time (and chose celibacy because it is the only way allowed at present). And those who have higher calling can be verified by a bishop and either become priest monks or be celibate priests.
This was the way before the Second Lateran Council in 1139 reform when the Pope Innocent II thought that making celibacy mandatory will raise the level of regular parish to those celibate ones. It did work to some extent then.
Since we deal with a different situation than what was in the feudal time (when the families of priests could try to seize property of the Church) which is more similar to the early Church when the Christians were a minority in a urban and decadent society maybe it is time to restore the ancient usage. To repeat, such restauration should preserve and elevate celibacy/monaticism as a more commited path (like marines compared to the regular soldiers). Bishops still would be recruited from the proven celibate clergy (same way as Eastern Orthodox Church is doing until this day).