That depends where you live. In California in the South Bay area (San Jose), most of the kids start as young as they can.
We didn't hold our son back. He's in 7th grade, his birthday is on Wednesday, and he'll turn 12. He was one of the smallest, but now he's not. He's started puberty earlier than some of his friends and now he's the youngest and one of the tallest of his friends.
My son is also gifted and does very well academically. He's currently taking 9th grade math. He is very happy that we did not hold him back.
Believe me, I know bright, mature kids like that too. But my sons are not on the large side and have late puberty in their genes (Dad) so I was OK with them starting a bit late. It didn't matter much since their birthdays are in February anyway.
But imagine if your son were genetically small and late with puberty -- it would mean two more years of being tiny past his friends looking like men. However, LOL, as my husband says, it really helps you develop your personality! (He eventually reached 6 feet tall, but started high school significantly under 5 feet.)