You shouldn't be able to put your feet down out of the stirrups and stand up. ;o)
Seriously though, it's a matter of personal preference. I've seen some seriously big people flying along on little Icelandic Horses (for heaven's sake don't call them Ponies or their owner will stoke out!), and they seem to do fine. They just look a little funny.
LOL!
Fjords are little too, but they're built like a Sherman Tank.
Funny story -- back before my husband turned up allergic to horses (and back before we were married), he started taking riding lessons (hey, he was courting me at the time, it was "love me, love my horse".)
He rode at the barn in NJ where our equestrian team trained. He was this huge big guy (6'6") on the biggest horse in the barn (half Belgian, half God knows what) in a class of little 8 year old beginner girls.
That was funny enough, but one day his class went on a trail ride, and this mean little gray pony named Lambchop decided to crowhop a little and his rider fell off. The little girl began crying and did NOT want to get back on Lambchop. The instructor looked around, but all the other little girls were afraid of Lambchop and they were hanging back. The instructor's horse was a wiggy T'bred and completely out of the question. So everybody looked at my husband . . .
You could hear them coming before you saw them because everybody along the stable road was laughing. We were up in the upper ring working, and here came this procession . . . this tiny little girl up on this Belgian, and my husband on a VERY unhappy Lambchop, with his feet dragging the ground. He could literally point his toes and make two drag marks in the dirt.
Lambchop probably thought twice before dumping any more little girls . . . at least while my husband was around!