Castration is different in horses than what you are comparing to in people. With castration, the testicles, the hormones and the ability are ~gone~ any urges that remain are purely instinctual or force of habit. In people, the hormones and testicles remain, they just cut the tubes... Function remains, there just isn't anyone on the highway. ;~D
It's not really common for geldings to go so far as mounting a mare. (though, if you remember a couple months back, Bay was doing just that, and I posted pictures of the spectacle) It is however, normal for geldings, as well as other mares, to show some interest and follow the mare around making curling lip expressions.
If he's mounting her, there is a little bit of danger if/when she decides she's had enough and decides to kick him. More likely, he'll lose interest before she does, when he realizes it's not getting him anywhere. I haven't noticed ours get ~as~ out of hand as they did the first heat cycle of the season. After that, Bay has given up, mostly. I'm guessing this is the first heat cycle these two have been together? Is it out of hand, or just curiosity?
Smokeys interest is not just interest, its the full blown act! We were told he was gelded but we may have to check? How? Sugar is in heat and they have been active at least 3 times. We moved them to another pasture and while they were still seperated they were very unhappy. They Love being togethera nd follow each other everywhere. (they now have the run of 4 fields!).