I’ve climbed lots of them.
People who are super sensitive to poison ivy sometimes have a reaction to the milky sap in mango tree branches and leaves.
I climbed one once, I was a youngster, head to toe poison ivey like rash, eyes swollen shut, the whole thing, pure misery for a few days. Doc told me to stay away from mango trees, I avoid them like the plague although I do enjoy a good mango every once in a while.
You must just be lucky.