Maybe he was getting too many offers from Mormon women to be another wife for him.
Attempts to label him in any way as some kind of polygamist, or his religion, which excommunicates known, or admitted polygamists from its membership, are pretty lame IMHO.