You're right.
As a point of history, Romney has two polygamist great-grandfathers.
Miles Park Romney had five wives. He married the fifth, Emily Henrietta Eyring, after the 1890 manifesto. Polygamy was illegal in Mexico, by the way.
Another of Romney's great-grandfathers, Helaman Pratt, had three wives.
Romney's didn't get to choose either of those great-grandfathers. However, I've never heard him disavow polygamy, when both great-fathers practiced polygamy illegally in Mexico after the 1890 Manifesto, and one of them took an additional wife after 1890.
Joseph Smith, who slept with at least 11 women who were already married to other men and claimed them as his spiritual wives, tried to do the same thing with Romney's great-great grandmother, Sarah Pratt, wife of Lds "apostle" Orson Pratt:
From the article, Would you share your spouse with the Prophet?:
Orson Pratt's Wife, Sarah
"Sometime in late 1840 or early 1841, Joseph Smith confided to his friend that he was smitten by the "amiable and accomplished" Sarah Pratt and wanted her for "one of his spiritual wives, for the Lord had given her to him as a special favor for his faithfulness" (emphasis in original). Shortly afterward, the two men took some of Bennett's sewing to Sarah's house. During the visit, as Bennett describes it, Joseph said, "Sister Pratt, the Lord has given you to me as one of my spiritual wives. I have the blessings of Jacob granted me, as God granted holy men of old, and as I have long looked upon you with favor, and an earnest desire of connubial bliss, I hope you will not repulse or deny me." "And is that the great secret that I am not to utter," Sarah replied. "Am I called upon to break the marriage covenant, and prove recreant to my lawful husband! I never will." She added, "I care not for the blessings of Jacob. I have one good husband, and that is enough for me." But according to Bennett, the Prophet was persistent. Finally Sarah angrily told him on a subsequent visit, "Joseph, if you ever attempt any thing of the kind with me again, I will make a full disclosure to Mr. Pratt on his return home. Depend upon it, I will certainly do it." "Sister Pratt," the Prophet responded, "I hope you will not expose me, for if I suffer, all must suffer; so do not expose me. Will you promise me that you will not do it?" "If you will never insult me again," Sarah replied, "I will not expose you unless strong circumstances should require it." "If you should tell," the Prophet added, "I will ruin your reputation, remember that."
(Article "Sarah M. Pratt" by Richard A. Van Wagoner, Dialogue, Vol.19, No.2, p.72. Also see: http://www.xmission.com/~country/reason/spratt.htm)
The article recounts four other failures by Smith to recuit the wives of other men -- and five "successful" efforts.