And LDS historian Todd Compton would agree with you.
Compton has a 'work in progress' on his geocities page titled Mitt Romeny's Polygamous Heritage in which addresses what he terms 'confusion' and 'factual mistakes' about Romey's polygamous heritage.
Compton's footnoted research indicates that Romey is the product of two prominent polygamist great-grandparents, Miles Park Romney and Helaman Pratt. Romney's grandparents do not appear to have been polygamists.
For the benefit of LDS and non-LDS readers, you should note before reading Dr. Compton's work that although he is a 'faith-promoting' historian, he is not a strict one.
For example, Dr. Compton cites Fanny Alger as "one of Joseph Smith's earliest plural wives" in his award-winning book, In Sacred Loneliness: The Plural Wives of Joseph Smith (the Smith/Alger relationship is discussed on pages 25-42 and the quote comes from page 25), but the LDS Church does not recognize Fanny Alger as a plural wife of Joseph Smith.
The Foundation for Apologetic Information & Research (FAIR) also recognizes (and here) the 1833 marriage of Joseph Smith and Fanny Alger.
FAIR has a love/hate relationship with Todd Compton. FAIR cites Dr. Compton favorably on at least one issue regarding polygamy but also cautions against reading him in the same footnote, citing reviews that say he does not bring a "faithful" view to all of his writing. If you wish sources on FAIR's simultaneous citing and criticism of Dr. Compton, please contact me.
A guy named Smith did a FAIR '09 workshop called Everything You Want to know About Polygamy (But was afraid to ask)...he conceded toward its end that the "revelation" for Smith to engage in polygamy may have been as early as 1831...Joseph F. Smith said in 1878 that polygamy started 1832.