Off the cuff quotes are nice to have, but until the church canonizes it it is just an opinion. I don’t think you want to be responsible for everyone’s opinion in your church either. I don’t let myself get exercised over talks given years ago that were never canonized. Heck, the Cherokee indians had slaves and plantations. I, myself, would rather own a judge than a slave. I think everyone should own a judge. The Clintons seem to own quite a few as they are not pounding big rocks into small rocks despite using government agencies against their opponents like the IRS. I bet there were lots of protestants and catholics that owned slaves and thought it was ok. I think on a sheer wait of numbers the protestants and catholics were bigger into slavery. We were a tiny group back then.
A few comments:
1) Statements and writings from a Mormon Apostle, from a past President of the Mormon Church, and from Brigham Young don't exactly count as "off the cuff" quotes, in my book. They sound like they're reiterating serious, even if "non-canonical" statements of belief.
2) I'm sure there were a lot of Protestants and Catholics who owned slaves too. But most of them stopped considering blacks as spiritual second class citizens before the 1970s.
3) While I (obviously) disagree with Mormonism, I think Huckabee is way off the mark and is really shooting himself in the foot with his attacks on Romney. Theology isn't really pertinent to the Presidency.