I suppose. But the American people really did not support the abolitionist cause in 1856 or 1860.
That's like saying the American people really did not support conservatism in 2008. It doesn't make it any less right. Frémont lived up to the creed that all men are created equal and that they are endowed by our Creator with certain unalienable rights. I will take a man like that over someone like James Buchanan any day.
Incidentally, I believe a Frémont Presidency would have advanced the cause of women's rights as well.