It is too easy to sit back in our warm, comfortable, politically correct world today and pass judgment on men far better than ourselves. Before passing any judgment, you should carefully consider all the circumstances of the situation. In addition, you should know than many of the free soilers held their beliefs to live free of blacks.
Many free soilers helped escaping blacks migrate to Canada, thereby encouraging them to move north. The free soilers from the West (or with plans to migrate there) just didn't want to have to compete with slave labor enterprises. Do you fault them for that?