Well, here's why one fought:
"... a North Carolina mountaineer wrote to governor Zebulon Vance a letter that expressed the non-slave holder's view perfectly. Believing that some able-bodied men ought to stay at home to preserve order, this man set forth his feelings: "We have but little interest in the value of slaves, but there is one matter in this connection about which we have a very deep interest. We are opposed to Negro equality. To prevent this we are willing to spare the last man, down to the point where women and children begin to suffer for food and clothing; when these begin to suffer and die, rather than see them equalized with an inferior race we will die with them. Everything, even life itself, stands pledged to to the cause; but that our greatest strength may be employed to the best advantage and the struggle prolonged let us not sacrifice at once the object for which we are fighting."
-- "The Coming Fury" p. 202-203 by Bruce Catton.
Oh, don't forget this:
"Though I protest against the false and degrading standard to which Northern orators and statesmen have reduced the measure of patriotism, which is to be expected from a free and enlightened people, and in the name of the non-slaveholders of the South, fling back the insolent charge that they are only bound to their country by the consideration of its "loaves and fishes," and would be found derelict in honor and principle, and public virtue, in proportion as they were needy in circumstances, I think it but easy to show that the interest of the poorest non-slaveholder among us is to make common cause with, and die in the last trenches, in defence of the slave property of his more favored neighbor."
-- DeBows Review, 1861.
Or this:
"If it is right to preclude or abolish Slavery in a territory, why should it be allowed to remain in the States? The one is not at all more unconstitutional than the other, according to the decisions of the Supreme Court of the United States. And when it is considered that the Northern States will soon have the power to make that Court what they please, and that the Constitution has never been any barrier whatever to their exercise of power, what check can there be in the unrestrained councils of the North to emancipation?"
-- Robert B. Rhett
The record shows amply that the average southern white absolutely abhored the idea of black equality.
Walt
Most people, North, South, East and West, abhored the idea of black equality back then. There were the abolitionist pockets in New England, I suppose they can stand up and say "not us!", but even then, not all of them could be excluded.
The Revised Code of Indiana stated in 1862 that Negroes and mulattos are not allowed to come into the state ; forbade the consummation of legal contracts with Negroes and mulattos ; imposed a $500 fine on anyone who employed a black person; forbade interracial marriage; and forbade blacks from testifying in court against white persons.
Illinois, the land of Lincoln, added almost identical restrictions in 1848, as did Oregon in 1857. Senator Lyman Trimball of Illinois, a close confidant of Lincoln, stated that "our people want nothing to do with the Negro", and was a strong supporter of Illinois black codes. Most Northern states in the 1860s did not permit immigration by blacks or, if they did, required them to post a $1,000 bond that would be confiscated if they behaved improperly. Prohibitions against testifying against whites, voting, etc., were common parts of Northern black codes. Michigan, Iowa, and Wisconsin were not much friendlier. As a result, the black populations of the northwestern states never exceeded 1 percent.
Northern newspapers were often just as racist as the Northern black codes were. The Philadelphia Daily News editorialized on November 22, 1860, that the African is naturally the inferior race. The Daily Chicago Times wrote on December 7, 1860, that nothing but evil has come from the idea of Abolition and urged everyone to return any escaped slave to his master where he belongs. On January 22, 1861, the New York Times announced that slavery would indeed be a very tolerable system if only slaves were allowed to legally marry, be taught to read, and to invest their savings. Just a few examples.
Northern blacks also became the frequent targets of mob violence. Whites looted, tore down, and burned black homes, churches, schools, and meeting halls. They stoned, beat, and sometimes murdered blacks. Philadelphia was the site of the worst and most frequent mob violence. City officials there generally refused to protect African Americans from white mobs and blamed blacks for inciting the violence with their "uppity" behavior. (PBS)
Alexis de Tocqueville, from his book 'Democracy in America': "The prejudice of race", he wrote, "appears to be stronger in the states that have abolished slavery than in those where it still exists."
The resistance of Northerners to arming blacks was well documented before and after it was done. In fact, it infuriated many Notherners and cost Lincoln a significant part of the military vote in the 1864 election. Not all were against it, just as all Southerners weren't against arming blacks in the war.
Most Americans, North and South, abhored the idea of black equality back then. An unfortunate, but true, aspect of America's history.