It wasn’t just the New York bankers who were concerned about the money they were making off of the south. The owners of the textile mills in New England needed the cotton from the south and didn’t want that cotton going to England.
There is a club in Massachusetts called the Somerset Club. It is located directly across from the Boston Common and down the street from the State House. Their members were people with money, many of them mill owners, and they depended on southern cotton.
The Somerset members were definitely in support of the cotton growers. No surprise, northern mill owners had children working 12 hours a day in their mills. Tradition has it that the servants in the Somerset Club were required to close the velvet drapes at the Club whenever the Union Army recruits were marching on the Boston Common. That way the members didn’t have to be reminded of the war.
Of course, the North was full of people with strong economic or family ties to the South, who didn't mind Southern slavery and certainly didn't want to go to war against their friends & family.
And had Confederates not acted in the most uppity arrogant ways imaginable, those people would never have supported Civil War.
But Confederate slave owners could not be other than what they were, and so war became inevitable.