Slavery started in the 13 Colonies. At the time of the Civil War....Union Soldiers owned ~315,000 slaves. Confederate Soldiers owned ~200,000.
Slavery may have existed in the 13 colonies but it certainly didn't start there.