There are several glitches in such an application. First, slavery was abolished in those parts of the South controlled by the Confederacy by the Emancipation Proclamation. As Sherman marched through Georgia, for example, slaves were freed. That proclamation did not free slaves in the four slaveholding states that could not join the Confederacy, nor in West Virginia, nor those portions of Southern states, like northern Virginia, east Tennessee, and south Louisiana, under Northern occupation as of January 1, 1863. By January 31, 1865, when the amendment was sent to the states, the areas still under Confederate control were greatly reduced. By August of that year, Union control was effectively established throughout all the South. By August 1865, Lincoln's proclamation had freed over 95% of American slaves. Slave labor was not a major economic element in the Border states by 1865, nor in the portions of Confederate territory occupied by the Union as of January 1, 1863. What the amendment did was to codify into America's fundamental law the concept that slavery was forever illegal.
That the Border states like Maryland and Missouri and the most conservative Northern states like Ohio, New Jersey, and Indiana ratified this amendment would indicate that the state legislatures believed that the amendment would merely formally abolish what was, by August 1865, a moribund institution.
If segregation was a "badge or incident of slavery" in the South, then what about its existence in Northern states? Keep in mind that the 1954 Supreme Court ruling of Brown vs. Topeka Board of Education was based not on a Southern, but a Kansas school district. Many Northern states had Jim Crow laws; if they did not, they certainly did not prohibit discrimination in the private sector. Also, both law and social custom discriminated against American Indians and Orientals, races that were never subject to mass chattel slavery in America, in many Northern and Western states. Hispanics, Irish, Germans, Italians, Mormons, and Jews, though never enslaved in this country(except for Irish and British indentured servants in colonial times), nor discriminated against by law, were subject to widespread economic and social discrimination.
Additionally, the 13th Amendment has had little penumbrance. It does not forbid a military draft, nor other compulsory duties of citizens that had existed, in the interpretation of the Supreme Court in 1918, from time immemorial. Its extension to the area of private sector discrimination would not appear warranted.
If segregation was an incident of slavery in South Carolina, it would be an incident of slavery in Kansas even if Kansas had been a free state. If black folks could not get accomodations or vote or purchase property they would be free in name only. Segregation would be something that perpetuated the institution and hence reachable by the 13th amendment. Second, because the 13th is not limited to state action, Congress could restrict private behavior or state action.