That’s interesting, I always thought the opposite.
Because there were later laws in Middle Ages regarding the types of clothing the classes could wear.
Freed slaves would wear a soft cap to show they were free but the still enslaved had to dress like everybody else. Just no Togas. You had to be a free born citizen to wear that. But since there were a whole bunch of people in Rome who were not citizens and lots of citizens did not wear togas unless it was a very formal occasion that was not a notable sign.