People who owned "chattel" slaves were racists. They believed people of other races (black, Chinese, sometimes Irish) could be owned, but others could not.
"Irish" is a race?
BTW, the existence of black slave owners who owned black slaves tends to torpedo your argument rather effectively.
AFAIK, the Irish, at least in America, were never chattel slaves. Those that were “sold into slavery” here were according to the sellers, criminals transported for committing treason.
They would thus have even been subject to forced labor after the 14th Amendment was passed.
Not saying the laws under which they were punished were just, but they were the laws at the time.