It’s Brit slang. “toe rag” is a derogatory term used mainly for lower class, ill-behaved white Britons. It has no ethnic slant.
Thanks for that info!