If conservatives don’t shun the part of the conservative movement that touts “white replacement theory,” then the association makes sense.
“White replacement theory” is a quantitative discussion.
The population numbers show that the black population in the country has not significantly increased in recent years.
So it is not a white/black issue.
I leave the rest of the discussion as an exercise.