Scalia was citing an argument that had been made earlier during oral argument.
Its the same argument that has been made by Thomas Sowell. Due to affirmative action, some minority students are accepted into schools for which they are unprepared. As a consequence, they fail. Whereas, had they attended a lower ranked school, they may well have succeeded.
The argument is not only a product of common sense, it is also a demonstrable fact.
At the same time, every affirmative action admission is keeping a qualified student from attending the school.
Accordingly, admission policies which feature affirmative action are actually unfair to two groups -- those accepted...and those left out.
I understand that now. He did not express that well at all.