I think support of segregation is a moral issue, one that has caused pain even within my own family as family members have disagreed.
Your point that the victime of ill will and of a failure of will has been wronged in either case is true as far as it goes, but it doesn't go very far. The former, ill will, is definitely morally culpable, while the latter, while not praiseworthy, represents only a failure to adhere to an ideal superogatory standard.
If we compare a public school board of trustees in Atlanta, Georgia, in 1922 treating black students in an inferior manner by not spending as much per pupil and their successors in 2002 failing to enforce an orderly atmosphere, both are negligent actions. In the former case, no doubt the trustees believed blacks were an inferior race and thus it was considered wasteful to spend as much on their education as on white students. In the latter case, the trustees may believe that tough-minded discipline would be harmful to the psyches of students. In both cases, the trustees may believe their motivations are good ones, but the consequences in both cases are disastrous.