As commander of the southern districts set up after the war, Grant on several occasions toured the south. The business of segregating blacks was only just beginning in some areas, and South Carolina was one of them. One his trips there, it was not uncommon to find that Grant would take a seat on the train in the cars reserved for blacks. In the 1960's such folks were called freedom riders. This was unheard of in Southern society of the day.
Another small point about Grant: In the KKK crisis of the reconstruction, it was Grant who signed the KKK Act which put the KKK out of business as a visible power and forced it to go underground. Although this did not stop it, the Act had a very profound impact on it's course and effectively ended it's real overt threat of revolution. This act, later warm-heartedly gutted by SCOTUS, remained the legal precedent that allowed for the breakup of the KKK in the 1960's when it started to take to it's old ways of murder and arson.
Most of the hostility to Grant that you find today stems from the steps he took in his administration which no President until Truman had the nerve to even touch again. Grant's attitude to blacks and racism was extremely critical to our development into the country we are today, and it was based in large part to his feelings about how blacks had served their country loyally in the Union Army while many other whites became traitors to the nation.