There is no proof Grant ever owned more than one slave, who he freed in 1859, at a time when his family was in severe financial difficulty.
Julia Grant may have owned several slaves, or they may have belonged to her father, who lent them to her.
According to folks that do the historical research at Monticello, Jefferson owned 700 slaves over his life time.