Lee was not a slave owner, he wasn't even a landowner. The 'plantation' was his wife's and he abhorred slavery. He and General Grant were alike in many ways.
Too many times we look at history through modern eyes. The USA was a very different country back then. People need to understand that before the civil war one's loyalty was to their state and the state's common union, not the federal government. When the 'northern' states started changing the rules to intrude on other state's sovereignty the southern states seceded.
Lee did what he felt was right, just as many of us will. Unfortunately history is written by the victors and skewed by time.
“If strife and civil war are to take the place of brotherly love and kindness, I shall mourn for my country and for the welfare and progress of mankind.”
~ Robert E. Lee, 1860