Lee described himself as not wanting to make war on his neighbors. It's also significant how important it was, in those times before the nationalization of everything, that your first loyalty was to your State. After all, the union itself is an arrangement constructed by the States–not the other way around.
Honor is at issue either way, which is why in charity we have to grant to Lee that the decision was legitimately difficult. It's impossible to read a biography of him, long or short, and conclude that honor wasn't important to him, or that he wasn't a man of virtue. That's probably why he was revered throughout the world after the war.
Bottom line is he did it .... he took an oath at the academy ...