That Lee believed that the Confederacy had only exercised its rights as guaranteed under the Constitution, defended by the founders, and invoked by states and statesmen "for the last seventy years," can be seen in his letter of 15 December 1866 to Lord Acton, in which he says precisely that. He wishes that "the judgment of reason" had not "been displaced by the arbitrament of war," but concludes it has been, and it is time for the South to move on, to accept "without reserve... the extinction of slavery.... [A]n event that has been long sought, though in a different way, and by none... more earnestly desired than by citizens of Virginia," and to "trust that the constitution may undergo no [further] change, but that it may be handed down to our succeeding generations in the form we received it from our forefathers."
As they were seeking English help, it was wise to speak so. The English weren't about to help the secessionist promote slavery.
It is too easy for Lee to side step the slavery issue by saying he was falling in with his family and friends on state's rights to practice slavery.
You can certainly count expansion of slavery in the west among the issues of secession which were vast and complex. You can also count on one hand the number of Union soldiers who went to war to "end slavery". Wasn't an issue save for a few abolitionists in New England.
The need for a powerful Federal government to implement a national rail system -- and the need for a powerful Federal government to maintain the commercial viability of the Ohio River Valley and the Midwestern states by ensuring free maritime access along the Mississippi River system -- were far bigger issues at the time than slavery was.