Untrue, having "a slave" anywhere was not dispositive of the question of slavery's acceptance -- it certainly wasn't true in Kansas or California. The Southern States walked away from the territories north of the Missouri Compact line, and Kansas and California, when they seceded. They might realistically have entertained some thoughts about New Mexico and the Indian Territory becoming part of the Confederacy eventually, but Colorado became a State during the war, and California and Oregon were already States -- Oregon having been populated by Northerners mostly (as its voting patterns still show today -- one of the things realized by Richard Nixon when he reviewed young Kevin Phillips's map of national electoral patterns, which he compiled as a teenager).
Slavery, as the document I quoted shows, was excluded from California by freesoil interest through devices very like those complained about by freesoilers in Kansas, when the Lecompton convention sat. Of course, we don't hear about Southerners' having received a raw deal in California, which was, as Rhett says, very well suited to plantation agriculture -- or any other kind of agriculture, since it has become one of the breadbaskets of North America.
Nevertheless, the Southerners were successfully shut out of California by a cabal, which obtained California's admission to the Union as a freesoil State. And so California was not on the table when the Southern States left the Union.
The Southerners were, I'm sure, perfectly aware that they couldn't just claim large territories and walk off with them, without some sort of resolution of the Territories' status between the Confederacy and the United States. But the Confederate commissioners attempting to discuss issues created by secession were repeatedly rebuffed by Lincoln, and so I'm sure it became pretty obvious that, the United States being on a war footing, there would be no negotiation where the Territories were concerned.
The secession convention delegates had to be aware of this possibility, that secession might mean the abandonment, in practice, of all access to the Territories. But they went ahead and voted for secession anyway -- truncating that leg of your "it was all about slavery" argument.
Why would Lincoln consider serious negotiations with criminals ("Confederate commissioners")? You would have us believe that the southerners meekly gave up on their expansionist desires. Nothing could be further from the truth. The expansion of slave territory was a southern article of faith.
"If, by your legislation, you seek to drive us from the territories of California and New Mexico, I am for disunion" - Robert Toombs, 1850
"We ask you to give us our rights [in the territories]; ... if you refuse, I am for taking them by armed occupation." - Albert Gallatin Brown (1850)
"I want Cuba, and I know that sooner or later we must have it.... I want Tamaulipas, Potosi, and one or two other Mexican states; and I want them all for the same reason - for the planting and spreading of slavery." - Brown (1858)
"With Cuba and St. Domingo, we could control the production of the tropics, and, with them, the commerce of the world, and with that, the power of the world" - Southern Standard
"We are playing for a mighty stake. The game must be played boldly.... If we win we carry slavery to the pacific ocean, if we fail, we lose Missouri, Arkansas, Texas, and all the territories." - David Atchison (1854)
"We are organizing. We will be compelled to shoot, burn & hang, but the thing will soon be over. We intend to Mormonize the Abolistionists." - David Atchison (1854)
"The admission of Kansas into the Union as a slave state is now a point of honor. the fate of the South is to be decided with the Kansas issue. if Kansas becomes a hireling state, slave property will decline to half its present value in Missiouri.... [A]bolitionism will become the prevailing sentiment [there]. So with Arkansas. so with upper Texas." - Preston Brooks (1856).
"Nevertheless, the Southerners were successfully shut out of California by a cabal, which obtained California's admission to the Union as a freesoil State. And so California was not on the table when the Southern States left the Union."
"The Nebraska principle of popular sovereignty and non-intervention smooths the way for an establishment of a slave state in southern California. For, if the people of California choose to divide their domain, and to set up another State with Southern institutions, of course Congress will not presume to interpose any objections." - New Orleans Bee (1854, four years after California became a state and at a time when Californians were considering splitting the state)
"I believe in less than two years from this time, if we are wise, we will have a slave state in Southern California. The State has been divided in the last six months for that purpose." - Harry S. Foote (1859)
the last two quotations show the slave-power conspiracy was still hard at work in the American West, including the free state of California.