The right to vote was certainly more restricted than it is today but you didn't have to be a "substantial" property holder to vote. Just a property holder in most of the South and even then townies like shop owners, lawyers, etc could vote. The things like literacy tests and poll taxes etc designed to exclude the working class became much more of a feature after the war. Broadly, yeoman farmers could and did vote. You say slavery was integral to the South's economy even for non slave owners. I won't deny all that labor was certainly important as a whole. For non slave owners however, the existence of slavery harmed the value of their labor.
More generally, the bottom rung of farmers tended to occupy small and often impermanent subsistence farms. Historians have identified a strain of proud, stubborn, combative Scotts-Irish hardscrabble farmers who worked pine scrub land they often did not even own. Beginning in Virginia, competition from large plantations gradually pushed many of them west or down the Atlantic states into south Georgia and Florida.
Several close friends were of such stock. And, as with others of that background, they provided the Confederacy with fighting men. As the saying went, it was a rich man's war, but a poor man's fight. Three friends had ancestors who were Confederate officers and small scale farmers and slaveholders.
Another family friend had ancestors who were significant plantation owners and political figures in Tallahassee. One such plantations became the City of Tallahassee Golf Course and Country Club, complete with unmarked graves of slaves and a dodgy maneuver that kept blacks out as members for decades.
Yet another friend in Tallahassee was a small businessman with ancestors who arrived in the 1830s of the hardscrabble type. His ancestors never owned slaves. Good hearted and tough as an old tree root, my friend clashed with the powers in local politics. Years after his death, many of his adversaries were prosecuted for corruption.
My excuse for bringing up old friends who have passed is that, as Faulkner put it, "The past is never dead. It's not even past."