You’re right. You have to look at the productive capacity of the investment and its ROI cash on cash. How much arable land would $500 buy?
In 1860, just as today, the price of land was based on three important factors: 1) location, 2) location and 3) location...
But, according to this site, farmland then was selling for circa $3 to $5 per acre, meaning $500 would purchase around 100 to 150 acres.
Today, a 100 acre farm in, oh say, central PA, costs circa $500,000 and in garden-spots like Lancaster County, over $1 million.
Again, the big point here is: you cannot possibly overestimate the economic value of slavery to their owners in 1860.
Slaves then represented over 20% of the US total assets' value, but in the Deep South, well over 50%.
That's why the political dividing lines, between those who demanded immediate secession, versus southern Unionists, was strictly based on the percentage of slave-owners in any particular region.