You are correct, nobody will ever know. But I bet those 2 million slaves tied to cotton production produced the vast majority of the cotton that was grown in the South.
We'll never know and there's no way to prove that. I tend to suspect it was less than a "vast majority". Cotton was a valuable cash crop. High margins tend to attract more producers into the market.