I thought the term cracker referred to white slaves owners in reference to them cracking a whip to makes slaves work harder or for punishment. Hence, the white slave owner was the "cracker" of the whip.
The “slaveowner” version of the origin appears to be a recent accretion to a word that applied to noisy, uncultured, servant-class people, irrespective of race.