To show a substantial difference between the circumstances, worldviews, and prospects of two different social groups is not necessarily freighted with disdain. It is also just the opposite of ignorance, but the product of observation and application of one's native intelligence.
I really disagree with your statement.
As for the differences between lower-income freehold farmers -- Jefferson's yeomanry -- and the Tidewater planter aristocracy of the Federal period, whole forests have rendered up their cellulose to historians who have busied themselves pointing out the salience of these group differences to American history and politics both then and later.
However accurate that might be, I do not believe that this factored into the President’s statement. I find it extremely offensive that his statement would be twisted for political gain.