You're quite right, but then that could be said of Adams I & II, Madison, and Monroe as well. It isn't until Andrew Jackson that we get a "down home" President--and I am an amateur when it comes to Presidential history, but I can't think of one intellectual in either Jackson's official cabinet or his kitchen cabinet.
Young Adams went to Harvard College at age sixteen in 1751. His father expected him to become a minister, but Adams had doubts. After graduating in 1755 with an A.B., he taught school for a few years in Worcester, Massachusetts, allowing himself time to think about his career choice. After much reflection, he decided to become a lawyer, writing his father that he found among lawyers “noble and gallant achievements" but among the clergy, the "pretended sanctity of some absolute dunces." He later became a Unitarian, and dropped belief in predestination, eternal damnation, the divinity of Christ, and most other Calvinist beliefs of his Puritan ancestors. Adams then studied law in the office of John Putnam, the leading lawyer in Worcester.
In 1758, after earning an A.M. from Harvard, Adams was admitted to the bar. From an early age, he developed the habit of writing descriptions of events and impressions of men which are scattered through his diary. He put the skill to good use as a lawyer, often recording cases he observed so that he could study and reflect upon them. His report of the 1761 argument of James Otis in the Massachusetts Superior Court as to the legality of Writs of Assistance is a good example. Otis's argument inspired Adams with zeal for the cause of the American colonies.
Source: Wikipedia
And Madison?
From ages 11 to 16, the young "Jemmy" Madison was sent to study under Donald Robertson, an instructor at the Innes plantation in King and Queen County, Virginia in the Tidewater region. Robertson was a Scottish teacher who tutored numerous prominent plantation families in the South. From Robertson, Madison learned mathematics, geography, and modern and ancient languages. He became especially proficient in Latin. Madison said that he owed his bent for learning "largely to that man (Robertson)."
At age 16, he returned to Montpelier, where he began a two-year course of study under the Reverend Thomas Martin in preparation for college. Unlike most college-bound Virginians of his day, Madison did not choose the College of William and Mary, because the lowland climate of Williamsburg, where mosquitoes transmitted fevers and other infectious diseases during the summer, might have strained his delicate health. Instead, in 1769, he enrolled at the College of New Jersey, now Princeton University, where he became roommates and close friends with Philip Freneau, later dubbed "the poet of the Revolution." Indeed, Madison and Freneau would have become brothers-in-law had Freneau's favorite sister, Mary, accepted Madison's repeated proposals of marriage.[11] But although Mary greatly admired and respected Madison, she had determined to stay single[citation needed]—one way a woman of her intelligence and accomplishments could hope to pursue her interests and remain independent in that era.
Through diligence and long hours of study that may have damaged his health,[12] Madison graduated in 1771. His studies included Latin, Greek, science, geography, mathematics, rhetoric, and philosophy. Great emphasis also was placed on speech and debate; Madison helped found the American Whig Society, in direct competition to fellow student Aaron Burr's Cliosophic Society. After graduation, Madison remained at Princeton to study Hebrew and political philosophy under the university president, John Witherspoon, before returning to Montpelier in the spring of 1772. He became quite fluent in Hebrew. Madison studied law from his interest in public policy, not with the intent of practicing law as a profession.
Source: Wikipedia
I'll stop here.