That's a very important point that is never brought up when tests are shown comparing "high school seniors" in the U.S. to other countries. For instance, in Germany, a decision is made at about 14 as to whether a student will go on an academic pre-college track ("Gymnasium") or not. Almost half don't go to Gymnasium. In England, only the college-bound stay in school past age 16 (for "A-levels").
In the U.S., you have a small minority of kids who "drop out" before finishing high school. But in most other countries you have large percentages (40% in many countries) who branch off into non-academic tracks and are thus not included in comparisons to our HS seniors.
I'll bet if you just compared U.S. kids who take A.P. or Honors classes (which is probably a similar percentage of the U.S. high-school age population) to the academic-track, college-bound kids in other countries, U.S. kids would not fare any worse.
I am more concerned with the fact that yesterday I jokingly told a coworker to "beware the Ides of March" and nobody had any idea what I meant.