Not really.
The student characteristics they used were gender, socioeconomic status (as measured by eligibility for free or reduced lunches), race/ethnicity, whether the students had disabilities (including learning disabilities, retardation, etc.), and whether the student spoke English well or was just learning the language.
I believe they also adjusted for whether the school was rural or urban, which state it was in (since different states have different standards and differing curricula), experience of the teacher, etc.
You can read the executive summary of the report here.
Also the thrust of the article was that government schools didn't improve by having vouchers. Well?....Why would they? Sclerosis is the hallmark of our unionized, collectivist, government schools.
And....The article complains that the growth of charters and vouchers has been glacially slow. Duh! When there are caps on vouchers and charters what would one expect???? The long waiting lists for vouchers and charters would indicate that the market exists, it is the government that is artificially retarding its growth.
That some voucher and charter schools have been completely shut down is a testament to the success of the private market. It is rare for a failing government school to be shut down, its doors shuttered, and a closed sign posted on its door. At best “closing” down a failing government school, means keeping the same egg carton and rearranging the same unionized eggs.
The comment about Sol Stern is a a distortion. Yes, he had hoped that vouchers would improve government schools. They haven't. Since we are stuck with collectivist government schools he is advocating an improvement in the curriculum. On this point, I agree with him. That does not mean that we should continue with collectivism.
Finally, the parent satisfaction with voucher, tax credit, and charter schools is very high. The waiting lists ( on average) are enormous. Legislators will not ignore their pleas. Only a few years ago the word “voucher” had to be defined for the average citizen. Today there are voucher bills working their way through the congresses of states across the nation. The cracks in the collectivist government school citadel grow daily.
Those defending compulsory attendance at government schools will lose this, regardless of how many massaged studies they produce.
Why?
Answer: Good ideas win. Why? Because people are not stupid. Collectivist government run monopoly schools are a bad idea. Freedom is a good idea.
Although not related to this specific article the following likely applies.
http://www.edspresso.com/2006/07/nces_study_friedman_foundation.htm
NCES Study: Friedman Foundation Responds
On July 14, the U.S. Department of Education released a study that the teachers unions are holding up as evidence that public schools are better than private schools. The study doesnt actually show this, and is riddled with methodological flaws anyway. If you tell the average American that public schools are better than private schools, shes likely to respond, What have you been smoking? In this case, the evidence shows that the average American is right.
The study tells us nothing whatsoever about the relative quality of public and private schools
It takes raw test scores from isolated years and applies statistical controls for demographic factors like race, income, and disabilities. While the raw scores are higher in private schools, once you apply the statistical controls, public school students actually have similar or even higher scores. The teachers unions are rushing to claim that this shows public schools are better than private schools. In fact, as the study itself clearly says, these data show nothing of the kind.
As every education researcher knows, single-year snapshots of test scores reflect student quality much more than school quality. The only real way to get at school quality is to examine year-to-year changes in test scores. A student whose test scores are high is probably just a good student; its the student whose test scores are rising who shows the quality of his school. A much more likely explanation for these data is that students who enter private schools tend to have test scores a little lower than other students of the same race and socioeconomic status. That sounds counterintuitive, because we usually think of private school students as privileged. But they are only privileged in terms of their demographic status which this study controls for. It makes perfect sense that its the low performers within each racial and socioeconomic group whose parents will make the sacrifices necessary to put them in private schools. Theyre the ones who need it the most.
But dont take our word for it. The study itself says the same thing in not one but two big sections labeled Cautions in Interpretation, the study forthrightly states that these data tell us nothing whatsoever about the relative quality of public and private schools. The teachers unions are just blowing smoke, as always.
The study is shot through with other methodological flaws
Paul Peterson of Harvard University, examining the studys data, has discovered that the study only produces a positive finding for public schools because it uses the wrong variable to measure Limited English Proficient students. When the correct variable is substituted, the results are positive for private schools. He also points out that public schools are much more likely to classify students as disabled (about 13% versus about 3%). Peterson doesnt say it, but the main reason for this is that public schools get bigger budgets when they slap the disabled label on a child, a perverse incentive that private schools dont have. If this difference isnt accounted for, any attempt to compare public and private schools is invalid. Peterson will release his analysis at the annual American Political Science Association meeting this fall.
Andrew Coulson of the Cato Institute has also pointed out that the study controls for variables that are endogenous, meaning that they are systematically related to the studys variable of interest, school sector (that is, public versus private schools). School size is systematically related to school sector, and absentee rates probably are as well, but the study controls for them. Dont control for endogenous variables is straight out of Social Science 101.
A much larger body of much better studies finds that, yes, private schools are better
Those who claim this study as evidence that public schools do better are standing against an enormous scientific consensus. If the available research shows anything, it shows that private schools provide a better education than public schools. The consensus among empirical studies on this issue is as strong as on any social policy question. If social science tells us anything at all, it tells us that private schools do better.
While this studys method of looking at single-year snapshots cant even examine whether theres a causal relationship between school sector (public or private) and test scores, it just happens that theres a large body of very high-quality research that does allow us to evaluate this connection, and it overwhelmingly finds that private schools do better. The most convincing evidence comes from seven studies using random assignment, the gold standard for scientific method. In all seven studies, students who won a random lottery to use a school voucher at a private school had significantly greater test score gains than similar students who lost the lottery and stayed in public schools. Numerous studies using other methods have also produced a strong consensus in favor of this finding.
Greg Forster, Ph.D. is a senior fellow and director of research with the Milton and Rose D. Friedman Foundation.