Posted on 02/03/2006 1:16:21 PM PST by Congressman Billybob
The New York Times ran a story on 28 January, 2006, entitled, “Public-School Students Score Well in Math in Large-Scale Government Study.” Well, it wasn’t a “government” study. It was only paid for by a government grant. When one looks into the methodology of the atudy and the histories of its two researchers, the results are highly suspect.
The Times wrote:
A large-scale government-financed study has concluded that when it comes to math, students in regular public schools do as well as or significantly better than comparable students in private schools.
The study, by Christopher Lubienski and Sarah Theule Lubienski, of the University of Illinois at Champaign-Urbana, compared fourth- and eighth-grade math scores of more than 340,000 students in 13,000 regular public, charter and private schools on the 2003 National Assessment of Educational Progress. The 2003 test was given to 10 times more students than any previous test, giving researchers a trove of new data.
Though private school students have long scored higher on the national assessment, commonly referred to as "the nation's report card," the new study used advanced statistical techniques to adjust for the effects of income, school and home circumstances. The researchers said they compared math scores, not reading ones, because math was considered a clearer measure of a school's overall effectiveness.
Source: http://www.nytimes.com/2006/01/28/education/28tests.html?_r=1&oref=slogin
The study itself says, but not until page 18, that “overall, due to the complexity of the issues involved, no single study can provide a definitive determination of the effectiveness of various forms of schooling.” The Times never mentions this caveat.
The study was based on the NAEP exams in math for 4th and 8th graders in 2003. The actual results showed that public school students scored about one grade level below private school students, and charter school ones. But the authors of the study then set out to “normalize” the data, by adjusting for sociological and home factors including race, income, and ownership of a computer.
It is not until page 21 that we find out that the factors used to adjust the data, and therefore reverse the results seen in the actual test results, were determined by a prior study co-authored by one of these authors, in 2004. Among the factors deliberately left out were, “school discipline climate, teacher quality, and even parental involvement.” Most studies conclude that all three of these have major impacts on student achievement.
A major factor the researchers used to change the results was “Home Resources,” detailed as homes which regularly receive newspapers and magazines, own a computer, an atlas, and books, coded by number owned. But household income was a separate factor used (in the form of SES, Socio-Economic Status). Anyone familiar with census data would know that all these items are more common in wealthier families. So, the researchers were actually double-counting household income.
Another fudge factor appears in footnote 12, where the authors wrote, “In order to preserve data, students who reported that they did not know if they had a particular resource were coded as ‘no’ on the grounds that even if the resource was present in his/her home, it was not (directly) enriching the student’s home experiences.” If the public school students were less likely to report accurately their “resources” at home, this assumption alone would question the entire results of this study.
On racial composition, the researchers used only “Black, Hispanic and American Indian.” Conspicuously absent from this list are Asian Americans. And anyone with the slightest knowledge of educational achievement in the US knows that the Asian-Americans outperform all other racial categories, including “Caucasian,” on standard tests like the ones here.
The reason why the data may have been manipulated to obtain the result that public schools are providing a better education – no matter that the actual test results show the opposite – appears on page 39, where the authors state that their results, “call into question the basic premise of such reforms [meaning charter schools, and voucher programs].”
An indication of the possible prejudices of the authors of this study appears in the bibliography, where the authors cite no less than nine papers written or co-authored by one of them, most of which challenge the premise that non-public schools produce a better educational product.
In short, had the reporters at the New York Times bothered to read the report they were reporting on, they would have found ample reason to question the conclusions it reached. But because the study supported the mantra of the Times that competition between public and private schools for parents with limited resources is undesirable, the Times trumpeted the results as solid.
And sadly, a fair number of other national media may take this study seriously without examining its methodologies, in part because of this stamp of approval from the Times.
John_Armor@aya.yale.edu
Read this and weep.
John / Billybob
Good 'ole multicolliniarity. Something any basic researcher should understand.
Let's see. I was homeschooled and scored 1500 on the SAT. But if we adjust for my family's "Home Resources"--3 or 4 magazines, 6 computers, about 5 or 6 atlases, and at least 16 bookcases of books--I guess I'm probably below the average public-school student.
When the study normalizes the results for household income, are they actually comparing the performance of private schools only with public schools in toney suburban districts?Exactly what use do they make of the data on the performance of the inner city government schools? What do they compare that to?
Well, by your own admission you scored minus 100. Serves you right. [:-)]
Hey no need to rub it in (my parents already do that enough).
Argument about the argument -
My take - I don't have the same objections about the study. Asians and homeschoolers are very small minorities in the context of the US. Homeschoolers aren't tested by NAEP, so they aren't in the database at all. Asians as a separate category would have been an insignificant sample size, and as I understand it they are not classed as a "minority" at all for the purposes of this study, which is fair enough. "School discipline climate" and other such factors are pretty much impossible to measure, and it seems to me if they were it would count against private schools in this model by becoming yet another reason to say that public schools are good at getting results in spite of their structural disadvantages. Others that were dropped may have been of no value in the regression model; this happens, they could easily be collinear with other things and add nothing to the model. Family income is not available in the NAEP data, what is used as a proxy is the school-lunch qualification data - see below, this is very important. Double-counting is not an issue in regression models.
I think the study is very badly flawed, for different reasons -
Things like this depend on a host of judgement calls as to how to handle issues such as missing data, data that is inapplicable across categories, sources of data, and purely statistical issues like assumptions of linearity.
The critical issues in terms of judgement calls in this one seem to be:
School lunch qualification. This is a universal thing in public schools, in studies it is used as a proxy for family income, but school lunch is not often supported in private schools no matter the income, and certainly isn't pushed like it is in the publics, and very often not in charter schools either. The authors acknowledge the issue but kept it anyway, on rather specious grounds. This is in my mind a quite questionable decision. This school lunch qualification factor is a very heavy one in the regression model, but is quite inapplicable across the categories they are comparing, and this alone should account for their results.
The retention of the IEP and LEP (students with special needs as defined by Federal law) statistics. This is rarely used in private schools, and typically educational problems are not diagnosed at anything near the extremely high rate they would be in public schools. These are also often de-emphasized in charters. They say it makes little difference, on grounds that I cannot fathom. This seems to me questionable, these are very heavy factors.
Reliance on survey information, as a demographic factor and as a means to make other data-inclusion decisions. In particular on student survey information, responses to which would carry a cultural load that would be affected by the school situation. A chicken-and-egg issue indeed.
No examination of a "ceiling effect". I.e., SES, even computed with their rather questionable manner, can only do so much. This is a restriction of range/nonlinearity effect. Non-selective schools tend to max out no matter how white/Asian/wealthy the kids are. If private schools are concentrated on the right side of a SES/achievement graph, their SES/achievement trend line would be much more affected by the ceiling.
These are just a few of the problems that can be found with this sort of analysis.
No, you would be above the average, and if a study were to take those into account these people would say that with all that at home (or actually using these as a proxy measure for parental attitudes) you would have done even better at a public school.
Which is a flawed assumption, but thats how it works.
Yes, they effectively compare private schools with higher-end public schools, or rather individuals who go to one or the other.
The bottom part of the public school range is difficult to compare to private schools. Their NAEP data has individuals test scores but their classification and comparison according to social status is very hard to do. Its a restriction of range problem in part.
They would have an easier time with charters as the data is more fleshed out across the range. But they have stated that their charter info is very sparse as well.
Can you write it up and get it published? This study calls out for an answer.
John / Billybob
This, of course, is the ultimate result manipulator. In the real world, employers don't, or shouldn't, give a rat's hindquarters about the prior circumstances of a prospective employee. Can you, or can you not, do the work; express yourself in English; and present a favorable image for your company?
Translation: They changed the actual results.
I am actually quite unqualified to do this, I just deal with this stuff at work. And I would have to analyze the NAEP data myself to make a good job of it, which is no joke.
I think someone at Hoover probably will write a response. At least over there they get paid for this.
http://www.educationnext.org/
The argument is that if private schools had to deal with the kind of kids that public schools have to they wouldn't do as well as the public schools are doing now.
Or, conversely, if kids that go to private schools now went to the sorts of high-end public schools such kids would tend to, they would do even better than they already do.
The very questionable study does not support either conclusion properly in my opinion, but thats what they are getting at.
If only it were so simple as being a year behind the solution would lie in a couple sessions of summer school; in reality, by the time the 8th graders in question get four years down the road, a third of them are ready to graduate from reform school and the rest are in residential post-graduate correctional institution studies.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.