NY Times Touts Dubious Conclusions on School Quality

Special to FreeRepublic ^ | 3 February 2006 | John Armor (Congressman Billybob)

[Congressman Billybob]

Especially when many members of this community encourage and support home-schooling, this article is important. Here, a government grant was used to manipulate government statistics to "prove" that public schools provide a better education than private, charter or parochial schools. (Home-schooling was left on the cutting room floor.)

Read this and weep.

John / Billybob

[Rodney King]

A major factor the researchers used to change the results was “Home Resources,” detailed as homes which regularly receive newspapers and magazines, own a computer, an atlas, and books, coded by number owned. But household income was a separate factor used (in the form of SES, Socio-Economic Status). Anyone familiar with census data would know that all these items are more common in wealthier families. So, the researchers were actually double-counting household income.

Good 'ole multicolliniarity. Something any basic researcher should understand.

[SteveMcKing]

53% don't graduate high school!

[GSlob]

“Concerning the advancement of learning, I do subscribe to the opinion… that, for grammar schools, there are already too many… the great number of schools which are in your Highness’s realm doth cause a want, and likewise an overthrow [surfeit] – both of them inconvenient and one of them dangerous; for by means thereof they find want in the country and towns, both of servants for husbandry and of apprentices for trade; and on the other side there being more Scholars bred than the State can prefer and employ… it must needs fall out that many persons will be bred unfit for other vocations and unprofitable for that in which they were bred up, which will fill the realm full of indigent, idle and wanton people…”
Francis Bacon, 1611 letter to James I.

[Young Scholar]

A major factor the researchers used to change the results was “Home Resources,” detailed as homes which regularly receive newspapers and magazines, own a computer, an atlas, and books, coded by number owned.

Let's see. I was homeschooled and scored 1500 on the SAT. But if we adjust for my family's "Home Resources"--3 or 4 magazines, 6 computers, about 5 or 6 atlases, and at least 16 bookcases of books--I guess I'm probably below the average public-school student.

[Ninian Dryhope]

I never believe anything the liberals say about anything, because I have found them lying too many times. We home-schooled our daughter and she is now a fine young woman, who will be graduating in May with a degree in Chemical Engineering.

My wife and I did not need a study to tell us that public schools suck.

[conservatism_IS_compassion]

When the study normalizes the results for household income, are they actually comparing the performance of private schools only with public schools in toney suburban districts?

Exactly what use do they make of the data on the performance of the inner city government schools? What do they compare that to?

[pabianice]

[GSlob]

Well, by your own admission you scored minus 100. Serves you right. [:-)]

[Young Scholar]

Hey no need to rub it in (my parents already do that enough).

[buwaya]

Argument about the argument -

My take - I don't have the same objections about the study. Asians and homeschoolers are very small minorities in the context of the US. Homeschoolers aren't tested by NAEP, so they aren't in the database at all. Asians as a separate category would have been an insignificant sample size, and as I understand it they are not classed as a "minority" at all for the purposes of this study, which is fair enough. "School discipline climate" and other such factors are pretty much impossible to measure, and it seems to me if they were it would count against private schools in this model by becoming yet another reason to say that public schools are good at getting results in spite of their structural disadvantages. Others that were dropped may have been of no value in the regression model; this happens, they could easily be collinear with other things and add nothing to the model. Family income is not available in the NAEP data, what is used as a proxy is the school-lunch qualification data - see below, this is very important. Double-counting is not an issue in regression models.

I think the study is very badly flawed, for different reasons -

Things like this depend on a host of judgement calls as to how to handle issues such as missing data, data that is inapplicable across categories, sources of data, and purely statistical issues like assumptions of linearity.

The critical issues in terms of judgement calls in this one seem to be:

School lunch qualification. This is a universal thing in public schools, in studies it is used as a proxy for family income, but school lunch is not often supported in private schools no matter the income, and certainly isn't pushed like it is in the publics, and very often not in charter schools either. The authors acknowledge the issue but kept it anyway, on rather specious grounds. This is in my mind a quite questionable decision. This school lunch qualification factor is a very heavy one in the regression model, but is quite inapplicable across the categories they are comparing, and this alone should account for their results.

The retention of the IEP and LEP (students with special needs as defined by Federal law) statistics. This is rarely used in private schools, and typically educational problems are not diagnosed at anything near the extremely high rate they would be in public schools. These are also often de-emphasized in charters. They say it makes little difference, on grounds that I cannot fathom. This seems to me questionable, these are very heavy factors.

Reliance on survey information, as a demographic factor and as a means to make other data-inclusion decisions. In particular on student survey information, responses to which would carry a cultural load that would be affected by the school situation. A chicken-and-egg issue indeed.

No examination of a "ceiling effect". I.e., SES, even computed with their rather questionable manner, can only do so much. This is a restriction of range/nonlinearity effect. Non-selective schools tend to max out no matter how white/Asian/wealthy the kids are. If private schools are concentrated on the right side of a SES/achievement graph, their SES/achievement trend line would be much more affected by the ceiling.

These are just a few of the problems that can be found with this sort of analysis.

[GSlob]

How do they say it in German? "Whatever the little Hansel has not learned, grown-up Hans will not know." How are you to compete in a global economy when you score minus 100? In my day you'd have to walk to and from your school - five miles each way, and both uphill, in waist-deep snow even in September...

[buwaya]

No, you would be above the average, and if a study were to take those into account these people would say that with all that at home (or actually using these as a proxy measure for parental attitudes) you would have done even better at a public school.

Which is a flawed assumption, but thats how it works.

[buwaya]

Yes, they effectively compare private schools with higher-end public schools, or rather individuals who go to one or the other.

The bottom part of the public school range is difficult to compare to private schools. Their NAEP data has individuals test scores but their classification and comparison according to social status is very hard to do. Its a restriction of range problem in part.

They would have an easier time with charters as the data is more fleshed out across the range. But they have stated that their charter info is very sparse as well.

[Congressman Billybob]

You are obviously schooled in statistical methodology and have read the study at issue. Wish I'd had access to someone like you when I was writing my own article. But I didn't.

Can you write it up and get it published? This study calls out for an answer.

John / Billybob

[southernnorthcarolina]

Though private school students have long scored higher on the national assessment, commonly referred to as "the nation's report card," the new study used advanced statistical techniques to adjust for the effects of income, school and home circumstances.

This, of course, is the ultimate result manipulator. In the real world, employers don't, or shouldn't, give a rat's hindquarters about the prior circumstances of a prospective employee. Can you, or can you not, do the work; express yourself in English; and present a favorable image for your company?

[Lizavetta]

the new study used advanced statistical techniques to adjust for the effects of income, school and home circumstances.

Translation: They changed the actual results.

[buwaya]

I am actually quite unqualified to do this, I just deal with this stuff at work. And I would have to analyze the NAEP data myself to make a good job of it, which is no joke.

I think someone at Hoover probably will write a response. At least over there they get paid for this.

http://www.educationnext.org/

[buwaya]

The argument is that if private schools had to deal with the kind of kids that public schools have to they wouldn't do as well as the public schools are doing now.

Or, conversely, if kids that go to private schools now went to the sorts of high-end public schools such kids would tend to, they would do even better than they already do.

The very questionable study does not support either conclusion properly in my opinion, but thats what they are getting at.

[Old Professer]

If only it were so simple as being a year behind the solution would lie in a couple sessions of summer school; in reality, by the time the 8th graders in question get four years down the road, a third of them are ready to graduate from reform school and the rest are in residential post-graduate correctional institution studies.