Scientists exposed as sloppy reporters

New Scientist ^ | 09:30 14 December 02 | Hazel Muir

[vannrox]

Scientists exposed as sloppy reporters

09:30 14 December 02

Hazel Muir

A cunning statistical study has exposed scientists as sloppy reporters. When they write up their work and cite other people's papers, most do not bother to read the original.

The discovery was made by Mikhail Simkin and Vwani Roychowdhury of the University of California, Los Angeles, who study the way information spreads around different kinds of networks.

They noticed in a citation database that misprints in references are fairly common, and that a lot of the mistakes are identical. This suggests that many scientists take short cuts, simply copying a reference from someone else's paper rather than reading the original source.

To find out how common this is, Simkin and Roychowdhury looked at citation data for a famous 1973 paper on the structure of two-dimensional crystals. They found it had been cited in other papers 4300 times, with 196 citations containing misprints in the volume, page or year. But despite the fact that a billion different versions of erroneous reference are possible, they counted only 45. The most popular mistake appeared 78 times.

The pattern suggests that 45 scientists, who might well have read the paper, made an error when they cited it. Then 151 others copied their misprints without reading the original. So for at least 77 per cent of the 196 misprinted citations, no one read the paper.

Spread like weeds

Still, you might think that the scientists who cited the paper correctly had been more dutiful about reading it. Not so, say Simkin and Roychowdhury. They modelled the way misprints spread as each new citer finds a reference to the original source in any of the papers that already cite it.

The model shows that the distribution of misprinted citations of the 1973 paper could only have arisen if 78 per cent of all the citations, including the correct ones, were "cut and pasted" from a secondary source. Many of those who got it right were simply lucky.

The problem is not specific to this paper, the researchers say. Similar patterns of errors cropped up in a dozen other high-profile papers they studied. The trouble is that researchers trust other scientists to repeat the key message of a paper correctly. This means that when misconceptions take root, they spread like weeds.

Simkin and Roychowdhury promise that between them they read all the references listed in their own paper including a book by Sigmund Freud. Their advice to other scientists is "read before you cite".

09:30 14 December 02

Return to news story

  © Copyright Reed Business Information Ltd.

 

[TopQuark]

It's not hard to tell who has decided to write a textbook and who is on the forefront of math research. They aren't the same people. Care to five examples?

Have you heard of Conway, who wrote standard texts on analysis, real and complex? (Another well-known author is Lang, with a series of texts used in many universities).

Or Halmos, whose measure theory book was a standard undergraduate text and is still recommended in most schools?

Or Arnold, whose book on differential equations is standard? Or Massey on Algebraic topology?

In physics, similarly, Dirac's book on quantum mechanics and Landau-Lifschitz series are still standard. Go check the contributions of these authors before you start saying such nonsense.

Not only is you statment is not based on fact, but you base is on a completely wrong premise, suggesting that inventors of ideas should be the ones writing the textbooks. They do not: these two activities requuire different talants.

Now, which of the above mentioned textbooks have you actually read?

[RightWhale]

Care to five examples?

Hmm. Did this semi-phrase mean "care to list 5 counter-examples?" BTW, Dirac isn't my fav physicist. The world isn't pretty.

[lelio]

demeaning someone make you look taller
I'm not sure how I got drug into this, but my point was that in many papers the author just looks at the references in a work he's sited and puts those down on his own, w/o checking that paper. In fact, that's what the article is about: blindly copying a reference without checking.

[RightWhale]

it does break down at times.

There is a weakness that was discussed on a thread a while back, perhaps a sub-thread there it was noted that in published papers it is common to list several names as co-authors while often the co-authors didn't have much to do with the article. The lead scientist or the lab assistant down the hall might be listed as co-authors without contributing substantially to the paper, or sometimes without seeing the paper. This practice varies with the publishing house and with the institute.

[TopQuark]

You did not answer the question, but decided to add something even more ridiculuous:

Dirac isn't my fav physicist

Most scientists do not have "favorites," which tells us about you quite a bit.

Most importantly, you have not read a word written by him. Why don't you learn to read before you make more silly statements.

[TopQuark]

There is nothing wrong with that practice: please see #19.

[Junior]

Anyone with experience of creationist quote-mining knows this phenomenon quite well.

[RightWhale]

you have not read a word written by him.

Now, who is being silly?

[lelio]

But we're not talking about having to read Newton's works to write down F=Ma, its more like siting a paper on global warming and then taking a study from that as gospel w/o reading it. Compounding of errors and all that. Perhaps the underlying paper made some assumptions that the 2nd paper doesn't accurately account for and you're making totally different ones in your own paper.
Its the academic version of Telephone.

[VOA]

. The lead scientist or the lab assistant down the hall might be listed as co-authors
without contributing substantially to the paper, or sometimes without seeing the paper.

While I can't speak for more than a few institutions that I've been at...
which have submitted papers I've helped critique/improve...

what I usually see is that the first and last names on the list of authors really know the paper.
Often the first is some junior researcher (graduate student/postdoc), while the last
name is the "famous name" principal investigator.

And too often the names "in the middle" are modest to even peripheral contributors;
conscientious ones will be able to discuss/defend the paper, the lax ones might have a hard
time naming another co-author.

This happens although I'll occassionally hear a journal/commentator in
the research world forcefully reminding folks that EVERY author on a paper should
know the paper well and be able to defend it.

This is an ideal that, sadly, not enough workers attain.

[_Jim]

In academia, the phenomena has long been known as "Publish or Perish".

a dysfunctional response ...

The story clearly states:

A cunning statistical study has exposed scientists as sloppy ... . When they write up their work and cite other people's papers, most do not bother to read the original.

...

They noticed ... that misprints in references are fairly common, and that a lot of the mistakes are identical. This suggests that many scientists take short cuts, simply copying a reference from someone else's paper rather than reading the original source.

What has this to do directly with "Publish or Perish" (OTHER THAN it may be the result of the pressure to 'P and P', but, that was not the *subject* of this story now was it ...)

[TopQuark]

But we're not talking about having to read Newton's works to write down F=Ma, its more like siting a paper on global warming and then taking a study from that as gospel w/o reading it. This does not happen in major journals. THere is a proper language to be used: "it has been shown in Smith et al (1981) that...", for instance. The paper has been refereed and may be cited as related to one's own work without being read. There is no dishonesty here, or lack of professionalism.

THere is also a self-correcting mechanism: if the results of a related paper are in contradiction with one's own, then the author typically reads it thoroughly. When to do that and to what extent to read --- that is why (in part) you go through a doctoral program.

WHat remains is misprints and inconsequential errors. The authors of the "study" make something out of nothing. Compounding of errors and all that. Perhaps the underlying paper made some assumptions that the 2nd paper doesn't accurately account for and you're making totally different ones in your own paper. Its the academic version of Telephone.

I agree with your concern, and every scientist I know does. That is why it does not happen: as I mantioned earlier, there are self-correcting mecjanisms of which the authors of the study do not show awareness.

When the "error" persists, as a version of Telephone, its was not an inadvertent error but a limitation of knowlege of the whole community. Correcting such "errors" is just what progress is.

{ There is no difference, incidentally, between not reading Newton's papers and those written in twenty years ago. In hard sciences, excellent reviews appear almost immediately. Having the benefit of time, they are usually better written than the orinal papers, and it is more effective to learn from them. In contrast, less scientific disciplines, such as economics and sociology, still insist on reading the original papers. Precisely because this cannot be done well, they are more likely to have Telephone-like results.

[AndrewC]

Anyone with experience of creationist quote-mining knows this phenomenon quite well.

Is that why PH keeps posting that link to the Sciam article that has Hardison's joke in it? This article doesn't mention creationists.

I would bet the vast majority of the scientists who do not read the source material are Darwininians. I would have said exclusively, but I suppose random chance would allow a few attentive Darwininians. It would also seem that natural selection favors the non-reading Darwininian. It makes for more interesting just-so stories.

[f.Christian]

Evolution...slight of hand/mind---spin/allusion!

[scripter]

A cunning statistical study has exposed scientists as sloppy reporters. When they write up their work and cite other people's papers, most do not bother to read the original.

If true that's really scary. Say it ain't so!

[Terriergal]

They modelled the way misprints spread as each new citer finds a reference to the original source in any of the papers that already cite it.

I wonder if this has anything to do with evolution?