LINKPeer review: the Holy Office of modern science
Keywords: electronic communication, objectivity, peer review, refereed journals, research grants, science funding, science publishing, subjectivity.
MACIEJ HENNEBERG Note 1Department of Anatomy and Histology, University of Adelaide, Adelaide 5005, Australia, mhennebe@medicine.adelaide.edu.au
Received February 17, 1997, published February 20, 1997
Summary: A brief historical overview of the origins of peer review reveals that it is neither the best means of evaluating contributions to science nor the one most commonly used during the period in which the modern scientific method developed. Throughout history, most scientists published their views without formal review and peers published their criticisms openly. It is argued here that peer review as now undertaken by most scientific journals stifles scientific communication, slows the advancement of knowledge and encourages dishonest behavior among referees. Alternatives to peer review that have already been used by some journals and funding bodies are described. Since these alternatives have proved themselves in practice, the now commonly practised form of peer review can be abandoned or modified. Electronic communication can facilitate this process.
Science, communication, society
Science is a specific collective human activity (Hull 1988). It is about acquiring an understanding of the world that is of practical value. It is different from both commerce and ideology and hence its practice in society has to be controlled by a separate set of rules and behavioral norms. Exchange of information and opinions among practitioners of science is crucial to its progress. Results of scientific research are useless unless they are communicated to other scientists and the public at large.
Presentation by scientists of the results of their research to others, be the work academic or industrial, is thus essential to the scientific enterprise. Because the nature of scientific work, at least according to Karl Popper (1972), is to approach the truth through a series of increasingly accurate or useful approximations, no piece of research can be held to be definitive. All results are transitional and approximative. Although scientists strive for perfection, by the very nature of their endeavor they cannot achieve perfection. It follows that neither a paper presenting new results of observation or theory nor a proposal for a new investigation can be perfect. They can only be “good enough.” One of the paradoxes of science, which purports to measure various phenomena with great precision, is that it cannot define a precise measurement of “good enough.” Assessment of what “enough” means is left to the judgement of individual scientists. Hence nonspecialists who cannot reach an informed opinion about some theory or discovery often tend to be guided by the consensus opinion of specialists.
Leaving assessment of research results and theories to the judgement of individuals, however well informed in a particular field of expertise, introduces a factor of subjective opinion into seemingly objective science. The ancient Greeks, who originated science as a specific endeavor based on logical argument and empirical tests, realized its susceptibility to subjective opinions. The Greek way of dealing with possibly biased opinions was to conduct open debates between scientists, to produce mathematically exact descriptions and to run empirical tests of statements made by learned persons. Broad discussion and empirical testing were the foundations of learning. Sometimes these were taken too far, as for example in Hellenistic Alexandria, where curious scientists conducted vivisections on condemned criminals (Persaud 1984). Roman ethics put an end to such practices and even forbade dissections of deceased humans. In Roman times, learning became more dependent on written texts than on observation of nature and theoretical arguments. Sanctity of the written word was entrenched by medieval European scholastics.
Science as a human endeavor is also open to social pressures expressed as moral and ethical norms. All human activities need to be organized and regulated by norms of behavior that constrain actions of individuals and attach value to decisions. In other words, individual opinions and actions must be censored and decisions justified.
There exists a basic contradiction between the free search for ideas and their empirical testing on the one hand, and a priori norms of behavior on the other. In the ideal world, free flow of ideas and unrestricted empirical testing should result in the greatest accumulation of knowledge. Empirical tests require time and resources that must somehow be provided by society. No economy is unlimited in size and, hence, no economy can support all possible experiments scientists might think of. Limited resources require that somebody decides which experiments are to be done and which not. Such decisions necessarily involve an element of subjectivity and are constrained by social norms. Scientists in their quest for a practical understanding of the world must navigate narrow straits between social conventions, economic reality and their own human limitations. Not least among these latter are egotism, greed and plain fear.
Historically, it seems that gentlemen-scientists with an elevated social position and independent means, fared the best in science. In the 16th century, at the dawn of the modern era, one such individual, the son of a middle-class family, holder of the respectable office of Canon of the Frombork Cathedral, published a book on the revolution of celestial bodies that revolutionized understanding of our place in the universe. The man was Nicholas Copernicus. His accomplishment was simple: he moved the Earth from the center of the world onto an orbit around the Sun. Being supported by the endowments attached to his post and having received a thorough education at Polish and Italian universities, he had the time and knowledge to conduct astronomical observations and to write a lengthy manuscript in relative peace. He realized the revolutionary nature of his work and dreaded the opinion of many learned colleagues as he clearly spoke against the then accepted views. And yet the book was published. It would not be possible today to publish a book full of minor errors and simplifications and arguing something so ludicrous as a complete reversal of a consensus of well-established authorities. Would it also be possible for a modern scientist to obtain a grant for a proposal based on the supposition that the major publications on the subject are wrong and that he will prove it by conducting observations from the roof of his residence? Absolutely not. Peer review protects us against such lunacies.
Debate of scientific theories and reinterpretation of results is a core of science. This debate benefits from the widest possible participation. This was well recognized by founders of scientific journals in the early centuries of the modern era. Scientific journals and published transactions of learned societies provided means of relatively fast and broad communication of ideas among scientists. Published works were there for everyone to read and it was open to everyone to publish counter arguments. Transactions of many societies included the text of both presented papers and the ensuing debate. All points of view were there in print for everyone to see and comment on. Reading such printed accounts could be bewildering for inexperienced readers, but sharpened their powers of reasoning as each argument had to be absorbed, evaluated and compared with other arguments.
With time it became obvious that not all written matter submitted to journals for publication or to societies for presentation could be published. A certain standard of quality had to be applied. Editors of journals and officers of societies made judgements as to what is acceptable and what not acceptable. Junior or less well-known scientists were not allowed to present their papers at the meetings of various societies, but their papers were introduced by recognized fellows who themselves were elected to fellowships by older and well-established colleagues. Fellowships of the Royal Society and many national academies are awarded in this manner even today.
Editors of journals and fellows of learned societies in the 19th century were usually broadly educated academics. They felt themselves to be well equipped to judge the quality of practically all research reports and theoretical papers submitted to them for publication or presentation. In making such judgements, they also kept in mind the reputation of the journals and societies, and so they also acted as guardians of publicly accepted ethical norms.
It is only natural for humans to evaluate possible consequences of a public statement before making it. Such evaluation, besides obvious reliance on an individual’s knowledge and experience, often tends to be based on the advice of friends, family members and colleagues. Hence many manuscripts of scientific works were read and commented on by the author’s friends and colleagues before being submitted for publication. Such was the regular practice of Charles Darwin, who tested his ideas on his friends before committing them to print (Desmond and Moore 1991).
Peer review today
As the 20th century dawned, the breadth of scientific endeavor became such that there were hardly any persons who could feel competent to evaluate contributions in more than a limited range of subjects. Thus, editors started asking specialists to evaluate papers submitted to journals. The specialists acted in an advisory capacity, and editors still took full responsibility for their decisions. In order to encourage frankness by referees, editors undertook to ensure that they would remain anonymous to the author. With time, overworked editors transferred all work of evaluating manuscripts and suggesting corrections to referees. Referees took make-or-break decisions, but still remained anonymous. There is a commonly held view that refereeing of papers ensures that they become definitive publications, and that what is published must be true and need not be questioned. This is obviously wrong as referees are as prone to errors as are authors. Although it is less likely that a paper will contain major mistakes after it has been scrutinized by several reviewers, errors in published papers are nevertheless common. The proof lies in a number of corrections published in Errata sections of major journals.
The most common form of peer review today is for an editor to send a manuscript to several specialists asking them to answer standard questions about the paper and to provide written comments. The questions nearly always include the one regarding recommendation to publish “as is,” “with revisions” or “reject.” The process used to referee grant proposals is virtually the same, although instead of “accept” or “reject” referees are asked to assign a score or a grade to the proposal.
Being asked to referee a paper or a grant proposal anonymously is the dream of anybody seeking power—power without responsibility. Referees acting under such circumstances may be tempted to reject, or delay, publication of papers they disagree with, as their pronouncements regarding the quality of work under review are unlikely to be questioned. The same goes for the refereeing of grant proposals (Goodstein 1996).
Since all the thinking about delaying publication of “unwelcome” results is done in the heads of anonymous referees, it cannot be examined directly. It can, though, be teased out statistically. Increased reliance of journals on decisions made by reviewers increases time from submission of a paper to its final acceptance, because referees recommend alterations to the content and form of the manuscript. The number and difficulty of proposed alterations may be proportional to the intention to delay publication. Many journals have adopted the practise of printing the “date submitted” and the “date accepted” as a footnote to a paper. I have examined the delay between submission and acceptance in two journals: the American Journal of Physical Anthropology (AJPA) and the Medical Journal of Australia (MJA). In 1980, when AJPA first published the information, the average waiting time from submission to acceptance was 6.9 months (Median = 4.8, SD = 4.5) whereas in 1996 it increased significantly (P < 0.01) to 15.7 months (Me = 11.3, SD = 9.8). Papers published in MJA in 1984 had an average waiting time of 3.5 months (Me = 2.0, SD = 3.3), which increased significantly (P < 0.05) to 5.6 months (Me = 4.4, SD = 2.5) in 1995. It is strange that these increases occurred during a period in which the quality of research work should have improved and the efficiency of mails and other communication systems increased. Slowdown in communication of scientific results is obviously to the detriment of scientific progress and appears linked to reliance on the process of peer review.
The peer review of grants has its origin in industrial practice. Many people propose to do many things but resources are limited. How can a small number of people who hold the purse strings know everything? The answer is that they cannot, and so they must ask specialists. The best solution is to ask people working on problems similar to the one a grant application proposes to solve. They will be able to highlight merits and point out errors of the proposal. Of course they will be more frank if ensured anonymity. Since people working in similar areas usually know each other and either agree and collaborate or dislike each other and compete among themselves, the result of a review of a particular application will depend on considerations external to the application. These considerations, besides likes and dislikes, also include political and financial elements. What being anonymous and having power means to a referee has been discussed above. A good grantsman will write an application in such a way as to avoid being criticized. It will either be bland, proposing an innocuous piece of research, or it will propose to continue what the author has been successfully (in terms of grants) doing before. Any testing of somebody else’s theory or encroachment into someone else’s area of research creates a high risk of failure.
Paradoxically, some good scientists who do not feel inclined to use anonymous powers to manipulate the system, are not keen to act as referees. This happens to the extent that one of the major science funding agencies now coerces people into refereeing grants by saying that it will automatically refuse grants to academics who do not agree to referee proposals of others. For an honest but busy academic there is very little attraction in refereeing a paper or a grant proposal. It takes time and effort to write a well thought-out assessment and, since it is anonymous, no credit accrues to the reviewer even if, in the process, he comes up with a bright new idea. Authors of papers who receive substantial help from referees certainly recognize their indebtedness, as they often thank referees for their input in the “Acknowledgments” section of their papers. Referees of grant proposals find themselves in an even worse predicament. Since they referee grants in their own area of interest, they may be strengthening their own competitors if they provide honest criticism that can be used to improve the proposal.
In general, peer review stifles scientific enquiry both by subjecting authors to anonymous critics whose comments cannot be directly challenged, and by self-censorship by authors who hesitate to state the exact rationale or goals of their proposed research for fear that it would not pass the scrutiny of their colleagues. The word colleagues describes referees more truthfully than the word peers, because, although all scientists have a right to do research and express opinions in any area, scientists are not all equal in experience, talent and diligence. Those who are experienced, talented and meticulous, rarely have peers in the sense of equals. How many physicists of the 20th century would consider themselves equals of Albert Einstein? Nevertheless, an enthusiastic and talented, but inexperienced postgraduate student is going to have her work judged by older, much more experienced professors as if her work had been done by a professor. A bit of understanding and encouragement, rather than strict criticism would perhaps help to develop a valuable scientist. As it is, she may be discouraged by rejection of her work following peer review.
Alternatives to current peer review practices
These days, everybody complains about deficiencies of peer review, but few believe that anything can be done about it. How such a view could pass peer review is beyond me. Over the years and in various countries systems of evaluation according to principles other than “peer review” have been applied with good results. These systems still rely on opinions of academics other than the author, but these opinions are not anonymous or dependent on reviewers ticking boxes or assigning a score.
Until about 20 years ago, many journals relied on editors to make decisions without formal refereeing of papers. Authors were expected to have their manuscripts read and discussed by their colleagues before submission, and it was customary for well-established academics to recommend to the Editor publication of papers written by younger colleagues or students, although this did not guarantee publication. At the same time, students were free to submit papers without professor’s recommendation. Editors, as they became conscious of good work by particular academics, approached them with invitations to publish in their journals. This tradition continues in a limited way even today. For example, only two years ago a respected French colleague approached one of my students to publish his freshly completed PhD thesis in a series of which he was an editor. Such an invitation does not mean that the manuscript will not need to be edited based on direction of the Editor and comments of whomever he has had review the manuscript. The decision, however, is very much out of the hands of peers. Although an editor is, like an anonymous peer reviewer, merely human, his name is known to authors and he has some recognized general policy for his journal. Furthermore, he has a clear interest in the reputation of the journal he edits. These goals are not in conflict with the specific research goals pursued by authors who may wish to publish in his journal.
The editorial system used by leading journals such as Nature run on somewhat similar lines. In the first instance, papers submitted to Nature are evaluated by its editors and only a small number that survive editorial scrutiny are sent for peer review, which therefore takes the form of technical advice to the editor. The extent of editorial control over the process of publishing papers in Nature is evident from the fact that the time from submission to acceptance has not changed over the last 12 years. My calculations based on 283 research papers indicate that the mean delay between submission and acceptance was the same in 1984 (3.1 months, Me = 2.9, SD = 1.7) as it was in 1996 (3.1 months, Me = 2.8, SD = 1.8). The system used by naturalSCIENCE is constructed along similar lines.
The need to allocate research monies unavoidably limits the freedom of scientific exploration, with the result that not all scientists get to pursue the research they want. However, the less the process of allocation of funds stifles scientific freedom the better. Giving money for three-year bits of circumscribed research, a practice that seems now to be quite common, is the worst means of fund allocation. It seems to have arisen from the need to give peace of mind to administrators who cannot make too big a mistake by giving away funds in dribs and drabs to projects having tight schedules and detailed budgets. How can one budget for a discovery? Until the end of 1995, an alternative system was used by the major South African government agency the Foundation for Research Development. In a modified form, combined with project-oriented funding, the same method is still used under the new democratic South African government. The system is based on the principle that good scientists produce even better science when given resources. It relies on a scientist’s track record. A funding submission consists of a Curriculum Vitae with a description of research achievements, concentrating on the most recent past. The candidate is invited to name internationally renowned referees, although the agency may use other persons as well. The submission is evaluated by several international referees in order to establish whether the applicant is an international leader in his field (A), a leader in a specialized area (B), or a scientist who makes regular internationally recognized contributions to his field (C). If none of those descriptions applies, the person is classified D (inactive, no funding will be allocated). There is a special provision for young scientists who have only just completed their doctorate but have proven their worth in other ways (Y). Formerly, an academic was reevaluated every 4 years. Funding was allocated for 4 years based on a category and a one-paragraph general description of the planned research. Now, a more detailed research plan is usually required. The idea is simple—trust a researcher and she will produce results. Similar methods of funding include giving a renowned researcher an endowed chair with research funds attached, or establishing a university research unit for an eminent scholar, which will receive university-funded equipment and staff. In case of younger people, funds such as scholarships or postdoctoral fellowships are allocated by a process based on recommendations by supervisors and evaluation of previous work and experience by variously constituted panels.
It seems ludicrous today to mention past systems in which academic appointments carried a salary designed to cover research expenses, or departmental budgets including lines for research equipment, consumables and travel. And yet, under such arrangements, good research was done because academics were genuinely interested in what they were doing and did not have to rack their brains for ideas that looked acceptable on a grant proposal. Most of the Nobel Prizes won earlier this century were for work done under such circumstances.
There are ways, short of abolishing it, to make peer review less obstructive. The simplest is to take away from reviewers the power to make decisions on a paper or a grant. Instead, they should be asked to provide substantive written comments but not to tick a box recommending a concrete decision or to assign a score. Then the editor or a panel of a grant agency will have to make a decision based on their own assessment of the work and specific points raised by the referees. More work for decision makers, but a fairer deal for authors. The other way is to make referees' names known to the author. This introduces subjective bias, but of a different kind—if a referee wishes to criticize the work he can still do it and it will carry more weight as his personal reputation is at stake. Every time I referee a paper from which names of authors have not been removed (for the purpose of “blind” refereeing), I insist that the editor reveals my name to the author, even if my comments are very critical. This ensures equality—I know her name, she knows mine.
Generally, it seems that the authors should be free to publish their results and conclusions provided that they are presented in a technically correct manner (which editors can check for themselves), and colleagues should be free to publish their criticisms and derive any credit that may be due for innovative comments, and for exposing themselves to debate. Introduction of electronic means of communication goes a long way toward making possible this ideal of free exchange of scientific information. Cost of electronic publishing is lower and hence the volume of exchanged information can increase. Traceability of electronically published pieces is good, and hence appropriate credit can be given to authors and their work can be formally quoted.
LINKPeer Review is Censorship and Intimidation
Peer review is a form of censorship, which is tyranny over the mind. Censorship does not purify; it corrupts.
Peer review is often assumed to be a purification process. There is no such thing as purifying science. Scientific knowledge must continually evolve. Like all truth, no one can arbitrate it; it must speak for itself through the evidence. There is a lot of junk science and trash that goes through the peer review process.
Prior criticism is always of value. But the author has to decide what to do with it. Imposing thought onto someone is a guaranteed corruption.
The purpose of review should not be to determine fact. Three scientists cannot determine fact for two million others. What should replace peer review is letting editors decide what to publish based on guidelines which are openly and accountably created. Editors don't need to look at technical details. Let all of the scientists evaluate the technicalities instead of three.
|
There is a natural inclination to review and criticize any document before it is published. Everything from fiction to news is handled that way. And it is censorship which restricts all forms of publishing to fit in some type of box. In industry, the owner has a right to do that. But there is no owner for science. Its product is supposed to be an evolving truth which does not fit into a subjective box. Therefore, the reviewing should be limited to suggestions, not a shaping and controlling process. After the suggestions, let scientists include their errors with their worth, particularly since there is seldom agreement as to what is error and what is worth in science.
Scientists are so intimidated that they cannot oppose official corruptions in science. One of the main causes is peer review.
If peer review were open and accountable, there might be a small chance of correcting some of the corruptions through truth and criticism; but the process is cloaked in the darkness of anonymity.
There is no place for secrecy in science after the research is done. A laboratory needs some protection from interference while it is working through the challenges, but the evaluation process cannot produce truth through secrecy and unaccountability.
You might assume that there are no official errors in science. Due to the exploitive and corrupt process, nearly everything in science has official errors within it. There is nothing in science entirely free from errors, while a culture of protecting and exploiting the errors creates an official reality which cannot be opposed.
Relativity was the most obvious and extreme example. It was imposed upon physicists, and none were allowed to dissent, as explained on other pages.
A recent parallel in biology is prion proteins as the supposed cause of Scrapies-like diseases. But unlike physics, the evidence in biology is far more available and less abstract. So prions are a much more open defiance of principles and standards. Prion promoters say or imply that the laws of natural selection can be contradicted on the basis of their junk science research.
Debunkers of corruption say no one conspires such things. So an explanation of how it works is necessary.
The first fact involved is that science research is extremely demanding, because the unknown does not easily yield information to technological gimmickry. The second most significant fact is that there is an extreme deficiency of abstract understanding among the world's elites including scientists. They do stupid things and make stupid statements which conflict with everything they are suppose to know on the subject.
They react by demanding that no one question their errors. That demand becomes a rule for power elites in all areas. Scratch each others backs and never oppose the group, or the result is to get shoved out.
In science the result is that errors keep getting compounded, while a pretense of normalcy is maintained. Scientists talk around obvious corruptions, as if they could not see an elephant in a bathtub.
Ultimately, there has to be external accountability for corruptions. In science, the public needs to be creating accountability through criticism.
There is a general assumption that peer review improves publications. Supposedly, deficiencies are corrected, and wording is clarified. It's a pipe dream. Purifying is how complex results are destroyed. It's like redesigning an elm tree or improving the Edsil. It isn't an elm tree or Edsil afterwards.
The mentality seems to be developed when students are writing term papers. They quote publications as being fact. So science publications are supposedly quotable fact. They never are. The pretense of fact destroys the process of evolving knowledge.
Having two or three experts modify someone else's work assumes that research should be perfected before being presented to everyone else. So the rest of the scientists have two or three persons doing their evaluating for them. Scientists are all supposed to be capable of doing their own evaluating. Scientists need to see the deficiencies as well as the value in research.
Because of these forces, there has been a devolution of science publications from a complete description with specifics to something resembling a news article or propaganda sheet. The specifics and details are gone, and all one sees is rehashed opinions.
There is no constructive form that peer review could take. Science publications should use their professional staff, which they already have, to evaluate basic standards only. All of the rest of the limitations need to be visible to everyone.
Addendum
In the arguments over global warming, some persons, such as Monbiot, take the position that if it is not peer reviewed science, it is not relevant to the subject. Since the IPCC represents the peer reviewed science, no one can question the IPCC conclusion that humans are causing global warming.
Monbiot's claim that if it is not peer reviewed, it is not science, is an arbitrary and useless way to define science, because science hasn't address many of the questions that people need to address in life. Science cannot prove that water is wet (or anything else), yet people need to know that it is. And science has become at least as corrupt as any power structure in society, which means everything about it needs to be criticized.
To eliminate external criticism of science by pretending that peer review purifies science is about the functional equivalent of preventing anyone from questioning Hitler's power. Nothing is supposed to be above external criticism; and the tendency of the public to not criticize science is one of the main reasons why it has become so corrupt.
Then, no one with half a brain would define science in terms of peer review. Science is a method of proceeding and a standard which shows evidence for basic questions. Peer review is nothing but an extremely questionable method of publishing. Science needs to be criticized through rationality based on objective reality by nonscientists as well as scientists. To replace rationality with peer review is an extremely debased method of railroading fraud onto society.
Perhaps the ultimate, most basic, reason why the claim of peer review is a fraud is because no one can completely represent someone else's reality, and certainly not ill informed persons. It's like the pope determining morality for everyone else. How do you ask the pope if it is moral to cut trees for an oil well, or anything else in question? The persons who shout peer review can't spell the word science, and yet they speak for peer reviewed scientists.
