FOIA is not enough. Why not legally mandate transparency in climate research? A Modest Proposal…

Not all scientific research is equal, in terms of its probable impact on humanity. If one is studying poison dart frog species in tropical rain forests, getting a number wrong or arriving at an incorrect conclusion will generally have very little impact on the life of somebody living in California, India, China, or for that matter, in the tropical rain forests in question.

On the other hand, medical research has a profound impact on us all and has a long history of abuse, both deliberate and accidental. From the egregious claims of snake-oil salesmen whose very name has come to be the universal metaphor for “science” subverted to special interest to modern cases of confirmation bias and manipulation of data in e.g. drug testing, because of the potential for profit from medical science and technology it has proven to be necessary to defend the public against bad science. All medical research at this point is strongly regulated at or before the point where the rubber meets the road and actual patients might be adversely affected or killed by bad science or self-serving deliberately manipulated science.

Some of the key standards of this regulation of research include transparency and reproducibility, as well as the near-universal use of double-blind experiments to prevent the pernicious advent of confirmation bias, backed up by the threat of liability if the research process is deliberately subverted because of any sort of vested interest (including the simple desire to “be famous”, or “win tenure”, or “keep one’s grant funding”).

Engineering is a second place where doing science (in this case applied science) badly is dangerous to the public weal. If a bridge, a car, a space shuttle is designed poorly or carelessly, society ultimately pays a significant cost. In this case the needs of engineering firms and private individuals for protections of intellectual property are carefully balanced against the need to protect society. Consequently, the building of bridges, cars, and space shuttles — with or without proprietary components — is subject to oversight, inspection, and again, legal liability.

Climate research has long since passed from the realm of being a tiny discipline with a handful of researchers whose mistakes had almost no impact on humanity to being an enormous, publicly funded research machine that has a huge impact on the public weal. Whether or not you agree or disagree with the Catastrophic Anthropogenic Global Warming (CAGW) hypothesis, there is no denying that it has a huge impact on people all over the world. Quite literally every human on earth is currently at risk either way relative to the conclusions of what is still a relatively small community of scientists with a remarkably homogeneous point of view.

Whether or not these scientists are honest — do or do not sincerely believe their own conclusions is not an issue (any more than it often is in the case of medical research or large scale engineering projects); what is important to the general public is that the scientists at this point have a clear interest that potentially conflicts with their own. At risk (to the scientists) is: loss of (enormous) funding; loss of prestige; loss of political power and influence. Many of them have staked their entire reputation and career on stating conclusions as near-certain scientific fact that have a multi-trillion dollar price tag to society associated with their conclusions — however objective and well-intended — turning out to be correct or incorrect.

This dwarfs the potential damage that could be done even by unscrupulous drug companies, by medical researchers seeking to make a name for themselves, by incompetent physicians and all of the other scientific activities in the general field of medicine that are so tightly regulated. It dwarfs the damage that can be done by a faulty braking system in an automobile, by whole cities of buildings that cannot (as it turns out) withstand earthquakes, by faulty O-rings in space shuttles. It dwarfs even the damage that can be done by unregulated banking systems leading to global financial collapse that lower the standard of living “suddenly” on a worldwide scale. It is larger than the combined probable damages from all of these activities put together over any reasonable time scale.

I must emphasize that from the public point of view, this risk is in some sense symmetrical. Taking global steps such as creation of an entire cap and trade financial instrumentation in order to combat CAGW is without any doubt enormously expensive and the loss in this case is certain and immediate. However, as proponents of CAGW theories are quick to point out, society is required to make some kind of wager regardless because the cost of doing nothing if their hypothesis is true may also be extreme.

However, it must be carefully noted that one is balancing a multi-trillion dollar and immediate certain cost against a future possible cost that is by no means certain. It is simply a matter of responsible governance that the cost-benefit of this risk be soberly and, above all, openly assessed. Furthermore, both regulation and liability are absolutely necessary — indeed, long overdue — in any scientific endeavor that has long since left the ivory tower of pure research and become the basis for such far-reaching policy decisions.

Unfortunately, climate research that not only has impacted, but has led the way in the public debate and scrutiny that should correctly attend the collective expenditure of vast amounts of wealth that could otherwise be put to better use has not, thus far, been conducted in an open way. Critical data and methodology have been hidden and treated as if they were proprietary by the scientific researchers involved, in spite of the fact that the data itself has rather often come from governmental organizations or is the direct product of research funded entirely by public research grants, as in most cases is the published work itself. That this has occurred, and continues to occur, is not at issue here — the evidence that this has occurred and continues to occur is conclusive and indeed, ongoing. The simple fact of the matter is that whatever the truth of the hypothesis, the methodology and data used to support it are largely hidden, hidden well enough that it is routinely true that they cannot easily be merely reproduced by a third party, let alone the conclusions be intelligently and critically challenged.

In the ivory tower it is not unreasonable or uncommon for this sort of practice to exist, at least for a time. Scientists’ only “commodity” of value at a research University or government organization is their ideas and their research work, and theft of both is far from unknown. It is perfectly reasonable for individuals to initially hide their research goals and methods from everyone but perhaps a small set of trusted collaborators until they are proven to the satisfaction of the researcher, lest some key discovery or idea be co-opted or pre-empted by a competitor.

Even in the ivory tower, such obfuscation is supposed to — and typically does — come to an end when a work is published. Publication is the final goal of the research process in the ivory tower of the University (and often in a government laboratory) and in both cases there is a careful separation between work that is done with an eye to obtaining a patent or protected intellectual property and work being done (especially work being done with public funding) for open publication with no related rights being preserved.

Once a scientist has published in the latter case, it is expected that they will make both methods and data public upon request and invite others to reproduce and either verify or criticize the methods and any results derived from them. Anything less is a corruption of the scientific process that — when it works correctly — eventually rejects error and advances the sound. This process is often imperfect — even with levelling/protecting structures such as “tenure”, there are differential rewards to scientists based on how well they keep key ideas, methods, or even data back to maintain an advantage over their competitors and it is not uncommon for only part of the story to be told in any given publication, especially early on in the development of a new idea.

Although one can therefore understand the origins of this sort of reticence and inclination to hide research methods and data or share them only with carefully selected collaborative colleagues, and although one might even still respect this right up to the point of publication in climate research, in the specific field of climate research the public stakes are too high for this practice, however common it might or might not be in the study of poison dart frogs or the physics of graphene, to be tolerated.

The conclusions of modern climate research are almost exclusively based on published results such as the (now infamous) “Hockey Stick” graphs produced by Mann, et. al. and data sets such as HadCrut3. HadCrut3 itself is currently made readily available, but only as processed results obtained by some means from streams of raw data that are not. It is, in fact, essentially impossible for a third party to take the actual data used in the current HadCrut3 snapshot published by the Met Office at the Hadley Center, feed it to the actual code used to generate the processed data, and verify even the very limited fact that the data and the code do indeed produce the same result when run on different computers, let alone that the methodology used to produce the result from the data is robust and sound.

It must once again be emphasized that public policy decisions that have been made, are being made, and will be made in the future based on the raw data and methodology used will cost every living person on earth on average several thousand dollars, at least. Again this is stated without prejudice concerning whether or not the published temperatures are, or are not sound, or whether CAGW is, or is not, a well-supported scientific hypothesis. If it is true and we do nothing, it will cost thousand of dollars per living person and many lives over decades. If it is false and we spend money like water to prevent it anyway, it will cost thousands of dollars per living person and many lives over decades; in addition, it will do incalculable cost to the credibility of “the scientist” in the minds of the public that further amplifies this monetary damage by altering the profile of government funded research and the level of trust accorded to all scientists in the public eye.

I am writing this article to call for new legislation to address this issue, legislation that creates direct oversight for climate researchers whose work directly impacts the decision making process directing this enormous but unavoidable gamble. I am writing this as a citizen that is already paying for decisions based on the “certain” conclusion of CAGW — if this conclusion is certain, then it is certainly true that it can be transparently certain, with the entire process used to arrive at it right back to the original raw data open to public and scientific scrutiny not only by those that agree with it but by those that honestly disagree with it or merely have doubts that it is true and would like to verify it for themselves.

I would suggest that this legislation be soberly and conservatively drafted so that it in no way hinders climate researchers from carrying out their research but adds the following requirements that must both precede and follow any published result that impacts the decision process.

a) All numerical code, and input data (that is, the raw input data including any that is for any reason available from one’s source but not included in the computation, along with the provenance of all the raw data) used in arriving at some conclusion must be openly published in an immediately usable form and made readily available to anyone in the world as of the date of publication in any journal, public presentation at conferences or workshops, or publication or inclusion as a reference in policy document such as IPCC reports.

This legislation shall apply to publications based on proprietary data as well as data from public sources. The immense cost of the public decisions based on such publications and the risk of corruption of the results cherrypicked or data that might have been altered in hidden ways by vested interests is too great to permit data to be used or selected from any source that cannot be checked in its entirety, including the data that is left out.

Note well that this precise measure is indeed needed. The FOIA has already proven to be inadequate to compel the release of code and data used to generate datasets such as HadCrut3 or the “Hockey Stick” or “Spaghetti Graph” curves that currently support many of the conclusions of climate researchers.

Note also that this is hardly a burdensome requirement. It is sound practice already to carefully provide provenance and good organization for one’s raw data, to provide sound backup and revision control for the computer code used to process the data that permits “snapshotting” of the code actually used to produce a result, and to archive both for any given publication in case one’s methodology is ever called into question. The only additional requirement this imposes is to set up a website and put the data and code snapshot there with a short piece of documentation accompanying it that frankly will be of as much benefit to the researchers in the long run as it is to anyone seeking to download code and data to check results. Nowadays the cost of this is so low as to be “zero” and in any event is trivially within the means of any grant funded climate research program that almost certainly is already using one or more web servers to disseminate both results and data.

b) The establishment of a board of governance for the science with the specific and narrow purview of addressing abuses of the open scientific process. The need for such a board, and the need to staff it with people who are completely disconnected from climate or environmental research or any political organization or corporate organization with any possible interest in the outcome is clearly demonstrated in the occult Climategate conversations where it is revealed that certain researchers working in the field are far more concerned with “causes” and “winning the PR war” than with the science and are willing to deliberate tamper with data and methodology to hide results that confound their desired conclusions or to directly and deliberately subvert e.g. the objectivity and independence of the journal review process to suppress competing points of view right or wrong as they might be! The only place such discussions should occur is openly, in the literature itself, in the form of critical counter-articles or published comments, not in behind the scenes efforts to discredit editors or have them fired.

Note well that implementing provision a) will make tampering with data or methods far more difficult, but not (as evidence from medical research abuses reveals) impossible, and nothing (so far) seems to have worked to maintain any semblance of fair play in the public debate — on both sides of the climate issues. Even if the only sanction used by the oversight board is public censure and the probable elimination of future funding, those are probably enough in a world where one’s scientific reputation and ability to continue work are one’s greatest treasure.

c) The establishment of personal liability for any work that is published wherein it is later shown that the researcher did knowingly and deliberately manipulate data or methods so that their arguments lead towards a predetermined end (confirmation bias, cherrypicking data) without openly indicating what was done and why in the publications. Again, there is ample precedent for such liability (and the corresponding governance and oversight) in all scientific and technical endeavors that directly impact on the public weal, in particular in medical research. There should be considerable freedom under this rule to make honest mistakes or to pursue unpopular or popular conclusions — one of the major purposes of provision a) above is that it should ensure that there should never again be a good reason for sanctioning a researcher after the fact of publication by guaranteeing transparency — but just as would be (and historically, often has been) the case when it is determined that a published medical study where the researcher or corporation sponsoring the research “fudged” the data and as a result patients died or suffered losses makes the researchers and/or sponsors legally liable for the damage, there need to be at least limited liability and public sanctions in climate research to provide a strong disincentive to academic dishonesty or the protection of interests that, in the end, are not strictly the pursuit of scientific truth.

Needless to say, no researcher can afford to pay the true liability cost of a mistake in a ten-trillion dollar public policy decision driven by their work, but actual overt dishonesty and work performed with hidden/vested interests cannot be allowed to proceed unpunished, either. This has proven to be absolutely true in countless other, far less costly, realms of scientific, economic and sociopolitical endeavor — wherever an unregulated marginal advantage exists to be exploited by an unscrupulous individual, sooner or later such an individual shows up to exploit it. There is far too much at stake here not to protect the public good.

To conclude, much of what happens on this and other blogs, e.g. Climate Audit, is ultimately fruitless. Much energy and time is expended discussing this abuse or that abuse of good scientific methodology without any real hope of putting it right or on e.g. FOIA requests and other straightforward (but openly obstructed) attempts to simply understand how various numbers that purport to show anomalous warming were generated. The place progress has been made is primarily in a very few, but extremely significant cases (e.g. the deconstruction of the infamous “hockey stick” graph that at this point is completely discredited in spite of having dominated public discourse and public policy decisioning for over a decade) when access has been obtained to raw data and actual computational code. Mistakes that may well have misdirected hundreds of billions of dollars of public money could easily have been averted by legislation like that suggested above mandating a completely open and above-board process.

Those who advocate the CAGW hypothesis should welcome such legislature — if they have nothing to hide and their results can indeed convince “97% of scientists” as claimed, then they should make it easy for those scientists to not just read their published results (working from hidden data) but to be able to verify how their work advances from the hiddent. They, and their “cause”, can only benefit from a completely data-transparent process if their conclusion is correct.

Advocates are mistaken in treating the CAGW hypothesis as a public relations problem to be solved or a cause to be fought for in the first place (terms bandied about in a most disturbing way in the Climategate communications however they were intended), often discussed as alternatives to the far simpler option of publishing papers that address and attempt to refute competing claims, ideally acknowledging points where they might have a point. Scientific discourse has no room whatsoever for either of these as the popularity of an idea is irrelevant to its probable truth, and “causes” smack of either political or religious thinking, both of them ultimately irrational in different ways. An idea is held to be correct when it is well-supported by a mix of good fundamental science, reliable data, and openly reproducible, openly critically examined methods, and any good scientist will always bear in mind the fact that however much they “like” their own conclusions, they could be wrong.

There is little that is certain in science, and good science is honest about the uncertainties even when — perhaps especially when — there is a lot at stake.

Yes, this is a high standard of truth, one that will take time to achieve, especially in a field as complex as climatology, where many results are obtained by means of rather complex computational or statistical methods that rightly should be closely scrutinized as it is all too easy to either “lie” with or be honestly misled by an incorrect model (again this happens so often that we have a whole terminology such as “garbage in, garbage out” to describe it) or incorrect statistical analysis — the latter especially is a bete noire in sciences (and medical research) with far less impact on the public purse than climate research.

Nowhere are the stakes higher; nowhere is the oversight lower and the methodology ultimately more deliberately hidden than it currently is in climate research. And why? If CAGW is indeed true, a truly open process of research and decision making should be openly and even enthusiastically embraced by supporters of CAGW, because it will equally well compel skeptics of CAGW to provide full access to their methods and data and reveal possible vested interests. How often have we all heard the litany “anyone who criticizes CAGW is supported by the oil industry” (and seen scurrilous allegations to that effect in the ongoing discussion revealed by Climategate). Well, here is an opportunity to provide objective oversight and liability in the unlikely event that this is true — but in both directions.

I would therefore strongly suggest that a sympathetic advocate be found who would sponsor the a-b-c rules above as actual legislation to govern all climate research, publicly funded or not, that is actually used to influence large scale public policy decisions. Indeed, I would call on all climate researchers and journal editors to enforce “voluntarily” compliance with rule a) whether or not such legislation is ever written! Climategate 1 and 2 documents have clearly, and shamefully, revealed that many climate researchers currently knowingly and deliberately refuse to make either data or code/methods publicly available even when proper FOIA requests have been made. Journals such as Nature or Science have a deep responsibility to ensure transparency in any papers they choose to publish that have such a huge real cost and impact on public affairs either way their hypotheses are ultimately resolved. Papers published in climate science that specifically address the issue of global warming, including papers published in the past, should be given a reasonable opportunity to provide provenance and access to raw data and methods and, if that provenance is not forthcoming for any reason, the papers should be publicly repudiated by the journal and withdrawn.

Perhaps we could call it “Mcintyre’s Law”, since few people have fought this battle more frequently, and more fruitlessly in far too many cases, than Steve Mcintyre.

From the comments:

********************************************EXCERPT*******************************************

crosspatch says:

December 2, 2011 at 7:37 pm

http://chriswilliamsonlabourleader.blogspot.com/2011/11/labours-climate-change-pledge.html
http://blog.tides.org/2010/06/23/media-misinformation-impacts-entire-progressive-agenda/
http://www.commonwealinstitute.org/archive/framing-progressive-agenda

There are thousands more such links. “Climate Change” is a key issue for a specific political agenda and it is used as a “hook” to get people to buy into more government control of their life. From government management of what lightbulbs they use to government mandates of what goes in the fuel for their cars to government regulations that redistribute industries globally.

This is not a scientific fight and if it is fought on that level, one is bringing a knife to a gunfight. You will be ignored or worse, you will be kowtowed too but in the end ignored.

What we need is someone to stand up and say with full authority of government that the misinformation they are putting out is simply not true. THAT will shut it down.

This is the fundamental problem right here. As long as this is the case, as long as there are truckloads of money on the line, there will be the temptation to keep the money-river flowing, period.

Whether or not these scientists are honest — do or do not sincerely believe their own conclusions is not an issue...

I disagree. I believe that is the current issue. I do not believe climate scientists are honest. They have shown, as a group, a startling lack of character and integrity. They are not to be trusted. "Climate scientist" is now right up there with used car salesman, lawyer, bridge salesman, etc. They have absolutely no-one to blame for this public image but themselves. No-one.

Many of them have staked their entire reputation and career on stating conclusions as near-certain scientific fact...

Exactly. It is precisely these "scientists" that will now go to any lengths to protect their source of funding, their reputations, and their agenda. Any lengths. Without these, what are they, what have they got? Nothing. This is not an environment conducive to scientific honesty.

If they want the public to start taking "climate science" as anything more than a punch-line to a joke they have to make some radical changes:

Publish (make available via electronic means) all climate data. That is, not a subset, not a cherry-picked set, all of it.
Publish raw data, not adjusted/filtered/processed data. Raw input data.
Publish all adjustments, corrections, scaling factors, etc.
Publish all algorithms for their models.
Perform and publish statistical analysis on the input data, including confidence intervals, accuracies, and precision.
Perform sensitivity analysis on climate models. (eg. via monte-carlo statistical runs) Determine how sensitive existing models are to fluctuations in input data. Publish results.
Compare and contrast model sensitivity to input data accuracies. (hint, you "scientists" probably aren't going to like the result of this one) Publish results.
Compare and contrast model sensitivity to input data adjustments. (hint, I know you "scientists" are really not going to like these results)
Perform studies with climate models using historical data, analyze the models' ability to predict recent past climate based on earlier data. Publish the results.
In short, publish everything you do, data and algorithms. More to the point, publish what you don't do/use. What data is excluded, and why. What modeling edge conditions are explored, and what is not.

However, the entire "climate science" research community's image is so tarnished, I'm not sure I would trust them to do #2. Unless you go out and collect the data yourself, I'm not sure I trust any of these shysters.

But, if they did these ten things, other independent scientists and computer-science types were able to reproduce their results, and they were able to justify their adjustments and data set selections... Then I might start believing their results are genuine.

However, from what little truth has leaked out, I doubt an open and honest appraisal is going to be able to draw many conclusions. I'd be willing to bet (a good dinner at least) that #7 and #8 are going to give them fits. Again, from what little truth has seeped out around the edges... It seems some (many?) of the models are sensitive to input conditions and can "diverge" off into a wildly unrealistic outcome based on small input changes. Similarly, many of the "trends" shown in the model are actually smaller than the um "corrections" made for land-use changes, population changes etc. In other words, existing models are basically useless.

The Prof responds to the comments....excellent blog by the way:

*******************************EXCEROT****************************************

Robert Brown says:

December 3, 2011 at 7:29 am

“I would never call Mr. McIntyre’s efforts, “fruitless”, many have already borne fruit, and the seeds from those fruit will themselves bear fruit in the future.”

All too often fruitless. I was referring to M&M’s well-documented efforts, over many years, to get access to the actual hockey stick proxy data and code, and to Mcintyre’s ongoing effort to gain access to still more actual code and data in the case of many other questionable publications. These efforts are so far fruitless, and they each require enormous energy to pursue — I literally don’t know how he does it, as he has to fight for years to get something that he shouldn’t even have to ask for.

That’s the fundamental motivation for Mcintyre’s Law. He shouldn’t even have to ask for access to the code and data and methods used to produce graphs and conclusions that affect the directed use of trillions of dollars wordwide! Indeed, this is so obvious that it is manifestly insane that he has to actively fight against researchers who don’t want to expose their work at a level that might reveal the hidden thumb on the scales. I can understand it — that thumb could be deliberate or it could be sheer incompetence in statistics (nobody understands statistics, making it so very easy to either lie with it or simply make a horrible mistake with it) but either way the discovery of a serious problem could be a career-ender. However, the stakes are too high for the current business-as-usual hidden methodology or methodology described only in crude terms that do not permit the precise numerical reconstruction of results to continue.

rgb

There are a LOT of problems with *some* aspects of this proposal, but I’ll just mention three:

1) Probably the biggest difficultly in improving the peer review process in general is finding competent reviewers: people well enough versed in any specialty to perform effective review of new findings usually prefer to be doing something more immediately relevant to their own careers - it’s *really* tough to put together “blue ribbon” panels to investigate *specific* subsets of results.

2)And even then, if you don’t like the results of (say) the NRC’s analysis of Mann’s “Hockey Stick”, you get someone who is highly credentialed, like Edward Wegman, to set up another group, which reaches a different conclusion.

And then who you do get to arbitrate THAT dispute?

God?

3) Even assuming that you *could* set up such a vetting process in the US, science is an *international* effort, and scientists in other countries, their non Governmental organizations, and their Governments are *highly* unlikely to recognize the authority of any such nationally based “legal system” to vet scientific effort, instead they will regard it generally as an attempt to “politicize” science, and specifically as an attempt to intimidate scientists into avoiding lines of inquiry likely to become politically “incorrect” depending on the political party in power at the moment.

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.