The Bellesiles of the Right? Another Firearms Scholar (John Lott) Whose Dog Ate His Data

What is it about statistics and guns? Last year, Michael Bellesiles, a historian at Emory College, came under criticism for his Bancroft Prize-winning book, Arming America: The Origins of a National Gun Culture, which argued that gun ownership was far less common during the 18th and 19th century than is generally supposed. His analysis, which was obviously pleasing to proponents of gun control, was drawn from probate records. But Bellesiles was unable to produce all of his data, owing, he said, to a flood in his office. After a committee of three scholars examined Bellesiles' research, they concluded that "his scholarly integrity is seriously in question." Bellesiles resigned from Emory in disgrace.

Now one of Bellesiles' principal critics, a Northwestern law professor named James Lindgren, has turned his skeptical attention to a scholar who is Bellesiles' ideological opposite: John R. Lott, author of More Guns, Less Crime. Once again, the issue is the disappearance of supporting data.

Lott's More Guns, Less Crime is the bible of the national movement to persuade state legislatures to pass so-called "concealed carry" laws, which permit citizens to carry concealed firearms. The book's thesis is that populations with greater access to firearms are better able to deter crime. Some scholars have quarreled with Lott's interpretation, but this controversy is about underlying data. Lindgren and others want to know where Lott got the evidence to support the following sentence, which appears on Page 3 of Lott's book: "98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack."

Initially, Lott sourced the 98 percent figure to "national surveys." That's how the first edition of More Guns, Less Crime put it. In an August 1998 op-ed for the Chicago Tribune, Lott appeared to cite three specific surveys:

"Polls by the Los Angeles Times, Gallup and Peter Hart Research Associates show that there are at least 760,000, and possibly as many as 3.6 million, defensive uses of guns per year. In 98 percent of the cases, such polls show, people simply brandish the weapon to stop an attack."

Alternatively, Lott would sometimes attribute the 98 percent figure to Gary Kleck, a criminologist at Florida State University. In a February 2000 op-ed for Colorado's Independence Institute, Lott wrote: "Kleck's study of defensive gun uses found that ninety-eight percent of the time simply brandishing the weapon is sufficient to stop an attack." But Kleck's research shows no such thing.

Eventually, Lott settled on yet another source for the 98 percent figure: "a national survey that I conducted," as Lott put it in a second edition of More Guns, Less Crime. When asked about the survey, Lott now says it was done by telephone in 1997 and that the data was lost a few months later in a computer crash.

Lott's conflicting explanations naturally attracted suspicion, first from Otis Dudley Duncan, a retired sociologist at the University of California, San Diego, who wrote an article on the matter for the Criminologist, and eventually from Lindgren, the Bellesiles gumshoe, who has been posting his findings online. (Chatterbox is indebted to Tim Lambert, a computer scientist and gun-control advocate at the University of New South Wales, for compiling various documents relating to the Lott case.) When Chatterbox asked Lott about the serial attributions to "national surveys," to three specific polls, and to Kleck, Lott conceded, "A lot of those discussions could have been written more clearly." He said that in the computer crash, he lost all his data for the book and had to reconstruct it, but that he couldn't reconstruct the survey. Lott has been able to produce witnesses who remember him talking about this obviously traumatic event soon after it occurred. But none of these people specifically remember him talking about losing data for a survey he'd conducted. Nor has Lott been able to produce the names of the college students he says conducted the phone surveys in Chicago, where Lott was teaching at the time. (Lott is now at Washington's American Enterprise Institute.)

The only compelling evidence that the 1997 survey ever took place is the testimony of David M. Gross, a Minnesotan who contacted Lott after the controversy spread to various Weblogs. (To date, the only mainstream news organization that's covered the data dispute is the Washington Times, whose Robert Stacy McCain had a piece about the Lott affair on Jan. 23. The Feb. 1 Washington Post examined a bizarre side issue, but we're getting ahead of ourselves.) Gross told Chatterbox, "I have come to the conclusion that I in fact did" participate in the study, "based on some of the details of my recollection." What Gross recalls is that in January 1999—a year before questions were first raised about Lott's data—he attended a talk Lott gave at the Minneapolis Athletic Club. (Gross can pinpoint the date, he says, because he bought a tape.) After Lott's remarks, Gross walked up to Lott and told him he'd figured out, while listening to Lott discuss the 1997 survey, that he, Gross, had participated in that survey. Both the timing and the content, as described by Lott, matched what Gross remembered about the survey, which is the only gun poll he recalls ever participating in. Gross recited his story to Chatterbox with a precision that seemed to reflect both his natural temperament and his professional training as a lawyer. It didn't sound as though Gross could be getting this wrong. But, as the bloggers Atrios and Mark Kleiman have noted, Gross is a pro-gun activist—indeed, a former national board member of the National Rifle Association. Gross was also the founding director of the Minnesota Gun Owners Civil Rights Alliance, and as an attorney he now represents that group in a legal challenge stemming from its appropriation of the name, Citizens for a Safer Minnesota, which previously belonged to a gun-control group that carelessly let lapse its registration with the Minnesota secretary of state. It's odd (though not impossible) that such a bare-knuckled advocate would turn up in a randomly generated survey.

Even if the survey did take place, why should we believe the stated finding? Lott says he repeated the 1997 survey last year. He can't reveal the results, he says, because the publisher of his next book won't let him. But he has shown the results to Daniel Polsby, a law professor at George Mason. Polsby reports that while he won't endorse the methodology—"I have questions about it"—the results were "approximately the same." (This time the percentage was slightly lower than 98 percent—by how much, Polsby won't say.) "John is a very intense man, he rubs a lot of people the wrong way," Polsby told Chatterbox. But "faking something like this would not be John's style."

One type of faking that apparently is Lott's style is the assumption of a fictional identity on the Internet. (This is the piece of the story that the Washington Post's Richard Morin zeroed in on.) Lott has posted Web comments defending his work using a "sock puppet" named Mary Rosh. He was busted by Julian Sanchez, a blogger who works at the Cato Institute, a libertarian think tank in Washington. One posting that Lott has admitted to posting read as follows:

"I had [Lott] for a PhD level empirical methods class when he taught at the Wharton School at the University of Pennsylvania back in the early 1990s, well before he gained national attention, and I have to say that he was the best professor that I ever had. You wouldn't know that he was a 'right-wing' ideologue from the class. ... There were a group of us students who would try to take any class that he taught. Lott finally had to tell us that it was best for us to try and take classes from other professors more to be exposed to other ways of teaching graduate material."

"Lott writes very well. He explains things in an understandable commonsense way. I have loaned out my copy a dozen times and while it may have taken some effort to get people started on the book, once they read it no one was disappointed. If you want an emotional book, this is not the book for you."

Lott says he didn't post the Amazon review; his 16-year-old son did. The "Mary Rosh" e-mail address belongs to his four sons, Lott told Chatterbox—it's derived from their first names—and Lott has used it now and then so that, if he fails to answer a response, it won't be interpreted as "me conceding things." Lott now says the deception was "wrong."

We know Lott invented an online persona. Did he invent the 98 percent figure? Did he invent the survey it purportedly came from? We don't know. "People who are on the gun-control side of the debate," says Polsby, "are hurting on account of Bellesiles. And they want a scalp. John, for one reason or another, is a beautiful scalp to get. For one thing, he's not a terribly good witness on his own behalf." Is Lott the Bellesiles of the right? Chatterbox is not yet prepared to say.

It doesn't hurt for it to be repeated on this thread.

I don't think that most people have actually read Lott's side of this account.

UPDATE: Dr. Lott has given me permission to post the following comments, which I quote:

"Dear Everyone:

Here is a response to some of what has been going on over the web. I have already sent much of this information to people who have already contacted me in person. If Eugene would like to post this on his web site, I must ask that all the e-mail addresses and telephone numbers be removed. If you all don't trust the leg work done by Dan Polsby on this issue since Christmas, you can nominate someone else to go and do it, but I don't think that it is appropriate for everyone from Lambert on to go and harass these people.

Regnery (the publisher of my new book due the middle or end of March) wants me not to release the results from the poll last year. They want me to keep quite about the book until it comes out. As has been reported previously, the survey was done with similar questions in a very similar way to what was done earlier and the results were essentially the same. I will check with the publisher about releasing this data early, but it is still two months before the book is due to be published. In the interim, I am sure that I could arrange it so that interested parties could question the person who keep the survey results as they came in to confirm that we only got one person who said that they had actually fired a gun and that the rest were brandishings.

Here are some of the things that I have done to try to establish a record of events. 1) My wife contacted the bank that we had in Chicago and tried to get copies of bank statements and checks from the period of time. Unfortunately, the bank does not keep copies of statements or checks longer than five years. (If you would like to verify, we talked to Yvonne Macias in the book keeping department at University National Bank.) Lindgren does not accurately report my conversation with him about how I paid people (in that I said that I possibly paid by check), but this information makes that point irrelevant. 2) I asked Sam Peltzman last year about whether the Alumni Association has the e-mail of past students. Sam, who seems to know virtually everything that is going on at the University, told me that they have the e-mail addresses for at most 10 percent of the former students. 3) I had a former alumni and several time co-author, John Whitley, placed in an ad in the Alumni magazine in the December issue to track down the students. I don't know if the ad has appeared but thus far I have gotten no response.

I have given out massive amounts of data to people on the guns and other issues, and I will be happy to do so on the new survey. Data has been given to critics as well as people who have been unwilling to share their own data on other projects. I have given out county, state, and city level crime data to academics at dozens of universities, with data sets ranging from 36MB to over 300MB. I have given out data on multiple victim public shootings as well as safe storage laws. These different data have often been given out before the research is published and sometimes even before it has been accepted for publications. We are not talking about recent events or conversations and there is a question about what is a reasonable time period for people to keep records. There is also a question as to why people have waited so long to ask for this additional information when people have known about the lost data for years.

As to the claims about 'apparently changing positions,' I disagree. I have told people directly (including Otis Duncan) from the beginning that the data were lost. Op-ed pieces and other public statements where I mention these numbers briefly usually do not lend themselves to discussions of the sources of numbers. The fact that David Mustard does not remember exactly when we discussed the survey 6+ years ago does not surprise me given how long ago this was.

Unfortunately, there are many problems with Lindgren's write up. He gives essentially uncritical acceptance of Otis Duncan's discussion of events in 1999. Yet, while Lindgren writes that 'Otis Dudley Duncan raised questions about the 98% figure . . . after exchanges between Lott and Duncan,' Duncan's write-up in the Criminologist news letter failed to mention any such possible discussions. In fact his newsletter piece leaves the opposite impression as he endlessly speculates about what I may have meant about certain statements. My response in the Criminologist also discussed other incorrect claims by Duncan.

As to the attribution of sources, look at the complete context of the quote Lindgren mentions:

Polls by the Los Angeles Times, Gallup and Peter Hart Research Associates show that there are at least 760,000, and possibly as many as 3.6 million, defensive uses of guns per year. In 98 percent of the cases, such polls show, people simply brandish the weapon to stop an attack. -- August 6, 1998, Chicago Tribune and August 14, 1998, Washington Times

References by Lindgren to things like the Linnet Myers piece in the Chicago Tribune to provide evidence that I didn't do a survey or that I have changed my statements over time are simply bizarre. Attached below is an edited down version of the letter that was published by me in the Tribune. Myers used her article to refloat claims such as my Olin Funding, inaccurately reported exactly what the concealed handgun research covered, and claimed that 'others haven't confirmed (my) findings.' I no longer have the original letter to the editor, but as I recall this is just a partial listing of her inaccurate statements. The Tribune was not willing to run a longer letter, though the letter that they ran was quite long.

As to so-called technical problems, I am have always acknowledged that theseare small samples, especially when one breaks down the composition of those who use guns defensively. Even the largest of the surveys have few observations in this category. The attached e-mail that I sent to Glenn Reynolds goes into this more in depth.

'No direct evidence of survey' discussing Lindgren's point-by-point discussion of our conversation

1) 'No funding for the project' I regularly have paid for research myself. Sometimes large amounts of money have been spent, but it is not uncommon for me to spend several thousand dollars. On the paper on multiple victim public shootings, I know that one payment that I made to Kevin, a research assistant to Landes and Posner, was $750. I paid for the special issue of the JLE in 1999 on sentencing myself, and the special issue and part of the conference cost me around $30,000. I have not applied for funds from outside sources over the years.

2) 'No financial employee records' This is not unrelated to the first point. Incidentally, I told Jim that there were 'two' Chicago students. Those students had also gotten others that they knew from other campuses from places such as I think the University of Illinois at Chicago circle (but I am not sure that I remember this accurately). What I told him was that I remembered that one of the two University of Chicago students was a senior.

3) 'calling was done by the undergraduates from their own phones.' most of this next statement is correct except the point about the 'possible' use of checks. But as noted earlier this point is irrelevant in terms of evidence.

4) 'does not remember names' I have had 12 interns and RAs just since I arrived at AEI. This excludes people whose only work was on the survey. I am horrible at names and I couldn't even give you the names for all of these folks let alone people who did something six years ago. All my names and addresses for everything were on my computer when the hard disk crashed.

5) 'no discussions with any samplers'

I had lunch Tom Smith during the fall of 1996. However, while I asked him many questions about surveys, I did not tell him what I was planning on doing because Tom works very closely with gun control organizations.

6) weighting the sample

I did not weight the sample by household size but used the state level age, race, and sex data that I had used in the rest of my book. There where 36 categories by state. Lindgren hypotheses why you can get such small weights for some people and I think that this fine of a breakdown easily explains it. I don?t remember who answered what after all these years, but suppose someone who fired a gun was a elderly black in Utah or Vermont.

7) 'commercially available CD-ROM with names on it. He does not remember where he got it from.'

It is true that I don't have the original CD-ROM. I have a telephone number CD from the end of 1997, but it is not the one that we used. I only picked up the other one on the off chance that I was going to have the time and resources to redo the lost data. The CD did have the features that the earlier one had and was not very useable. I was so rapped up in trying to replace my lost data on so many other projects that I had no thought of going back to what I regarded as a minor project. I had revise and resubmits at the JPE and other journals that had much greater importance and the data for the book had to be replaced.

8) 'Lott does not remember how he drew his sample from the CD-ROM'

Not true. I told Jim that one of the students had a program to randomly sample the telephone numbers by state. My guess is that it was part of the CD, but on that point I can?t be sure.

9) 'doesn't remember the wording of the questions.'

It is also not quite correct to say that 'doesn't remember the wording of the questions.' I told Jim that I don't remember the 'exact wording' of the questions, but I gave him the general outline of the questions.

10) more on weighting

See point 6 above.

11) 'A chapter he had not yet written'

This is not correct. What I had done is write up the section, but I only had a computer file of it. When the hard disk crashed, I only had a hard copy of the book and I had to spend considerable time scanning in the book and correcting the new file. I was unable to replace the lost polling section that I had recently added. I didn't think that it was worthwhile relying solely on memory for different things and I had too much else to do to concern myself with something that wasn't central to the book.

12) 'did not retain any of the tally sheets'

I have looked through some things but I haven't found anything. As Lindgren correctly notes, I have moved three times in the last six years.

13) Sheets versus entry of data into computers

Lindgren has the 'impression' that the students entered the data on sheets. I do not directly recall this part of our conversation, but I would have said that both were done.

I sent Lindgren two e-mails on December 26th. Just so no one accuses me of adding new things in now, one of my e-mails to Lindgren noted: 'I did not take the time to correct or respond to all the issues raised, but I wanted to mention a few points.' Recent e-mails to Lindgren have also already responded to some of these points beyond the e-mail that he apparently posted.

I have not participated in the firearms discussion group nor in the apparent online newsgroup discussions, but what I have done is respond to e-mails. (The one exception are those from Lambert whose e-mail address was placed on my blocked list.) If you all have questions, I will be happy to discuss them, but I am not going be involved in these online groups. My response to Glenn below goes through some of the history of what I heard on this and when I heard it. The bottom line is that you all should not assume that everyone participates in these discussions."

Please Note: Dr. Lott's email also included a clipped letter to the editor of the Chicago Tribune published on June 20, 1999, which should be accessible through public sources, and shortened version of another, slightly older and mostly duplicative email also reviewing his account of events and discussing his new 2002 study. I'd post it all, but this is a heck of a long blog entry. Go ahead and email me at marie@mariegryphon.com if you must have the balance. I'll supply a link if I find any additional comments elsewhere. Posted by Marie Gryphon on January 14, 2003 | Comments (0) | TrackBack (0)

January 10, 2003 Tangled Webs

Defenders of gun rights were rightly unstinting in their criticism of Michael Bellesiles, the former Emory professor who was found to have fabricated evidence for his book, Arming America. Unfortunately, a time may also arrive when they are obligated by their high standards of academic honesty to censure one of their own. A controversy that has festered on academic email discussion lists for some time may finally be coming to a head.

Dr. John Lott, Jr., author of More Guns, Less Crime - a groundbreaking study of the relationship between concealed-carry laws and crime rates - has been accused in some detail of fabricating a survey in order to support his oft made claim that merely brandishing, rather than firing, a firearm will scare off an attacker 98% of the time.

Dr. Lott made this claim in the first edition of More Guns, Less Crime, published in 1998, citing only "national surveys." He wrote at page 3, "If national surveys are correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack." According to critic Tim Lambert, Dr. Lott has made reference to the 98% statistic at least 48 different times.

During this same period, other commentators were also citing a 98% figure. They were apparently attempting to cite a study on defensive gun use published by Florida State University criminologist Gary Kleck. However, they were forgetting that the 98% figure cited by Kleck included defensive gun uses involving warning shots and missed shots at an attacker as well as mere brandishing cases.

When confronted with a critique of the 98% figure, individuals including C.D. Tavares have explained their misinterpretation of Kleck’s work and apologized. Dr. Lott, though, began attributing the 98% figure to a national survey he now says he personally conducted in the year 1997. In the second edition of More Guns, Less Crime, he offers the following alteration on page 3: "If a national survey that I conducted is correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack." (emphasis added)

Dr. Lott’s accusers suggest that he was unwilling to admit that he had misinterpreted Kleck’s study, opting instead to credit a never-before-mentioned national survey of his own for the 98% figure. Wrong as it would have been, one can certainly imagine an author of a 321-page book convincing himself that a small sleight of hand on page 3 would never be questioned. If the naysayers are correct, though, Dr. Lott significantly worsened matters for himself in 2000, when he responded in The Criminologist to criticism from U.C. Santa Barbara’s Otis Duncan with a detailed description of the survey Lott supposedly conducted in 1997, and its findings. Dr. Lott apparently discussed the details of the survey over the telephone with Duncan as well.

Having reviewed credible-sounding critiques of Dr. Lott’s description of his survey by Lambert and Duncan, Northwestern University Professor James Lindgren undertook to investigate whether the survey actually took place. It is Lindgren’s report on his efforts, dated December 24, 2002, that is most concerning.

Dr. Lott claims that his telephone survey was conducted over three months in 1997, and garnered responses from "2,424 people from across the United States." Because many called parties fail to answer calls or refuse to respond to survey questions, it is certain that thousands more calls would have been made in order to generate 2,424 responses.

But Lindgren reports that Dr. Lott had no funding for this survey, and says he covered the expenses out of his own pocket. Dr. Lott claims to have used student volunteers to help him place thousands of phone calls, but lacks any records listing student participants, and cannot remember the name of even a single volunteer. Dr. Lott says he had the students make these phone calls on their own home telephone lines, reimbursing their long distance charges from his personal account, but that he cannot prove this because he discards his cancelled checks after three years. Dr. Lott apparently told Lindgren that he did not recall discussing this survey project with any colleagues at the University of Chicago.

As for the data itself, Dr. Lott apparently told both Lindgren and Duncan that he cannot produce the data because it was lost in a computer crash in 1997. He apparently explained at least once that survey data was entered directly into students’ computers, and then electronically transferred to his computer during the survey process, but apparently no student retained copies of the data allowing him to reconstruct the survey following his computer crash. He has also apparently suggested that some handwritten data collection may have occurred, but that any handwritten survey results must have been inadvertently discarded when he moved out of his office at Yale University.

Dr. Lott is an exceptionally bright man who has conducted some of the most important research to date on criminological issues related to gun control. If he did indeed conduct the 1997 survey – as I hope he did – he should proactively work with others to find at least one of the graduate students who assisted him, obtain his old bank records and their old telephone records. In short, he should take the time right now to set this controversy at rest.

If Dr. Lott’s accusers are correct, immediate action is even more important. In very little time this issue will find its way into the mainstream media. If Dr. Lott has lied to his colleagues he must now tell the truth, difficult as that might be. The alternative will surely transform a journalistic footnote into a media circus.

I also looked up the CHicago Tribune letter

Chicago Tribune

June 20, 1999 Sunday, CHICAGOLAND FINAL EDITION

SECTION: MAGAZINE; Pg. 4; ZONE: C; LETTERS TO THE EDITOR.

LENGTH: 684 words

HEADLINE: GUNS AND CRIME

BODY:

The article accompanying "Anne, Get Your Gun" (May 2), discussing my book "More Guns, Less Crime" (University of Chicago Press, 1998), made several inaccurate claims.

Despite the claims in the article, my research looked at much more than just the "impact of laws that allow guns to be carried outdoors." My book analyzed FBI crime statistics for all 3,054 American counties from 1977 to 1994 as well as extensive cross-county information on accidental gun deaths and suicides. This is by far the largest study ever conducted on crime, accidental gun deaths or suicide. I examined not only concealed-handgun laws, but also other gun-control laws such as state waiting periods, the length of waiting periods, the Brady law, criminal background checks, penalties for using guns in commission of crime and the impact of increasing gun ownership. The only gun laws that produced benefits were those allowing concealed handguns. The evidence also strongly indicates that increased gun ownership on net saves lives.

More disappointing were inaccurate references to the funding of my research. The claims previously floated by gun-control groups like Handgun Control were found by the Tribune's own Steve Chapman to be false (Aug. 15, 1996). Chapman pointed out that not only was the Olin Foundation "independent" of the ties the Sunday Magazine article discussed, but also that the "foundation didn't (1) choose Lott as a fellow, (2) give him money or (3) approve his topic."

The article's claim that "others haven't confirmed (my) findings" is bizarre. To date, I have made the data available to academics at 37 universities, from Harvard to Berkeley. Everyone who has tried has been able to replicate my findings, and only three have written pieces critical of my general approach. Although the vast majority of researchers concur that concealed weapons deter crime, not even those three critics have argued that more guns cost lives or increase crime.

-- John R. Lott Jr., University of Chicago

22 posted on 01/17/2003 1:28 AM EST by Washingtonian (May be people should read Lott's response)

[ Post Reply | Private Reply | To 1 | View Replies | Report Abuse ]

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.