« Department of Unintended Consequences | Main | Libertarians and Their Admirers »
October 13, 2006
The Lancet Study
Well, this is silly. It turns out that the editor of The Lancet is -- get this! -- opposed to the Iraq War! No! And because he gave voice to those opinions, Jane Galt thinks he's "sowed doubts" about the study. Which I guess is true. Of course, it would also mean that I can no longer take seriously anything Jane writes on economics, because she's committed to a particular economic and political vision that I don't share. And she should no longer read me, as I've got my own biases. It's a pity because her doubts on the study are, I think, valid, even if I don't agree with them. Her attempts to discredit it by attacking the Journal's editor for holding opinions aren't. As for the substantive analysis, I'll stick with the arguments of Daniel Davies, who's actually dug into the study and returned impressed:
The results speak for themselves. There was a sample of 12,801 individuals in 1,849 households, in 47 geographical locations. That is a big sample, not a small one...And the results were shocking. In the 18 months before the invasion, the sample reported 82 deaths, two of them from violence. In the 39 months since the invasion, the sample households had seen 547 deaths, 300 of them from violence. The death rate expressed as deaths per 1,000 per year had gone up from 5.5 to 13.3.
Talk of confidence intervals becomes frankly irrelevant at this point. If you want to pick a figure for the precise number of excess deaths, then (1.33% - 0.55%) x 26,000,000 x 3.25 = 659,000 is as good as any, multiplying out the difference between the death rates by the population of Iraq and the time since the invasion. But we're interested in the qualitative conclusion here.
That qualitative conclusion is this: things have got worse, and they have got a lot worse, not a little bit worse. Whatever detailed criticisms one might make of the methodology of the study (and I have searched assiduously for the last two years, with the assistance of a lot of partisans of the Iraq war who have tried to pick holes in the study, and not found any), the numbers are too big. If you go out and ask 12,000 people whether a family member has died and get reports of 300 deaths from violence, then that is not consistent with there being only 60,000 deaths from violence in a country of 26 million. It is not even nearly consistent.
There may indeed be perfectly reasonable criticisms of the study. One of them is not that the editor of The Lancet disagrees with The Iraq War, or believes the conclusions of his journal's research.
October 13, 2006 | Permalink
Comments
I certainly think that people with a particular opinion can provide good insight, and even good facts about a subject even if I don't share that opinion. However, when it seems like the facts might not be so accurate, looking at the motivations of the person can provide additional information. And if a result is 'surprising' but matches up with the political motivations of the person providing that result extra skepticisim is warrented.
As an example, although I disagree with you on health care, I find your opinions and analysis very worthwhile. If though, you produced a study that claimed that if we adopted your health care propossal we would live twice as long and our health care costs would be a small fraction of what they are now, I would find those results 'surprising' and expect that your bias had probably affected the study. On the other hand, if you published a study showing that free market private sector health care would achieve those same results, it would still surprise me but I would be more accepting of it since you obviously would have rechecked your work multiple times before publishing such a thing.
Posted by: Dave Justus | Oct 13, 2006 2:27:30 PM
Fair enough, Dave. If the results of the study were so far off what an expert in the related fields would expect (I don't count failing the gut test of the lay public), some skepticism would be expected. But your knowledge of the suspect motivations of others should only motivate you to show how the study is flawed. It should not substitute for analysis.
The fact that a smart, principled conservative like Daniel Davies thinks the study holds water speaks volumes to me. I don't know enough statistics to analyze the study myself, but I have some trust in people who support work that may be against their interests or the interests of their allies -- just as you would be more accepting of anything Ezra published that supported free market, private sector health care.
Posted by: Mary | Oct 13, 2006 2:39:59 PM
60,000 or 600,000 - either number is way too many civilians being killed by "freedom" from Saddam.
Posted by: CParis | Oct 13, 2006 2:43:12 PM
Dave,
There's a distinction here that you may not be aware of. The Lancet study was peer-reviewed. That means it was submitted to a panel of experts in the field of survey research, and would not been published had it not passed muster. The biases of the editor, whether real or perceived, have been controlled for by the peer review process.
If Ezra had conducted his own study, and then posted it here without review, you'd be right to be skeptical for all the reasons you cite. But if he'd circulated it among a bunch of experts and had them declare its method and findings valid, I think you'd have to suspend at least some of your doubt, wouldn't you agree?
That said, can we still say with a straight face that the critics, spin doctors and administration officials (each with their own biases and agendas) know something that the rigors of survey research and peer review somehow missed?
Posted by: Rick | Oct 13, 2006 3:08:38 PM
To me, the number that most debunks the study is 61%- the percentage of Iraqis who think the overthrow of Saddam was worth it, according to the last poll taken in September.
No way, no way in HELL they'd be saying that if 600,000 people had died.
Posted by: Adam Herman | Oct 13, 2006 3:10:58 PM
I of course didn't address whether this study was good or not, only Ezra's claim that the motivations of the author are irrellevent, or that what Jane Galt said was not valid.
I have my doubts about the study, for things that Davies doesn't mention. While the sample is not small, that is only one factor in determining if the sample is good. The study itself, by its huge confidence interval admits that their sample isn't all that great. As an illustration, a random phone survey sample of 1000 people will almost certainly reach better results than an uncontrolled internet survey with 100,000 respondents. It is also the case that sample size depends on the frequency of what happens.
Davies does offer up some good questions that critics of the study need to answer. Unfortunately, their isn't enough data released by the authors of the study to answer those questions. When only the results are released, not the data and the methods, than 'proving' it wrong become impossible.
One red flag for me is that the study claims that 80% of the death claims were backed up by death certificates, but the numbers they reached seem well higher than the official death counts. One would expect that the official counts would be 80% of what the study claims, this clearly is not the case.
As CParis points out though, in many ways this study does not really tell us anything important. How many died before and after does not give any information about the worth of the enterprise. More people died in the Civil War than in the Spanish American war but few people would argue that the Spanish American war was a more just, or better war than the civil war.
Posted by: Dave Justus | Oct 13, 2006 3:11:39 PM
If the Lancet editor was one of those who were engaged in the peer review of the article, Galt might have a point. Since the editor wasn't a reviewer, it does not appear that she has a point.
Quite frankly I've pretty much ignored Galt, since I determined that she was something of a nut.
Posted by: raj | Oct 13, 2006 3:11:43 PM
Daniel Davies isn't a conservative, Mary; maybe you have him mixed up with Daniel Drezner?
But I don't see that Galt's objections are valid. She says that no one has any idea whether the study is right or wrong. But the study's authors are respectable scientists, using the methods of their discipline, and the study was peer reviewed by people whose political leanings we don't know. Aren't they a lot more believable than Galt here?
Dave, your point is well taken, but keep in mind that the editor of the Lancet is not the author of the study nor the reviewer. And as Mary says, it should motivate you to find flaws, not to assume there are some.
Posted by: Matt Weiner | Oct 13, 2006 3:13:47 PM
Dave, I hadn't seen your last post when I posted.
One red flag for me is that the study claims that 80% of the death claims were backed up by death certificates, but the numbers they reached seem well higher than the official death counts.
Davies addresses this; a lot of death certificates may not make it to the central government.
As Davies has said, if you ask a large number of families if they've had a member killed in the last few years, and a quarter of them say yes, it's pretty much ironclad that lots and lots and lots of people are dying.
Posted by: Matt Weiner | Oct 13, 2006 3:18:20 PM
I think the inference to be drawn from the furor over these figures is that even in the minds of the war supporters 660,000 deaths would mitigate the justness of the invasion/occupation. And if in the minds of the war supporters, 666,000 doesn't do it, I'd like to know what number would--how many lives would be too big a price to pay? It's a crime that so few people have even attempted to get an answer to this question. Dave Justus, since you're here, I'll ask you (I assume you support this war). How many people would have to die before you were to conclude that the war wasn't worth it?
Posted by: david mizner | Oct 13, 2006 3:18:28 PM
Y'all, forgive me, but . . . who do you think picks the peer reviewers? The PR fairy?
Actually, I wasn't trying to cast aspersions on him, particularly; I just think he's funny. And tactically not very sensible. But anyone who's worked as an editor knows how to get the "scientific" answer he wants -- if he really wants it.
Posted by: Jane Galt | Oct 13, 2006 3:23:36 PM
Most educated people have a political bias. But if merely HAVING a political bias AUTOMATICALLY makes one's work suspect, then Jane must automatically reject -- oh -- Katherine Harris in the 2000 elections, or the WSL editorial page, or Bush on just about any subject.
Of course, that's the thing about statistics. It removes bias. So Jane is basically saying that this guy not only committed fraud, but convinced others -- his co-authors and research assistants -- to LIE with him.
That's a serious accusation, and I wonder if Jane can back it up.
Posted by: K Ashford | Oct 13, 2006 3:24:49 PM
Actually no; I might be able to accept 660,000 deaths, if the end game were worth it. I'm not sure it is at this point. But that's beside the point; I just don't think it's really possible to get a good count of who's died based on cluster samples of a nation in this much turmoil; too many people leave, move, etc. The number could conceivably be higher, though I find that unlikely because of the casualties that would imply; at a 2:1 ratio (derived entirely unscientifically from checking news reports of killings in Iraq), the median estimate would imply that 5% of Iraq is wounded, 5% has fled, 2.5% are dead . . . given that the deaths are concentrated among young men, and half the population of Iraq is under the age of 20, there should at this point be almost no one left to do the killing. If the figure is higher still, then they're going to have to start importing young men to shoot.
The direction of the survey is probably useful, and confirms what we already know: the violence is getting worse, and more concentrated among civilians, and also is at this point almost all being inflicted by Iraqis on other Iraqis. And Bush's 30,000 figure is obviously too low. But I don't think we can know with any precision what the right number is.
Posted by: Jane Galt | Oct 13, 2006 3:30:20 PM
K Ashford, have you seen me touting the WSJ editorial page as a model of fairness and objectivity? Or excusing Katherine Harris? Or saying anything about her at all, except that she wears way too much makeup, but even if you don't like her, you shouldn't try to run her over with your car? I'm not some cachepot for "all conservatives" or "all libertarians"; I'm me.
Nor am I accusing the editor of anything. I'm just pointing out that it is possible to stack the peer review process, if you know the outcome you want. As I say, I just think he's funny. And funnily good at shooting himself in the foot.
If a peer reviewed study finding no excess deaths had been published in the Wall Street Journal editorial page, would you find it credible? I find it obvious that people who believe things go looking for evidence that confirms their beliefs, especially when the beliefs are this strong. The author of the study has publicly announced that he rushed the previous version into print too fast in the hopes of affecting the 2004 election. You may find this admirable; undoubtedly many here do. But once you say things like that, you cannot really fault people for wondering how much your confirmation bias affected your work. And no, neither can the Wall Street Journal editorial board; the effect of their editorials (and, for that matter, those in the New York Times) is considerably weakened by the fact that 98% of the time, I know what it is going to say before I read it.
Posted by: Jane Galt | Oct 13, 2006 3:38:52 PM
"I might be able to accept 660,000 deaths, if the end game were worth it."
Jane,
You slipped by my question, on to more comfortable ground. Let's say, for the sake of argument, that the "end game" is a relatively stable Shiite-dominated country with theocratic leanings, closely aligned, as it must be, with Iran. That's the best possible outcome at this point, and a long shot at that, the stable part. Would you be willing to spend a million human lives on it?
Posted by: david mizner | Oct 13, 2006 3:39:22 PM
"But I don't think we can know with any precision what the right number is..."
This isn't a "count", Jane -- as in a NOSE count, and it doesn't claim to be "precise".
But that doesn't mean it lacks accuracy. The authors themselves acknowledge that the precise number falls between 400,000 and 900,000.
given that the deaths are concentrated among young men
I wasn't aware that women, the elderly, and children in Iraq were impervious to shrapnel from an exploding car bomb in a public market.
given that the deaths are concentrated among young men, and half the population of Iraq is under the age of 20, there should at this point be almost no one left to do the killing
"Almost no one"? WTF?
First of all, people under the age of 20 can do killing.
Secondly, even taking into account your "unscientific" percentages of those who have died or fled or are injured, there are STILL several million Iraqis capable of killing, and tens of thousands who are doing so.
Posted by: K Ashford | Oct 13, 2006 3:44:30 PM
Reviewers for a journal are typically--though not always--selected from people who have previously published in that journal.
More interestingly, some journals requesting a publication review will send along a sort of "punt" option: if you are not interested in the topic, or are not knowledgable about it, they ask you to pass it on to someone who is.
Frequently this ends up being a graduate student.
This is both better and worse than you might imagine: grad students are frequently closer to the literature than their research advisors, and are a shockingly conscientious lot. However, they frequently lack skills to evaluate 'the bigger picture.' That's a process largely aided by the existence journal editors: they are frequently established academics in their own right with long exposure to their field.
Posted by: TJ | Oct 13, 2006 3:48:27 PM
If a peer reviewed study finding no excess deaths had been published in the Wall Street Journal editorial page, would you find it credible?
Why no, because the WSJ editorial page is a notable fount of conservative crackpottery and outright lies, as opposed to a well respected journal that's unlikely to trash its reputation to score points.
As for the rest, why we should trust your judgment about this study over the judgment of people who work in the field? Why? Do you think they haven't considered that the nation is in turmoil? Do you think you're the first person to think of that? Do you think you have some reputation for intellectual probity that would cause people who aren't already biased toward your viewpoint to take your word for anything?
Posted by: Matt Weiner | Oct 13, 2006 3:49:09 PM
Nor am I accusing the editor of anything. I'm just pointing out that it is possible to stack the peer review process, if you know the outcome you want.
Well, YES, Jane, of course it's POSSIBLE.
But you're stating a point so blindingly obvious that it doesn't bear mentioning....
Why would you suggest that unless you were trying to lay the seeds that he actually DID "stack the peer review process"?
"The author of the study has publicly announced that he rushed the previous version into print too fast in the hopes of affecting the 2004 election. ...But once you say things like that, you cannot really fault people for wondering how much your confirmation bias affected your work."
Well, if there is something about the final product which suggests that the research and analysis was hastily done (for WHATEVER reason) and therefore flawed, I would like to hear about it.
But frankly, if a kid turns in his homework assignment a week early, we really shouldn't care about his motives, should we, as long as its complete and meets the standards?
Posted by: K Ashford | Oct 13, 2006 3:57:17 PM
One more thing: one of the reasons we don't have more definitive stats is because the U.S. doesn't even try to record civilian deaths: that's crassness bordering on racism.
Posted by: david mizner | Oct 13, 2006 4:00:15 PM
I doubt racism has much to do with it, David. I think the US government knows it benefits from there being as much uncertainty as possible about civilian death tolls. Even if they massively undercounted deaths they'd have to admit that freedom is looking very untidy in Iraq. As it is, there seems to be a climate in which it's easy for supporters of the war to adopt the hear-no-evil pose about its effects on Iraqis.
Posted by: Matt Weiner | Oct 13, 2006 4:08:22 PM
K Ashford, if you'd read the study, you'd know that the Lancet study reported that the deaths were concentrated among young men.
On the other hand, if you want to argue that the Lancet study is off . . .
As for Mr Roberts, he announced that in the course of explaining why there were some problems with the study. He didn't do his homework a week early; he did 80% of his homework a week early, and got full credit from his ideological compatriots.
Yes, seeing that video makes me question his credibility; as do reading Les Roberts' statements on the war and the bush administration. It's not crazy when you know someone's ideological predisposition to assume it could affect their work. In fact . . . why bless my stars . . . that's exactly what you're doing to me! Any of the various amateur statisticians who agree with you are obviously fonts of impartial wisdom, while those who disagree with you are innumerate hacks with an axe to grind.
The truth is probably somewhere in between. I don't feel as if I have any particular ideological dog in this fight; though I supported the war, I'm itching to see the Republicans voted out this fall. But obviously I, like everyone else, have ideas about the world that can interfere with perfectly clear-minded assessment of the evidence.
Posted by: Jane Galt | Oct 13, 2006 4:11:47 PM
David Mizner: according to the Lancet study, the overwhelming majority of civilian deaths currently take place at the hands of other Iraqis. The US doesn't have any particular reason to record them--that's the Iraqi government's job--much less any way to do so, since they tend to happen outside of American controlled areas. AFAIK, the US does record the number of Iraqis it kills, but that is now (everyone agrees)only a small fraction of the total.
Posted by: Jane Galt | Oct 13, 2006 4:14:35 PM
It's not crazy when you know someone's ideological predisposition to assume it could affect their work. In fact . . . why bless my stars . . . that's exactly what you're doing to me! Any of the various amateur statisticians who agree with you are obviously fonts of impartial wisdom, while those who disagree with you are innumerate hacks with an axe to grind.
Good Lord, Jane, do you think that people who cast aspersions on your work are doing so simply because they know your ideology? In my case, at least, I do so because you have an extensive record of (possibly unconscious) failures of objectivity, to be kind.
Posted by: Matt Weiner | Oct 13, 2006 4:25:04 PM
I see, Matt . . . it's just objective reality. it's funny how only people who disagree with me ideologically are able to discern this objective reality, while only people who disagree with Daniel Davies are able to see his unconscious biases.
Thanks for clearing that up.
Posted by: Jane Galt | Oct 13, 2006 4:29:13 PM
