Did Wall Street Journal Find Fatal Flaw in Lancet Iraq Study?

Too few cluster points render Iraq casualty figure “bogus,” claims op-ed piece

For Stats.org at George Mason University, Rebecca Goldin, Ph.D., and Trevor Butterworth write:
In an opinion column for the October 18 edition of the Wall Street Journal, Steven E Moore argues that the Johns Hopkins researchers who conducted a survey of excess deaths in Iraq since 2003 screwed up – not through the statistical tools they used, which are sound, but through parsimony:

“…the key to the validity of cluster sampling is to use enough cluster points. In their 2006 report, "Mortality after the 2003 invasion of Iraq: a cross-sectional sample survey," the Johns Hopkins team says it used 47 cluster points for their sample of 1,849 interviews. This is astonishing: I wouldn't survey a junior high school, no less an entire country, using only 47 cluster points…

…What happens when you don't use enough cluster points in a survey? You get crazy results when compared to a known quantity, or a survey with more cluster points….

…With so few cluster points, it is highly unlikely the Johns Hopkins survey is representative of the population in Iraq.”

On the face of it, this sounds like a fatal flaw. But unless the sample is actually biased, a smaller number of cluster points only has the effect of widening the confidence interval. Polls don't like large confidence intervals, but for the purposes of estimating large numbers of people, even the wide confidence interval of the Lancet study is informative.

The point is that the number of clusters relative to the size of the population is less relevant than whether the sample of clusters is representative of the population. So when Moore implicitly criticizes the Lancet study in relation to a similar study on Kosovo which used 50 cluster points, “for a population of just 1.6 million, compared to Iraq's 27 million,” the issue is not one of brute numbers, but whether the clusters chosen are representative of the overall population.

Research biostatistician Steve Simon (by way of Deltoid at Science Blogs, who is highly critical of Moore’s article) explains the principle:

“‘Every cook knows that it only takes a single sip from a well-stirred soup to determine the taste.’ It's a nice analogy because you can visualize what happens when the soup is poorly stirred.

With regards to why a sample size characterizes a population of 10 million and a population of 10 thousand equally well, use the soup analogy again. A single sip is sufficient both for a small pot and a large pot.”

Moore also argues that the Lancet’s figures would have been more trustworthy if the researchers had taken demographic data such as gender, age, and education.

Unquestionably, it would have been better if the Lancet study had added demographic
Information as it's possible that they didn't control for some demographic bias. But when Moore says this would have enabled them to compare results with “a known demographic instrument, such as a census,” he is quite possibly overestimating the accuracy and usefulness of the only other demographic instrument available to the researchers, the 1997 Iraq Census.

What the John’s Hopkins survey has in its favor is that it extrapolated its cluster points to the general population using the 2004 "UNDP/Iraqi Ministry of Planning population estimates".

In the end, Moore has opened up some interesting lines of inquiry, but he has ended up over-reaching in an effort to prove the Lancet figures “bogus.”

How The Media Covered The Lancet Study

A surprising inability to convey the study’s findings accurately

For Stats.org at George Mason University, Robert Lichter, Ph.D., writes:
A new study has generated heavy news coverage with its finding that over 650,000 Iraqis have died as a direct or indirect result of the March 2003 US-led invasion. It has also created widespread controversy, largely because this total is far higher than any previous estimate, which creates political problems for President Bush and other supporters of the war. The study, which appeared in the British medical journal The Lancet, is a follow-up to a previous study that attracted similar criticism.

Surprisingly, however, some leading news outlets both here and abroad have proved unable to correctly state the major findings. Extrapolating from a randomly-based survey of Iraqi households, the researchers estimated that approximately 655,000 Iraqis had died as a result of the war, including 601,000 whose deaths involved some form of violence. These two numbers are the core of the current debate.

These figures were prominently displayed both in the article and in the press release sent out by the Lancet. Yet some major news organizations somehow failed to get them straight.

Most prominent among them was the New York Times, which headlined its article, "Iraqi dead may total 600,000, study says." Of course that number was 55,000 too low, because it referred only to the violent deaths.

One might argue that this error did not materially influence the basic message of the story. In fact it was echoed by other major papers such as the Wall Street Journal: "Iraqi death toll exceeds 600,000, study estimates," the Los Angeles Times: "study puts wars Iraqi death tally at more than 600,000." and the Financial Times: "conflict in Iraq has killed more than 600,000 people."

All the above headlines are literally correct. They are just misleading, because they are nearly 10 percent too low, and they clearly reflect confusion between the study’s two separate estimates. But the Washington Post inverted this mistake in a front-page teaser referring readers to a story on an inside page: "a new study says 655,000 more Iraqis have died violently [emphasis added] since the invasion than otherwise would have been killed." Ironically, the story itself got the numbers right. But readers who glanced at the front page got the impression that the study’s estimate of violent deaths was over 50,000 higher than the actual figure.

However, the lead sentence in the New York Times article contained a much more significant blooper than appeared in any of the other papers: "A team of American and Iraqi public health researchers has estimated that 600,000 civilians have died in violence across Iraq since the 2003 American invasion... [emphasis added] The error was compounded later in the article, which described last July as "the highest [month] for Iraqi civilian deaths since the American invasion."

This is a glaring error, because it vastly increases the number of the war’s innocent victims, making the conflict seemed even more terrible than it already is. Moreover, the researchers were quite clear about having counted all deaths rather than civilian deaths. For example, they noted that almost 60 percent of deaths occurred among military-age men. In fact it is probably impossible to reliably separate civilian from combatant deaths. Many Iraqis surveyed would be highly unlikely to admit that their deceased family members had been involved in military activity, either because they were fighting with the insurgents or because they were fighting for the Iraqi security forces but sought to hide this information from the insurgents.

Because this error appeared in the New York Times, it was picked up by other papers subscribing to the Times news service, appearing as far away as the San Francisco Chronicle and the Times of India. But foreign media didn’t need the New York Times to confuse the total death toll with the civilian death count. For example, Der Spiegel, the Time magazine of Germany, headlined its story, "new study estimates 600,000 civilian Iraqi death toll." the Sydney, Australia morning Herald led with: "More than 600,000 civilians have died in violence across Iraq since the 2003 US invasion..." And Al Jazeera managed to combine the errors made by both the New York Times and the Washington Post: "More than 650,000 Iraqi civilians have died in violence as a result of the US-led 2003 invasion of Iraq, a new study says..."

What makes these errors particularly egregious is that the same authors, using the same methods, published earlier findings from this research project only two years ago, in October 2004. They reached the same conclusion, that the number of Iraqi deaths attributable to the war was far higher than any previous estimate. Finally, their report produced the same sort of controversy that has recurred this month. You might expect, therefore, that some of the world’s leading news organizations would be at an advantage in reporting the facts this time around. But you would be wrong. To make these mistakes the second time around recalls an Oscar Wilde character’s criticism of an orphan: "To lose one parent... may be regarded as a misfortune; to lose both looks like carelessness."

Iraq's Missing Dead

In Baghdad, thousands of bodies have been pulled from the Tigris, but the deaths aren't reported. How bad is the violence?

For Macleans.ca, Adnan R. Kahn writes:
Ali is a collector of the dead. That's his job, or at least one of them. He is also a cook at a kebab house in Baghdad and a member of the Mahdi Army, a Shia militia loyal to the militant cleric Muqtada al-Sadr. As a collector, his morbid duty is to sweep up the carnage of a sectarian war spiralling out of control -- one that Iraqi officials and their American overseers are trying desperately to downplay -- and quietly transport it to Iraq's main morgue, located in the heavily fortified Medical City in Baghdad's Bab al-Muatham neighbourhood, where all suspicious deaths are taken.

Every three days, Ali says, he and other al-Sadr militiamen go to the Tigris river to pick up bodies. At a spot on the bank just downstream from the Aima bridge in central Baghdad, a series of eddies gently gather in the dead. "More and more are coming there," Ali says, "from north of Baghdad, from villages like Taji and Balad. Many have their hands tied, most are blindfolded." The method of execution varies, Ali adds, from the basic bullet to the head to more macabre and viciously novel techniques involving power tools, electric cords and other such domestic instruments. "These are all Shia brothers and sisters murdered by Sunnis," says Ali, a Shia militant himself who has carried out his own revenge attacks on Sunnis. When pressed, he admits there "may be" some Sunnis floating down the Tigris as well. "But they were killed in defence of our Shia brothers and sisters," he claims. "They are not innocent victims."

Sectarian hostility aside, there is another aspect to Ali's work that is troubling: the deaths of the people whose bodies he pulls out of the river often go unreported, leading to questions about the real scale of the violence in Iraq. Even the wildly fluctuating official death counts are a stark reminder that Iraqi, and by association U.S. officials, are attempting to minimize a problem getting worse by the day. Earlier this year, the figures released by the government following the Feb. 22 bombing of the Askariya shrine in Samarra, a Shia holy site, which has been cited as the spark that started the current round of killings, were suspiciously lower than numbers provided by morgue officials. But as for the overall picture, a September report published by the United Nations Assistance Mission for Iraq paints a grim picture: civilian deaths reached a record high for July and August with 6,600 civilians killed.

Still, even these figures don't tell the whole story. For that, a visit to Medical City is in order. The Ministry of Health has instituted a strict policy for journalists, requiring them to seek permission before visiting the facility. Those allowed in get only a truly sanitized tour; more often than not reporters are barred from entering. But at the gate, guards who have worked at the facility tell a chilling tale. "Last year, I saw maybe 1,000 bodies a month coming into the morgue," says one man who, fearing for his life, requested his name not be published. "Now we're getting nearly 1,000 a week."

All, he says, are victims of sectarian violence, both Sunni and Shia, but the officials at the morgue inside Medical City will not tell you that. "The officials don't want us talking to the media," says another guard, also requesting anonymity. "I've heard them telling people that most of the deaths are because of terrorists, but I've also seen the bodies myself and I can tell you that most of them were executed by death squads."

While he describes the bodies, a dump truck pulls out of the facility. The guards open the gate, holding back a rush of people from all over central Iraq hoping to get in to look for loved ones. As the truck passes, the smell of decomposing flesh fills the air. "That's just the clothes from bodies pulled from the Tigris over the past few days," says the first guard. "The trucks come and go regularly." The stench is overwhelming. People cover their noses and mouths with cloth. Women wail.

When told about this back at the kebab house, Ali simply shrugs. "You get used to the smell," he says, putting ground beef onto skewers for non-existent customers in a city in which people are too afraid to leave their homes. That job appears to have no potential. But tomorrow, he will be going back to the Tigris to pick up more bodies. That job, at least in Baghdad, has a future.

2,660 Iraqi Civilians Killed In September

The Washington Post reports:
More than 2,660 Iraqi civilians were killed in the capital in September amid a wave of sectarian killings and insurgent attacks, an increase of 400 over the month before, according to figures from the Iraqi Health Ministry.

The increase came despite an intensified U.S.-Iraqi sweep of Baghdad that was launched in mid-August to try to put down the wave of violence that has swept over the capital. The violence consists of a deadly combination of bombings and shootings by Sunni insurgents, and slayings by Shiite and Sunni death squads.

The September numbers come as a controversial new study contends that nearly 655,000 Iraqis have died in the three-year-old conflict in Iraq _ more than 10 times higher than other independent estimates of the toll.

President Bush dismissed as "just not credible" the study's estimate study that contends nearly 655,000 Iraqis have died because of the war.

Bush, who in the past has suggested 30,000 civilian deaths in Iraq, would not give a figure for overall fatalities. "A lot of innocent people have lost their life," he said at a news conference in Washington.

The study, which is to be published Thursday on the Web site of The Lancet, a medical journal, was based on a survey of households in Iraq, not a body count, and quickly raised skepticism among some Iraq experts.

Pentagon spokesman Bryan Whitman said it was difficult "with any certainty" to estimate the number of Iraqi civilians who have died, and said the department does everything possible to prevent civilian casualties.

"We take great precautions in our military operations," he said. "That's in stark contrast to what the enemy in Iraq is doing. They take no such precautions. In fact, they deliberately target innocent civilians in their attacks."

An accurate count of total Iraqi deaths since the war's start has been difficult to obtain. According to an Associated Press tally, at least 13,414 Iraqis have been killed in war-related violence through Tuesday since the new government took office on April 28, 2005. Of those, 9,300 were civilians.

The AP tally is compiled from hospital, police and military officials cited in news stories, as well as accounts from reporters and photographers at the scenes. The actual number is likely higher as many killings go unreported or uncounted.

A private group called Iraqi Body Count says it has recorded about 44,000 to 49,000 civilian Iraqi deaths. But it notes that those totals are based on media reports, which it says probably overlook "many if not most civilian casualties."

The figures for the number of civilian deaths in September in Baghdad came in an official monthly report from the Health Ministry to the Cabinet on the number of Baghdad victims of violent deaths, two senior ministry officials told The Associated Press.

The two officials spoke on condition of anonymity because the government has issued orders that the death figures not be released. Civilian casualty numbers are always sensitive, and several other officials in the Health and Interior ministries contacted by the AP refused to give statistics.

The report said 2,667 civilians had died violent deaths in September _ an average of 89 a day. Those deaths include bodies found dumped around Baghdad and the victims of explosions, shootings and other attacks, the two officials said.

By comparison, 2,222 people died violently in August in Baghdad, according to a U.N. report published in September, which is also based on official statistics from the Health Ministry.

The two ministry officials said the U.N. number was accurate for the August deaths.

The monthly figures include two categories. One is the number of bodies found in Baghdad, provided from the city morgue, where the bodies are taken to determine the cause of death.

In September, the morgue reported 1,471 bodies of people who died from violence.

Shiite and Sunni death squads are known to kidnap members of the opposing sect, then dump bodies of their victims, often bound and tortured. So a large proportion of the 1,471 bodies are likely from sectarian killings _ though they also would include victims of criminal kidnappings and murders.

The other category included in the monthly figure encompasses the victims of explosions, shootings or other attacks, reported by hospitals. They numbered 1,196 in September, according to the report, the two Health Ministry officials said.

In August, 1,536 bodies were brought to the morgue, according to the U.N. report.

The past summer has seen a startling increase in bloodshed, centered in the capital, after the wave of sectarian violence was sparked by the February bombing of a Shiite shrine in the city of Samarra, north of Baghdad.

The deadliest month was July, when 3,590 people were killed across the country _ 2,884 of them in Baghdad, according to the U.N. The number killed countrywide fell in August to 3,009, the U.N. said.

The Health Ministry officials who spoke to the AP did not have September figures for the entire country, only for Baghdad.

Old Death In The New Iraq

Cervantes at Stayin' Alive weighs in on the Lancet study:

Okay, the Lancet doesn't come out until tomorrow but you can already get the manuscript of the Johns Hopkins/Al Mustansiriya/MIT study on excess deaths in Iraq here. Since we're already hearing that it's just a political stunt, not reliable, etc., I figured I ought to offer my take on it.

As readers of Today in Iraq know, there are daily death tolls reported by the Baghdad morgue, the police, and stringers around the country who feed information to the major news services who have offices in Iraq -- AP, AFP, Reuters, KUNA (the Kuwait news agency) and Xinhua (the Chinese agency) provide the bulk of the available information. Many people assume that the reports they read in the newspaper, usually from AP, or the daily Reuters "fact box" report that many bloggers (including Atrios and Juan Cole) often repost, are more or less complete descriptions of the day's violence. In fact, they don't even come close.

By combining information from all of the available sources, we always come up with at least two or three times as many violent incidents as you will find in any one source. Even so, it's pretty obvious that most deaths by violence in Iraq never get noted by the police, the morgue, or the news services. Out of deference to their masters, the Iraqi authorities generally don't try to count people killed by the occupation forces. Much of the country is off limits to journalists. Many people who are shot dead never end up in the morgue and there certainly isn't any reason for Iraqis to make police reports. (Police or other security forces, or people dressed like them, are responsible for most of the murders in the first place.) Muslims bury their dead quickly and relatively unceremoniously, and, while the health ministry issues death certificates, it has no system for aggregating and reporting vital records data.

So, the researchers set out to estimate deaths by means of a household survey using area probability sampling methods. This is a method used all the time in health surveys. It's a method I have used myself, in fact. To begin, you just need census data -- it actually doesn't even have to be highly accurate as long as any errors are essentially random, or unrelated to your study questions. Then, you pick geographic areas based on probability proportionate to the population they contain. This is usually done in stages. In the Iraq study, they first determined the number of clusters they would select in each province based on population size (Baghdad, with its population of over 6 million, got 12; Muthanna, with a population of 570,000, happened to get none.) Then, towns, blocks, and starting households were selected at random. For each household selected, the 39 nearest houses were also included. This survey had a total of 47 clusters, including 12,801 persons.

The researchers interiewed adult household members between May and June, 2006, to learn about births, deaths, and migration since January 1, 2002. They also asked people to report if an entire neighboring household had been wiped out, to account for households with no-one left to speak for them. They report that for 92% of reported deaths, the respondents were able to produce a death certificate. A substantial omission in the report, I must say, is the failure to state the response rate. The investigators also refer to procedures for substituting areas which were too unsafe to visit. They do not say how often this happened, but if anything, it would tend to bias the results downward.

To arrive at an estimate of total deaths for the country, they simply multiply the deaths in the study population by the appopriate weights for the number of people each cluster represents (i.e., the inverse of the probability that a person living in that province would have been selected). The clustering does not directly affect the estimates, but it does affect the so-called confidence interval. Since people living in a specific area are at greater or lesser risk of violent death than average, the statistical power of the study is less than it would be for a single stage probability sample of 12,801 persons, because of the possibility that the selection of clusters introduced sampling error. Although the manuscript does not discuss the specific calculations that were done to adjust for this, I am willing to give investigators from these institutions the benefit of the doubt that they did it correctly.

It is conventional to report 95% confidence intervals. The researchers find that there is a 95% probability, assuming no systematic biases in their data, that there have been between 426,369 and 793,663 excess deaths from violence among Iraqis since the invasion -- i.e., deaths that would not have occurred had the death rate continued as before. (There were very few violent deaths in Iraq prior to the invasion. The famous mass graves date from the era of the Iran-Iraq war, and suppression of Kurdish and Shiite rebellions associated with that era.) The investigators also estimate that there have been about an additional 54,000 deaths from non-violent causes, mostly in 2005-2006, as Iraq's health care and public health infrastructure severely deteriorated.

The steady increase in violent death rates is quite appalling, from 3.2/1,000/year in March 2003-April 2004; to 12/1,000/year from June 2005-June 2006. Not surprisingly, the deaths are concentrated in Baghdad and the predominantly Sunni Arab areas of the country. The three provinces of autonomous Kurdistan have been peaceful. 31% of violent deaths were caused by coalition forces, 24% by other actors, and in 45% of cases the perpetrators were unknown to the respondents. Even if none of these were caused by coalition forces, it results that U.S. troops have killed about 200,000 Iraqis, with perhaps a modest contribution from the British.

Are these results reliable? They are in fact the most reliable information we have about this subject. Particularly powerful confirmation comes from the very close match in this survey between deaths reported to have occurred in 2003-2004; and the results from a similar study conducted by the team in 2004. That of course had an entirely different sample of households, but used the same methods. People often misunderstand the concept of the confidence interval. It is far more likely that the true number of violent deaths is close to 600,000, than that it is close to 427,000. People also do not understand how a sample consisting of such a small percentage of the population can give us confidence in saying something about the entire population. But that results from the laws of probability, which assure that state lotteries and casinos will always win.

Was the release of this report politically motivated? Possibly the authors made a special effort to get it out before the election, but that has no relation to its truth.

Finally, as I have said many times, Iraq Body Count should go out of business. They are doing positive harm to the reality based community by giving the perpetrators of this world historical crime cover for saying that the death toll is only 10% of what it really is. That is not helping the Iraqi people.

UPDATE: Thanks to a tip from Whisker, here's an article that shows that innumerable violent deaths in Iraq go unreported.

Study Claims Iraq's 'Excess' Death Toll Has Reached 655,000

The Washington Post reports:
A team of American and Iraqi epidemiologists estimates that 655,000 more people have died in Iraq since coalition forces arrived in March 2003 than would have died if the invasion had not occurred.

The estimate, produced by interviewing residents during a random sampling of households throughout the country, is far higher than ones produced by other groups, including Iraq's government.

It is more than 20 times the estimate of 30,000 civilian deaths that President Bush gave in a speech in December. It is more than 10 times the estimate of roughly 50,000 civilian deaths made by the British-based Iraq Body Count research group.

The surveyors said they found a steady increase in mortality since the invasion, with a steeper rise in the last year that appears to reflect a worsening of violence as reported by the U.S. military, the news media and civilian groups. In the year ending in June, the team calculated Iraq's mortality rate to be roughly four times what it was the year before the war.

Of the total 655,000 estimated "excess deaths," 601,000 resulted from violence and the rest from disease and other causes, according to the study. This is about 500 unexpected violent deaths per day throughout the country.

The survey was done by Iraqi physicians and overseen by epidemiologists at Johns Hopkins University's Bloomberg School of Public Health. The findings are being published online today by the British medical journal the Lancet.

The same group in 2004 published an estimate of roughly 100,000 deaths in the first 18 months after the invasion. That figure was much higher than expected, and was controversial. The new study estimates that about 500,000 more Iraqis, both civilian and military, have died since then -- a finding likely to be equally controversial.

Both this and the earlier study are the only ones to estimate mortality in Iraq using scientific methods. The technique, called "cluster sampling," is used to estimate mortality in famines and after natural disasters.

While acknowledging that the estimate is large, the researchers believe it is sound for numerous reasons. The recent survey got the same estimate for immediate post-invasion deaths as the early survey, which gives the researchers confidence in the methods. The great majority of deaths were also substantiated by death certificates.

"We're very confident with the results," said Gilbert Burnham, a Johns Hopkins physician and epidemiologist.

A Defense Department spokesman did not comment directly on the estimate.

"The Department of Defense always regrets the loss of any innocent life in Iraq or anywhere else," said Lt. Col. Mark Ballesteros. "The coalition takes enormous precautions to prevent civilian deaths and injuries."

He added that "it would be difficult for the U.S. to precisely determine the number of civilian deaths in Iraq as a result of insurgent activity. The Iraqi Ministry of Health would be in a better position, with all of its records, to provide more accurate information on deaths in Iraq."

Ronald Waldman, an epidemiologist at Columbia University who worked at the Centers for Disease Control and Prevention for many years, called the survey method "tried and true," and added that "this is the best estimate of mortality we have."

This viewed was echoed by Sarah Leah Whitson, an official of Human Rights Watch in New York, who said, "We have no reason to question the findings or the accuracy" of the survey.

"I expect that people will be surprised by these figures," she said. "I think it is very important that, rather than questioning them, people realize there is very, very little reliable data coming out of Iraq."

The survey was conducted between May 20 and July 10 by eight Iraqi physicians organized through Mustansiriya University in Baghdad. They visited 1,849 randomly selected households that had an average of seven members each. One person in each household was asked about deaths in the 14 months before the invasion and in the period after.

The interviewers asked for death certificates 87 percent of the time; when they did, more than 90 percent of households produced certificates.

According to the survey results, Iraq's mortality rate in the year before the invasion was 5.5 deaths per 1,000 people; in the post-invasion period it was 13.3 deaths per 1,000 people per year. The difference between these rates was used to calculate "excess deaths."

Of the 629 deaths reported, 87 percent occurred after the invasion. A little more than 75 percent of the dead were men, with a greater male preponderance after the invasion. For violent post-invasion deaths, the male-to-female ratio was 10-to-1, with most victims between 15 and 44 years old.

Gunshot wounds caused 56 percent of violent deaths, with car bombs and other explosions causing 14 percent, according to the survey results. Of the violent deaths that occurred after the invasion, 31 percent were caused by coalition forces or airstrikes, the respondents said.

Burnham said that the estimate of Iraq's pre-invasion death rate -- 5.5 deaths per 1,000 people -- found in both of the Hopkins surveys was roughly the same estimate used by the CIA and the U.S. Census Bureau. He said he believes that attests to the accuracy of his team's results.

He thinks further evidence of the survey's robustness is that the steepness of the upward trend it found in excess deaths in the last two years is roughly the same tendency found by other groups -- even though the actual numbers differ greatly.

An independent group of researchers and biostatisticians based in England produces the Iraq Body Count. It estimates that there have been 44,000 to 49,000 civilian deaths since the invasion. An Iraqi nongovernmental organization estimated 128,000 deaths between the invasion and July 2005.

The survey cost about $50,000 and was paid for by Massachusetts Institute of Technology's Center for International Studies.

Bush Discredits Iraqi Death Toll Report

The Guardian reports:
The US president, George Bush, tonight admitted "a lot of innocent people" had lost their lives in Iraq but rejected the findings of a controversial report that the civilian death toll in the war-torn country had reached 655,000.
"I don't consider it a credible report," Mr Bush told a White House press conference in response to a study published in the medical journal The Lancet. "Neither does General Casey, neither do Iraqi officials," he added, referring to George Casey, the top US general in Iraq.

The Iraqi government said the report's death toll was "inflated" and "far from the truth", but did not give its own figure for the deaths.

Mr Bush, who last December suggested there may have been 30,000 civilian deaths in Iraq, would not give a figure today for overall fatalities.

The study found that the equivalent of 2.5% of Iraq's population had been killed since fighting began more than three years ago. The study, by the Johns Hopkins University's Bloomberg School of Public Health in Baltimore, was based on household interviews - not a body count - and was a follow-up to a report by the same group two years ago.

An Iraqi government spokesman, Ali al-Dabbagh, said in a statement that the toll in the report "exceeds the reality in an unreasonable way".

The report "gives inflated numbers in a way that violates all rules of research and the precision required of research institutions", he said.

Study: 655,000 Iraqis Died Due to War

CBS News reports:
A controversial new study contends nearly 655,000 Iraqis have died because of the war, suggesting a far higher death toll than other estimates.

The timing of the survey's release, just a few weeks before the U.S. congressional elections, led one expert to call it "politics."

In the new study, researchers attempt to calculate how many more Iraqis have died since March 2003 than one would expect without the war. Their conclusion, based on interviews of households and not a body count, is that about 600,000 died from violence, mostly gunfire. They also found a small increase in deaths from other causes like heart disease and cancer.

"Deaths are occurring in Iraq now at a rate more than three times that from before the invasion of March 2003," Dr. Gilbert Burnham, lead author of the study, said in a statement.

The study by Burnham, of the Johns Hopkins Bloomberg School of Public Health, and others is to be published Thursday on the Web site of The Lancet, a medical journal.

An accurate count of Iraqi deaths has been difficult to obtain, but one respected group puts its rough estimate at closer to 50,000. And at least one expert was skeptical of the new findings.

"They're almost certainly way too high," said Anthony Cordesman of the Center for Strategic & International Studies in Washington. He criticized the way the estimate was derived and noted that the results were released shortly before the Nov. 7 election.

"This is not analysis, this is politics," Cordesman said.

The work updates an earlier Johns Hopkins study _ that one was released just before the November 2005 presidential election. At the time, the lead researcher, Les Roberts of Hopkins, said the timing was deliberate. Many of the same researchers were involved in the latest estimate.

Speaking of the new study, Burnham said the estimate was much higher than others because it was derived from a house-to-house survey rather than approaches that depend on body counts or media reports.

A private group called Iraqi Body Count, for example, says it has recorded about 44,000 to 49,000 civilian Iraqi deaths. But it notes that those totals are based on media reports, which it says probably overlook "many if not most civilian casualties."

For Burnham's study, researchers gathered data from a sample of 1,849 Iraqi households with a total of 12,801 residents from late May to early July. That sample was used to extrapolate the total figure. The estimate deals with deaths up to July.

The survey participants attributed about 31 percent of violent deaths to coalition forces.

Accurate death tolls have been difficult to obtain ever since the Iraq conflict began in March 2003. When top Iraqi political officials cite death numbers, they often refuse to say where the numbers came from.

The Health Ministry, which tallies civilian deaths, relies on reports from government hospitals and morgues. The Interior Ministry compiles its figures from police stations, while the Defense Ministry reports deaths only among army soldiers and insurgents killed in combat.

The United Nations keeps its own count, based largely on reports from the Baghdad morgue and the Health Ministry.

The major funder of the new study was the Massachusetts Institute of Technology.

Lost Count

Researchers rushed a rigorous study of Iraqi civilian casualties into print. Is that why it was dismissed as pure politics?

Lila Guterman at the Chronicle of Higher Education writes:

When more than 200,000 people died in a tsunami caused by an Asian earthquake in December, the immediate reaction in the United States was an outpouring of grief and philanthropy, prompted by extensive coverage in the news media.

Two months earlier the reaction in the United States to news of another large-scale human tragedy was much quieter. In late October, a study was published in The Lancet, a prestigious British medical journal, concluding that about 100,000 civilians had been killed in Iraq since it was invaded by a United States-led coalition in March 2003. On the eve of a contentious presidential election -- fought in part over U.S. policy on Iraq -- many American newspapers and television news programs ignored the study or buried reports about it far from the top headlines.

The paper, written by researchers at the Johns Hopkins University, Columbia University, and Baghdad's Al-Mustansiriya University, was based on a door-to-door survey in September of nearly 8,000 people in 33 randomly selected locations in Iraq. It was dangerous work, and the team of researchers was lucky to emerge from the survey unharmed.

The paper that they published carried some caveats. For instance, the researchers admitted that many of the dead might have been combatants. They also acknowledged that the true number of deaths could fall anywhere within a range of 8,000 to 194,000, a function of the researchers' having extrapolated their survey to a country of 25 million.

But the statistics do point to a number in the middle of that wide range. And the raw numbers upon which the researchers' extrapolation was based are undeniable: Since the invasion, the No. 1 cause of death among households surveyed was violence. The risk of death due to violence had increased 58-fold since before the war. And more than half of the people who had died from violence and its aftermath since the invasion began were women and children.

Neither the Defense Department nor the State Department responded to the paper, nor would they comment when contacted by The Chronicle. American news-media outlets largely published only short articles, noting how much higher the Lancet estimate was than previous estimates. Some pundits called the results politicized and worthless.

Les F. Roberts, a research associate at Hopkins and the lead author of the paper, was shocked by the muted or dismissive reception. He had expected the public response to his paper to be "moral outrage."

On its merits, the study should have received more prominent play. Public-health professionals have uniformly praised the paper for its correct methods and notable results.

"Les has used, and consistently uses, the best possible methodology," says Bradley A. Woodruff, a medical epidemiologist at the U.S. Centers for Disease Control and Prevention.

Indeed, the United Nations and the U.S. State Department have cited mortality numbers compiled by Mr. Roberts on previous conflicts as fact -- and have acted on those results.

What went wrong this time? Perhaps the rush by researchers and The Lancet to put the study in front of American voters before the election accomplished precisely the opposite result, drowning out a valuable study in the clamor of the presidential campaign.

A Risky Proposition

Mr. Roberts has studied mortality caused by war since 1992, having done surveys in locations including Bosnia, Congo, and Rwanda. His three surveys in Congo for the International Rescue Committee, a nongovernmental humanitarian organization, in which he used methods akin to those of his Iraq study, received a great deal of attention. "Tony Blair and Colin Powell have quoted those results time and time again without any question as to the precision or validity," he says.

Mr. Roberts's first survey in Congo, in 2000, estimated that 1.7 million people had died over 22 months of armed conflict. The response was dramatic. Within a month, the U.N. Security Council passed a resolution that all foreign armies must leave Congo, and later that year, the United Nations called for $140-million in aid to that country, more than doubling its previous annual request. Later, citing the study, the State Department announced a pledge of an additional $10-million for Congo.

About a year ago, Mr. Roberts decided to study mortality in Iraq. He connected with a colleague at Columbia, Richard Garfield, a professor of nursing who had done research in Iraq since the mid-1990s. Mr. Garfield knew Riyadh Lafta, a mortality researcher at Al-Mustansiriya University, who recruited interviewers to do the door-to-door survey.

"He had to ask many people before he could find five interviewers willing to work in a study that involved an American," Mr. Roberts says.

Mr. Roberts planned to travel to Iraq last spring. After an American hostage was beheaded on video, however, Dr. Lafta told him that the danger was too great. "I was going to go in June, but in June it was even worse," says Mr. Roberts. Finally, in August, with a fall teaching commitment looming, he decided to go.

On September 1, Mr. Roberts sneaked into Iraq from Jordan lying on the back seat of an SUV. He trained the interviewers, who tried out their questions in a relatively safe neighborhood of Baghdad before embarking on the study.

The researchers visited 30 homes in each of 33 neighborhoods in Iraq. They selected the communities to be surveyed using a random process adjusted so that more populous areas were more likely to be picked, giving each person in Iraq an equal chance of being interviewed. Within each community, a spot was chosen at random, and the interviewers visited the 30 households nearest to that point.

At each house, the interviewers asked for the age and sex of everyone living there currently and on January 1, 2002. The interviewers asked about deaths since the first day of 2002 and recorded the day, cause, and circumstances, so that they could compare the time just before the 2003 invasion with the period since then.

In each neighborhood, in at least the first two households where an adult's death had occurred, the interviewers ended by asking for death certificates. They received confirmation of deaths in 63 of the 78 houses where they asked.

Mr. Garfield says the high proportion of death certificates assuaged his concern that lying might be widespread. In unstable countries, where records of deaths aren't always thorough, ascertaining lies or simply faulty memories becomes difficult.

At first Mr. Roberts accompanied the Iraqi researchers. To mask his identity, he dyed his graying brown hair black, wore Iraqi clothing, and never spoke in public. But he was acutely aware of the danger his presence created for his colleagues; one interviewer refused to ride in a car with him.

On the eighth day, the interviewers ended up in Balad, a town north of Baghdad, whose main street was dominated by a huge portrait of the radical Shiite Muslim cleric Moktada al-Sadr. "As fate would have it," says Mr. Roberts, "one of the first doors we knocked on was the governor's. There I am, I'm sitting in the car, and a police car rolls up and my two interviewers get hauled away."

Mr. Roberts and his driver decided to wait. "I laid on my side and pretended to be asleep so no one would see my blue eyes," Mr. Roberts says. After the interviewers had been gone for about 40 minutes, he says, "Two little kids walked up to the car and in English said, 'Hello, Mister!' It's just impossible for a Westerner to stay invisible in Iraq."

After more than an hour, the two interviewers, who were physicians for the Iraqi Ministry of Health, managed to talk their way out of the situation. Mr. Roberts retreated to a hotel in Baghdad for the duration of his stay, getting daily reports from Dr. Lafta.

Canvassing Fallujah

The researchers saved the most dangerous location for last: On September 20, Dr. Lafta went to violence-wracked Fallujah with the only interviewer willing to travel there. The researchers had done a haunting bit of calculus before the journey: given that the chance was high of an interviewer's or researcher's getting killed there, the study would be better served by getting the other data first.

The Fallujah data were chilling: 53 deaths had taken place in the study's 30 households there since the invasion commenced on March 19, 2003. In the other 32 neighborhoods combined, the researchers had counted 89 deaths. While 21 of the deaths elsewhere were attributable to violence, in Fallujah 52 of the 53 deaths were due to violence.

The number of deaths in Fallujah was so much higher than in other locations that the researchers excluded the data from their overall estimate as a statistical outlier. Because of that, Mr. Roberts says, chances are good that the actual number of deaths caused by the invasion and occupation is higher than 100,000.

Speed Kills?

Mr. Roberts took a few days in Baghdad in late September to compile and analyze the data. He discovered that the risk of death was 2.5 times as high in the 18 months after the invasion as it had been in the 15 months before it; the risk was still 1.5 times as high if he ignored the Fallujah data. Because he had found in many other wars that malnutrition and disease were the most frequent causes of civilian deaths, he was "shocked," he says, that violence had been the primary cause of death since the invasion.

"On the 25th of September, my focus was about how to get out of the country," he recalls. "My second focus was to get this information out before the U.S. election." In little more than 30 days, the paper was published in The Lancet.

Mr. Roberts and his colleagues now believe that the speedy publication of that data created much of the public skepticism toward his study. He sent the manuscript to the medical journal on October 1, requesting that it be published that month. Mr. Roberts says the editors agreed to do so without asking him why.

Despite the sprint to publication, the paper did go through editing and peer review. In an accompanying editorial, Richard Horton, editor of The Lancet, wrote that the paper "has been extensively peer-reviewed, revised, edited, and fast-tracked to publication because of its importance to the evolving security situation in Iraq."

Dr. Horton declined repeated requests for comment on the study and the decision to publish it before the U.S. presidential election. But three other major medical journals told The Chronicle that they, too, occasionally fast-track papers of immediate importance, and that the time from receipt to publication can be days or a few weeks.

Mr. Roberts calls the peer-review process that his paper underwent "rigorous." One of the peer reviewers told The Chronicle that he had had about a week to comment on the paper.

A Question of Timing

The timing of the paper's publication opened the study to charges of political propaganda. So did Mr. Roberts's admission to an Associated Press reporter on the day that the paper came out that he opposed the war. "That was the wrong answer," Mr. Roberts says now, "because some of the other study members hated Saddam and were in favor of the initial invasion."

Mr. Garfield, one of the co-authors, says he did not feel the same urgency about publishing before the U.S. election. "I was afraid that the importance of the topic would get lost among many other electoral issues," he says.

Mr. Garfield appears to have been correct.

The Lancet released the paper on October 29, the Friday before the election, when many reporters were busy with political stories. That day the Los Angeles Times and the Chicago Tribune each dedicated only about 400 words to the study and placed the stories inside their front section, on pages A4 and A11, respectively. (The news media in Europe gave the study much more play; many newspapers put articles about it on their front pages.)

In a short article about the study on page A8, The New York Times noted that the Iraq Body Count, a project to tally civilian deaths reported in the news media, had put the maximum death count at around 17,000. The new study, the article said, "is certain to generate intense controversy." But the Times has not published any further news articles about the paper.

The Washington Post, perhaps most damagingly to the study's reputation, quoted Marc E. Garlasco, a senior military analyst at Human Rights Watch, as saying, "These numbers seem to be inflated."

Mr. Garlasco says now that he hadn't read the paper at the time and calls his quote in the Post "really unfortunate." He says he told the reporter, "I haven't read it. I haven't seen it. I don't know anything about it, so I shouldn't comment on it." But, Mr. Garlasco continues, "Like any good journalist, he got me to."

Mr. Garlasco says he misunderstood the reporter's description of the paper's results. He didn't understand that the paper's estimate includes deaths caused not only directly by violence but also by its offshoots: chaos leading to lack of sanitation and medical care.

Online, the words flew. Some bloggers denounced the study. The online magazine Slate published an essay by its military columnist, Fred Kaplan, saying that the wide range of possible deaths, 8,000 to 194,000, "isn't an estimate. It's a dartboard."

The U.S. government had no comment at the time and remains silent about Iraqi civilian deaths. "The only thing we keep track of is casualties for U.S. troops and civilians," a Defense Department spokesman told The Chronicle.

Mr. Garfield now regrets the timing of the paper's release because he believes that it allowed people to dismiss the research. "The argument is an idiotic one of, 'You're playing politics, so then the data's not true,'" he says.

Such logic angers him. "Hey," he says, "this is valuable information. The fact that somebody wants to convince you of it -- how is that suddenly illegitimate? Why is that a reason to ignore it? If it's wrong, then ignore it! If it's dealing with deaths of people that don't count in the world, then ignore it! I don't think it's wrong, and I don't think Iraqi deaths don't count."

Mr. Roberts insists that his primary motivation for rushing the paper to press was not political. He says he is glad the paper appeared before the election because he was concerned for his Iraqi colleagues' safety. Had the paper come out after the election, he argues, it would have looked like a cover-up. Dr. Lafta, he says, "would have been killed. There is just no doubt."

Dr. Lafta, in an e-mail message to The Chronicle, disagrees: "My personal opinion is that this was an unjustified fear."

Mr. Roberts acknowledges that he also hoped to ignite a policy change or public response. "This was going to do more good in terms of changing policy if it came out in October than if it came out in November," he says. "But we never had any delusions that this might affect the U.S. election."

Reassessing the Evidence

The reception of the Iraqi mortality study by scientists has been far friendlier than by the news media.

Scientists say the size of the survey was adequate for extrapolation to the entire country. "That's a classical sample size," says Michael J. Toole, head of the Center for International Health at the Burnet Institute, an Australian research organization. Researchers typically conduct surveys in 30 neighborhoods, so the Iraq study's total of 33 strengthens its conclusions. "I just don't see any evidence of significant exaggeration," he says.

David R. Meddings, a medical officer with the Department of Injuries and Violence Prevention at the World Health Organization, says any such survey will have uncertainty because of extrapolation based on small numbers, and because of the possibility that people gave incorrect information about deaths in their households.

"I don't think the authors ignored that or understated" those factors, he says. "Those cautions I don't believe should be applied any more or any less stringently to a study that looks at a politically sensitive conflict than to a study that looks at a pill for heart disease."

The uncertainty leads to the breadth of the so-called 95-percent confidence interval -- in other words, the 95-percent chance that the number of deaths in Iraq resulting from military activities is between 8,000 and 194,000. Critics like the Slate writer seized on that range, says Dr. Woodruff, the government epidemiologist.

"They thought, 'Well, it's just as likely to be 18,000 as 100,000.' That's not true at all," he says. "The further you get away from 100,000, the probability that the number is true gets much smaller."

The gap between the Lancet estimate and that of Iraq Body Count does not trouble scientists contacted by The Chronicle. John Sloboda, a professor of psychology at the University of Keele, in England, and co-founder of the body count, says his team's efforts will lead to a count smaller than the true number because not every death is reported in the media.

Dr. Woodruff says, "Les [Roberts] has the most valid estimate."

Dr. Toole agrees: "If anything, the deaths may have been higher [than the study's estimate] because what they are unable to do is survey families where everyone has died."

Robin M. Coupland, a medical adviser on weapons and armed violence in the legal division of the International Committee of the Red Cross, has only one concern: Mr. Roberts's team did not document how many people were wounded.

"In every recorded context where conventional explosive weapons have been used in armed contact," Dr. Coupland says, "there's usually two or more people wounded per person killed. The question that glares out from that article is, Where are all the 200,000 wounded?"

Mr. Roberts says they didn't ask about injuries because of the difficulty of defining both what constitutes an injury and whether the injury stemmed from violence. "If someone is running from fighting and they cut their foot, is that a war wound?" he asks.

Burden of Proof

Despite the muted public response, public-health professionals are glad that the study brought to light the human toll of the Iraq war and continuing occupation. Both the study and the Iraq Body Count, says Mr. Sloboda, are "shoestring attempts by private citizens" to do work he says the government ought to be doing.

Mr. Garlasco, of Human Rights Watch, is mystified that the Defense Department has not expressed interest in such studies. "Civilian casualties can be a bellwether for the actual conduct of the war," says Mr. Garlasco, who was an intelligence officer at the Pentagon until 2003. "They're using all these precision weapons, so one would expect that if you're striving to minimize casualties, you'd have very low casualties. In Iraq we've seen the exact opposite, so one has to wonder why."

Besides, he says, counting civilian deaths could actually be useful for the Pentagon's public image. "I truly believe when the U.S. military says we're not there to kill civilians, it's absolutely true," he says. "The problem is, though, there are many people who don't accept their reasoning. The only way they'll change their minds is if the U.S. military shows they take civilian casualties seriously enough that they quantify them and attempt to minimize casualties in the future."

In the Lancet article, Mr. Roberts and his colleagues write, "It seems difficult to understand how a military force could monitor the extent to which civilians are protected without systematically doing body counts or at least looking at the kinds of casualties they induce."

That's why surveys like the Lancet one are important, says the WHO's Dr. Meddings, even if the immediate response is hesitant: "If you can put accurate information out, it shifts the burden of proof onto militaries to substantiate why what they're doing is worth this humanitarian cost."

At the end of the day, Mr. Roberts worries that his study may play little part in that crucial debate. Although he blames the American news media for being embedded not only with the military but also with the military point of view, he also partially blames himself for the lack of public response.

"Maybe we the scientists have mismanaged this information," he says. "We had a message that was of interest to most Americans. We had a message that was extremely robust scientifically. And we failed to get it out into society where they could use it."