A few days ago on Real Time with Bill Maher, Rick Lazzio was discussing crime trends in the United States, which appear to be declining in spite of the poor economy. Maher pointed to research linking violent crime to childhood lead exposure, and suggested that simple remedies like removing lead from paint and gasoline can sometimes have profound effects on certain kinds of social problems.
I think there is some truth to Maher’s comments, but I would also argue that it’s extremely difficult to measure these kinds of things accurately.
The study that Maher is referring to was popularized by a 2007 article in the Washington Post examining then-Republican presidential hopeful Rudy Guliani’s record of crime reduction in New York City. The author, Shankar Vedantam, cited evidence by economist Rick Nevin which seems to show that “lead poisoning accounts for much of the variation in violent crime in the United States.”
I’ve been thinking a lot lately about how research findings are reported in the media, and I’ve come to believe that journalists have a responsibility to discuss the potential limitations of various methodological approaches. This article is a case in point.
Nevin’s study relies on time-series regression models, which explain national crime rates as a function of the blood lead content rate in preschoolers. Looking at various industrialized countries, he shows that “the violent crime rate tracks lead poisoning levels two decades earlier” in virtually every instance.
I find this conclusion very appealing, but the research design strikes me as problematic for two reasons.
First, lots of factors are associated across time. In fact, the key problem with time series analysis is that you almost always find highly significant relationships with large coefficients of determination. When I was doing research on joblessness a few months back, for example, I came across this study showing that the (lagged) frequency of certain kinds of Google keyword searchers explains about 89 percent of the variation in German unemployment. That’s pretty astounding, but no one in their right mind would assert that keyword searchers actually cause unemployment. This relationship probably holds in almost every industrialized country, and we may even find similar lag times. None of this means that Google is killing jobs.
Second, even if Nevin relied on cross-sectional data, there are some important confounding factors that he probably couldn’t capture very well. In particular, it’s likely that some measure of social responsibility may explain at least part of the relationship between lead poisoning and crime rates. Heightened levels of social responsibility could lead to public calls for lead removal from various products. This same change in the social mindset could lead to more responsible parenting, which would likely lead to lower rates of crime about 20 years later when males are most likely to become violent offenders. Finding a proxy for “social responsibility” is pretty difficult, and Nevin doesn’t seem to control for this at all in his study.
The general point here is that inferring causality is difficult, and most studies have drawbacks that really need to be highlighted. I don’t expect every journalist to become an expert on research design, but I do think that it’s incumbent upon them to know something about the limitations of the specific research they’re summarizing for the public. Peer-review doesn’t guarantee a flawless study.
Maybe I'm putting too much blame on journalists, but I suspect the public might be a little less cynical about seemingly contradictory research findings if they had a better idea of how reserach is conducted, and at least some understanding of how we actually "know" the things we think we know.
I’ve been thinking a lot lately about how research findings are reported in the media, and I’ve come to believe that journalists have a responsibility to discuss the potential limitations of various methodological approaches. This article is a case in point.
ReplyDeleteThis is something I've thought about for quite some time. It's why I was so excited to take "Math for the Media" at Georgetown. Studies are regularly promoted in the media, but findings, especially in health, are often contradictory and don't have much significance, or their significance can't be accurately sussed out. It's really difficult to understand surveys, financial statements and budgets without some sort of training, and it's really difficult to write about when you don't understand. (I know, because this is a regular component of my job.)
I remember, in I believe What the Dog Said, that Malcolm Gladwell's advice for aspiring journalists is to major in statistics. I am absolutely of the belief that those in the media should not shy away from math and statistics. But it's also important to note that space and the market dictates a lot of what is written. The Washington Post, at the top of the top, has the ability to print a long and involved article on lead poisoning and crime, and try to explain the numbers. Aggregators, the evening news, and other outlets are only going to go for the take-away.
In fact, the key problem with time series analysis is that you almost always find highly significant relationships with large coefficients of determination. When I was doing research on joblessness a few months back, for example, I came across this study showing that the (lagged) frequency of certain kinds of Google keyword searchers explains about 89 percent of the variation in German unemployment. That’s pretty astounding, but no one in their right mind would assert that keyword searchers actually cause unemployment. This relationship probably holds in almost every industrialized country, and we may even find similar lag times. None of this means that Google is killing jobs.
The link is dead, and without it, I am unsure of what you are saying.
Here's the link: http://ftp.iza.org/dp4201.pdf
ReplyDelete(I'll fix it in the text, too.)
The key problem with time series analysis is that lots of factors are correlated over time, but - as I'm sure you know - correlation doesn't necessarily imply causation.
The coefficient of determination (R-Square) is just the square of the correlation coefficient. It's supposed to tell you what proportion of the variation in your dependent variable (crime rates) is explained by your independent variable (lead poisoning). But, again, "explaining" doesn't mean causing.
There are still lots of possible confounding factors that could be responsible for both the declining crime rates and the declining rate of lead poisoning.
The Google Econometrics example is just intended to illustrate the point that even a time series regression with two factors that we KNOW aren't actually causally related can have an insanely high R-Square. It doesn't prove anything.
-Jeremy
Well, sure. Even if some data stream is pure random, if you have a lot of data you can predict that with high probability some pretty improbable things will happen. For example if the data is binary, with a long enough stream, you will see a row of 1 million 1's with probability 0.999.
ReplyDeleteHow this pops up here is illustrated by the Texas Sharpshooter Fallacy:
A drunken Texan one evening went out and shot up the side of a barn and then passed out. Since he was shooting randomly the bullet holes were all over the place, but there were some places where they clustered together. When he awoke the next morning, he looked at the barn and drew bull's eye targets around the clusters. Then he told everybody what a good shot he was even when he was drunk
This came up in the case Emily cited in "A Civil Action." There were lots of child leukemia cases in Woburn, MA, but was it reasonable to assume that was simply a random event? It turned out that the situation was finite enough, so that one could prove is was very unlikely.
So if you look at a lot of parameters, you will expect to find some unrelated ones that correlate in any way you want to measure. There was a great Sci-Fi story based on this idea, but I've lost it.
So what's my point? It is often hopeless to try to _prove_ causality, but in the real world it is usually hard to _prove_ anything. What you need is a theory with perhaps other pieces that can be checked to make it reason that causality occurs. For example, perhaps one could argue that the brain damage caused by the lead makes this kids more likely to become criminals. This could, maybe, be checked by following the lives of a statistically signifcant number of children with lead poisoning.
Anyway, that's what I try to do.
It is often hopeless to try to _prove_ causality, but in the real world it is usually hard to _prove_ anything. What you need is a theory with perhaps other pieces that can be checked to make it reason that causality occurs. For example, perhaps one could argue that the brain damage caused by the lead makes this kids more likely to become criminals. This could, maybe, be checked by following the lives of a statistically signifcant number of children with lead poisoning.
ReplyDeleteAgreed. I'd only add that not all study designs are created equal. A true experiment is inherently less suspcious than a case-control study, and a regression discontinuity design is inherently less suspicious than than a time-series regression.
Some situations just don't lend themselves to true experimentation (or any kind of design with a strong degree of internal validity), and it's always more difficult to infer causality in these situations. I just think that journalists should understand and acknowledge this.
Here, I think you're absolutely right. If I were going to design this experiment, I'd gather up a fairly large pool of kids with lead poisoning, use some form of propensity score matching (PSM) to find a comparison group that didn't get lead poisoning, and then follow those kids over time to see whether we find significant differences in terms of criminal activity.
There are stil some serious flaws with this design. What if there are important variables that are unobservable and must be left out of our PSM? What happens if kids in the comparison group get lead poisoning? These potential problems reduce our internal validity. Doesn't mean we should scrap the study, but it does mean we should be a little more suspicious of the results.
-Jeremy