Don't Interpret an Article's Findings based its Title

I've mentioned a few times that research is nowhere near as definitive as I imagined it was before I started grad school. Every paper has flaws, limitations, and shortcomings -- no matter how the prestigious the journal in which it was published. Peer review isn't perfect. Sometimes people are so focused on examining an issue in one manner that they forget to look at it in other ways.

I found one such example yesterday. In an influential piece in the American Economic Review (a very prestigious journal) written by Caroline Hoxby* in 2000 ("Does Competition among Public Schools Benefit Students and Taxpayers?") found that schools with more competition were "more productive" (higher scores with less money spent). In the paper, competition is determined by the number of school districts within a metropolitan area and the number of streams (rivers/creeks, not any fancy academic term) in that same area. The theory is that more streams led to the creation of more towns (and, hence, more school districts) because they provided geographic boundaries. Those areas where more streams led to the creation of more districts were said to have more choice b/c people could choose to move to more districts closer to home and, therefore, the districts had to compete more not to lose students. The statistics are really sophisticated, and we devoted two full periods of class to examining the underlying theory, the statistical analysis, and later re-analysis (which got quite heated, and in which I have no desire of involving myself at this point in time), but that is the general gist.

Anyway, we were focused on understanding how more competition and choice led to more productive schools when a thought occurred to me. Metropolitan areas that have more streams and, therefore, smaller towns and smaller school districts, don't just have more school districts nearby -- they also have smaller towns and smaller districts. I wondered aloud whether it was possible that the effects of smaller districts (perhaps a tighter-knit community, more parental involvement, less bureaucracy, etc.) might explain the positive effects in addition to, or instead of, more choice.

I'm still somewhat in doubt that the author and numerous reviewers missed that point, but I don't see any indication that district size was controlled for and can find no other method of counting this out. So, for now, my conclusion is that people get stuck thinking about something one way and forget to think about it in others.


*I often refrain from naming names. I mention it in this case only b/c understanding my thought is wholly dependent upon knowing to which article I am referring. I have absolutely no desire to start a fight with Caroline Hoxby (who has proven many times over that she is quite smart -- I particularly like her outside-the-box idea of using streams as exogenous predictor of # of districts); I simply wanted to share an interesting thought on education policy.