Apples to Apples? Not Necessarily

Mike Petrilli has a good post over at Flypaper on the Hoxby et. al. charter school study that was recently released.  It's good to see I'm not the only one pointing out that the Wall St. Journal and others shouldn't be writing NYC charters don't cream -- because they probably do.

But I have to take issue with one thing that Petrilli writes about the study: "because it’s a gold-standard random-assignment study, we can be sure that it’s an apples to apples comparison".  Not so fast Mike . . . the random assignment part might well be true, but there are plenty of reasons to think that this isn't quite an apples-to-apples comparison.

First, in regards to the random assignment* designation: it may not actually be random assignment.  I have yet to hear back regarding the number of charter schools each kid applied to, but it's certainly possible that those who won a charter school lottery applied to more charters, on average, than those who lost a charter school lottery.  In other words, it's possible that the winners were more motivated than the losers to begin with.  On the other hand, it's also possible that losers were more likely to apply to charters with long waiting lists -- and it might be true that those with longer waiting lists are superior schools that attract superior students.

In addition to these possibilities, I can think of at least one other way that the study design could result in a comparison that is not quite apples-to-apples (note that the authors did address some of these in the paper):

Attrition may not be the same for charter enrollees as it is for the others.  22% of the lottery-losing students transferred to a school outside of NYC or a private school, while 14% of the charter students transferred to a traditional NYC public school.  While it's likely that both groups leave their current school due to some level of dissatisfaction, it may be the case that those who transfer from charter schools to another NYC school do so because they're struggling at their school and/or are encouraged to leave due to discipline or other problems.  Meanwhile, it may be the case that those who transfer from traditional public schools to non-NYC schools do so because they view their schools as failing them and think they can do better elsewhere.  In other words, it's plausible that the most motivated non-charter students transfer out while the least-motivated charter students transfer out.  The authors do take a look at this and conclude that there's no difference in the relationship between test scores and likelihood of leaving between the two groups, but there could be no difference in test scores but a large difference in motivation, attitude, family support, or a myriad of other factors that are more likely to lead to growth in test scores (which, remember, is the outcome variable in the study).


*For those of you without a research background, a random assignment study is just what it sounds like -- a study in which those being studied were randomly assigned to a control group or treatment group (for example, if we flipped a quarter for every kid who walked in the door and those who got tails were assigned to Mrs. Johnson's class and those who got heads were assigned to Mr. Smith's class).