A Few Thoughts on the NYC Charter Schools Evaluation Project

The NY Times profiled a report on charter schools that was released today.  The group conducting the investigation was led by Caroline Hoxby, an economist based at Stanford's Hoover Institute who has published a number of articles in prestigious journals that find positive returns to school choice.  The report is written in plain English and is very easily readable even for those who have little background knowledge or statistical training (which was quite a surprise to me given the highly technical nature of most of her work).

The bottom line of the report is that students who enroll in charter schools do much better than those who enter lotteries to enroll in charter schools, but don't have their number picked out of the hat.  The methodology of comparing "lottery winners" to "lottery losers" is probably the best way to go when evaluating charter schools because you should end up with comparison and treatment groups that are demographically similar.  Though, of course, that doesn't mean it's perfect.  A few thoughts:

-It's unclear whether students who are lottery winners apply to more charter schools than those who are lottery losers (one could imagine that a more motivated parent would apply to 10 charters and be more likely to end up with their child in one of them than the parent who applies to 1).

-It's also  somewhat unclear how they treat a student who leaves or enrolls in a charter school in a subsequent year.  They mention that about 14% of students leave charter schools (p. 71), but I didn't notice a part where they addressed the latter.  (correction: the 14% figure is the percentage of treatment group students that have transferred to traditional public schools over the life of the study, at least 17% have transferred to other schools and 24% have dropped out of the study for various reasons (p. 72))

-The study points out that lottery losers are 32% more likely to transfer to a private school or a school outside of NYC than are lottery winners (p. 72), but I'm not sure they fully investigate the effects of this.  It sounds as though lottery-losing leavers have test scores that are no different than lottery-winning leavers, but this isn't really the full story.  Current test score is only one predictor of future test score, and since they're measuring growth in test scores, I'd have to imagine that motivation is a better predictor of growth in test scores.  And as far as I can tell, it's entirely feasible that more motivated students are more likely to leave the system if they're not enrolled in a charter school.

-They estimate the average growth in test scores for charter school students to be .12 (math) and .09 (reading) standard deviations above and beyond those of lottery losers each year (p. 42).  For one year, that's not particularly impressive, but they calculate that these effects add up over time to create quite a powerful effect.

-Perhaps more interesting are the effects they calculate for individual schools.  Charter schools are supposed to vary widely, so it shouldn't really be possible to generalize to all charter schools.  Here are how the effects of charter schools broke down in each subject (p. 57-59)

	reading	math
negative	8%	14%
0 - .1 SD	16%	17%
.1 - .2 SD	45%	59%
> .2 SD	31%	10%

Assuming that these measurements are accurate and that higher test scores reflect more meaningful growth as well (neither of which should be taken for granted), I'd argue that schools over .2 might be worth getting excited about and schools between .1 - .2 are probably doing something right.  Making those two dangerous assumptions, we might say that a majority of schools seem to be doing something right.  It also becomes evident why, as I've argued in the past, charter school advocates should advocate closing more charters.  Think how much higher the estimates would be if the bottom 20% or so that are getting returns that are negative or no different from zero were closed.

-Why are charter schools doing better (same two assumptions apply)?  The authors measure the effects of a wide variety of variables on page 64.  The most notable number is that adding 10 days to the school year boosts achievement .15 SD.  This is the only number that is both meaningful to me and statistically significant at traditional levels.  The average charter school has about 10 more days of school than the average traditional public school.  Meanwhile, the average charter school advantage is about .10-.11 SD.  Which raises the possibility that charters are doing better mostly because they have longer school years -- a reform which could very easily be replicated in all schools.

-You should ignore all the calculations regarding the benefits of attending a charter high school.  In the last year of data, there are four charter high schools in NYC.


I'll e-mail the team about the first two bullet points, hopefully I'll hear back -- I'll post the answers if and when I do.