Sunday Commentary: Do NYC Charter Schools "Cream?"

Probably.  At least that seems to be the most sensible interpretation of the recently released report on NYC charter schools.  And it's important to note that the Wall St. Journal gets this dead wrong in their editorial on the report.  The main conclusion of the report was that students who enrolled in NYC charter schools performed much better than did students who tried, but failed, to enroll in these charter schools.  So I found it more than a bit odd that the Wall St. Journal's headline trumpeted something else.  Why lie when there's plenty of truth to extol?

Nowhere in the report is the claim made that charter schools do not "cream" (select only the top students, like one would cream the fat off of milk, for those unfamiliar with the lingo).  The claim that charter schools don't cream seems to be based on the fact that charter applicants are more likely to be black or poor than the average NYC student.  This is an important, and entirely unsurprising, piece of information.

But here's what they missed: the authors go out of their way to stress that they cannot accurately compare the prior test scores of charter school applicants to non-applicants, writing that "it is not possible to draw conclusions about how charter school applicants' achievement compares to that of students in New York City's traditional public schools" (p. 28).

Nonetheless, they display a chart on the same page with calculations that the average test score of charter school applicants is no different than the average score of non-applicants.  This despite the fact that applicants were much more likely to be poor or Black and much less likely to be White or Asian.  And any cursory glance at test scores in NYC will tell you that the average poor Black student scores far below the average non-poor White or Asian kid.  Which means that we should see average scores considerably below the citywide average from charter school applicants.  The fact that there is no difference would indicate, to me at least, that charter school applicants are out-performing other demographically similar students.

Similarly, the authors write on page 71 that students who are accepted into, but do not enroll in, a charter school have prior test scores no different from those who do enroll.  But, at the same time, admitted students who choose not to enroll are much more likely to be White or Asian.  Which raises two questions: 1.) If the authors had previously argued that there wasn't test data on enough students to make any claims about test scores, why are they now making claims about test scores? and 2.) If the non-enrolling admitted students are more likely to be White or Asian, would we not expect them to have higher previous test scores than do enrolling admitted students?

Besides, comparing test scores is, to some extent, beside the point.  It might very well matter more how motivated a student and their family is, what kind of support they are receiving at home, etc.  If you were running a school and your goal was for students to make as much progress as possible, which would be more important qualification for students: previous success on tests or the willingness to work hard?

In the end, we need to see both disaggregated data (e.g. comparing Black enrollees to Black non-enrollees) and comparisons of test scores and other variables controlling for various demographic variables before we can make any firm claim about NYC charter schools creaming students.  But the authors' position is firm: they can't tell.  And if you ask me, all signs indicate that they likely do.  Either way, the Wall St. Journal still needs run a correction.

Last, but certainly not least, should we care if charter schools cream?  If charter schools enroll more motivated or more capable students, might that not be a good thing (especially since they're clearly attracting students from the most at-risk demographics)?  The answer, of course, is that it depends.  It's hard to imagine that enrolling more motivated and higher achieving kids in a school wouldn't be a good thing for those who attend that school.  At the same time, there's a possibility that it might negatively influence kids in the surrounding schools that are losing the cream of the crop.  But it's hard to say to what extent either of these scenarios are playing out.  At this point, whether or not creaming makes charter schools bad is largely in the eyes of the beholder.

But, more importantly, it affects the way that charter schools should be analyzed -- because, ultimately, charter schools are supposed to serve as incubators of ideas that spread to other schools.  Which means that we should make an effort to figure out which of these ideas are worth replicating.  And if charter schools cream a little or a lot and we don't take this into account, it skews both the analysis of their performance and the analysis of which ideas should be replicated.  Which has the possibility of harming the students we ultimately want to help.