Evaluating the Evidence on Non-School Interventions

I've been meaning to finish writing this piece for six weeks, and now I finally have.  Enjoy.

One of the most e-mailed articles in the NY Times shortly before Christmas was this piece by Helen Ladd and Edward Fiske on social class and educational achievement, in which the authors call for more non-school interventions ("education policy makers should try to provide poor students with the social support and experiences that middle-class students enjoy as a matter of course"). Overall, I thought it was a pretty good piece, but two things in particular struck me.

1.) That they build an argument for focusing on what happens outside of schools and then their first recommendation is to expand pre-schools.

2.) The recommendations after the pre-school discussion are fairly vague.

While the first is interesting, I'm more intrigued by the second -- and I wonder to what extent it's because they want to recommend that we change 30 things they can't possibly list in the limited space and to what extent it's because they're not sure exactly what to address.

Which begs the question: what do we know about which non-school programs will make a difference?  One particularly promising young scholar has argued that we don't yet know enough (you'll get the joke if you click on the link) to draw many conclusions on the topic.

The authors are certainly right that "Large bodies of research have shown how poor health and nutrition inhibit child development and learning" and they could've included numerous other factors at the family and neighborhood level.  Since we know that these social factors and environmental conditions are causally related to academic performance, trying to ameliorate their impact on low-income children makes all the sense in the world.  But, at the same time, I have yet to find (after extensive searching) a whole lot of evidence that we've been able to successfully do this in ways that rigorous research has found subsequently improved academic performance.  And Russ Whitehurst argues the point even more strongly, writing in a recent report that "There is no compelling evidence that investments in parenting classes, health services, nutritional programs, and community improvement in general have appreciable effects on student achievement in schools in the U.S."

Let's take a look at the few programs they do mention in the piece.  When I search Google Scholar for research on the programs they name, this is about all I can find on the East Durham Children's Initiative, Syracuse's Say Yes to Education program, Omaha's Building Bright Futures, and Boston's Citizen Schools.  Only the last one links the program to any educational outcomes, and it appears to be an internal report.  If there's evidence in peer-reviewed academic journals that these programs have improved students' academic performance, I've yet to see it (note: this is not to say that any of these four aren't working, just that we don't yet have really good evidence that they are).

At this point, some of you may be saying "you forgot about the Harlem Chidren's Zone!".  That's certainly the most-cited example of social policy impacting academics.  But there's a funny thing about that.  As far as I can tell, only one study has linked HCZ to academic outcomes.  And one thing that recently caught me eye is a chapter by Roland Fryer and others in the new Duncan/Murnane book on inequality and schools (highly recommended, btw).  In particular, I find it interesting how they've changed their tune on HCZ the past couple years.

In 2009, Fryer put out an NBER working paper with PhD student Will Dobbie arguing that the HCZ had effectively closed the black-white achievement gap.  The paper got all sorts of play in the press, with David Brooks claiming it proved once and for all that the "no excuses" schools were all that we needed and some of the Broader, Bolder folks replying that, no, it proved once and for all that community resources made the difference.

Shortly thereafter, I asked Geoffrey Canada which it was when he visited Vanderbilt -- he said that we needed both and that it was a "terrible, phony debate" to try and separate them.  Nor could Dobbie and Fryer definitively separate them; in the introduction, they write (emphasis theirs) "We cannot, however, disentangle whether communities coupled with high-quality schools drive our results, or whether the high-quality schools alone are enough to do the trick." (p. 4)

But now they've updated the paper and, according to Fryer's Harvard info page, it's been accepted at the American Economic Journal: Applied Economics. This is from the abstract: "We conclude with evidence that suggests high-quality schools are enough to significantly increase academic achievement among the poor. Community programs appear neither necessary nor sufficient."

This would go nicely with the new book chapter (here's a slightly different version) in which they write, on the first page:

The evaluation of the Harlem Children's Zone allows us to conclude that a high-quality school coupled with community-based interventions does not produce better results than a high-quality school alone, offering further evidence that school investments offer higher social returns than community-based interventions.

That seems like a rather sweeping statement to make based on one preliminary estimate of one program's effects but, nonetheless, their findings do put the burden of proof back on those supporting the Broader, Bolder position.

The closest thing I've seen to a collection research citations indicating that we do have evidence that community-based interventions can work is David Kirp's recent book, but even that involved a good deal of cherry-picking and mostly discussed small programs not explicitly linked with local schools.

So, where does this leave us?  As I wrote above, we have plenty of evidence that a wide range of experiences associated with living in poverty negatively impact kids' academic performance.  And we have plenty of reason to believe that altering these experiences could, potentially, improve kids' academic performance.  But I, and others, would argue that we have precious little empirical evidence that social policy has (or will) alter kids' lives in ways that will subsequently improve their grades, test scores, graduation rate, attainment, etc.  So I find it a bit odd that Ladd and Fiske conclude by writing

But let’s not pretend that family background does not matter and can be overlooked. Let’s agree that we know a lot about how to address the ways in which poverty undermines student learning. Whether we choose to face up to that reality is ultimately a moral question.

I'd make a different pitch if I were they.  I'd write something more along the lines of this: Let's not pretend that family background and living conditions don't matter and can or should be overlooked.  Let's agree that we know a lot about how poverty undermines student learning and how large this impact is.  And let's agree that we urgently need more research on ways to address the links between poverty and education.  The Promise Neighborhoods and other initiatives deserve our full attention and support in the short-run and can potentially provide that will help us better address the problem in the long run.

Of course, twice as many words with half the certainty is a really bad formula for an op-ed.  And there's no quicker way to frustrate policymakers than to write "more research is needed."

But, at the same time, I'm not sure it's helping their cause to claim that we know how to solve the problem.  If I'm in charge of a new Promise Neighborhood, my immediate reaction would be "We do? Great!"  Quickly followed by asking "which factors should I aim to address and which programs do we know are best to address these?"  I don't know the answer to that, and I've yet to hear from anyone who does.

So, in the end, I'd say there's about as much empirical evidence that social policy will close the achievement gap as there is that charter schools, merit pay, and vouchers will close the gap.  That is, very little.  So if we insist on arguing for an either/or approach, this leaves us at a standstill.  Both sides can yell that the other side's evidence is weak.  Which doesn't seem particularly productive to me.

As a researcher, this seems like an excellent argument to conduct a lot more research on the links between social policy and academic performance (as well as on in-school interventions).  Were I a policymaker, I'd want to avoid putting all my eggs in one basket.  We know the status quo doesn't work, but we can't really say for sure what else would be better.  That seems like a golden opportunity for policymakers and researchers to work together and experiment (literally) with a wide variety of reforms -- the former would get to hedge their bets and look prudent and open-minded while the latter would get to conduct groundbreaking research on a crucial issue.

In sum: Do we have conclusive evidence that a particular set of non-school interventions will close the achievement gap?  No, we don't.  So let's not claim we do.  But, let's also vow to keep searching for it.