One of the most e-mailed articles in the NY Times shortly before Christmas was this piece by Helen Ladd and Edward Fiske on social class and educational achievement, in which the authors call for more non-school interventions ("education policy makers should try to provide poor students with the social support and experiences that middle-class students enjoy as a matter of course"). Overall, I thought it was a pretty good piece, but two things in particular struck me.
1.) That they build an argument for focusing on what happens outside of schools and then their first recommendation is to expand pre-schools.
2.) The recommendations after the pre-school discussion are fairly vague.
While the first is interesting, I'm more intrigued by the second -- and I wonder to what extent it's because they want to recommend that we change 30 things they can't possibly list in the limited space and to what extent it's because they're not sure exactly what to address.
Which begs the question: what do we know about which non-school programs will make a difference? One particularly promising young scholar has argued that we don't yet know enough (you'll get the joke if you click on the link) to draw many conclusions on the topic.
The authors are certainly right that "Large bodies of research have shown how poor health and nutrition inhibit child development and learning" and they could've included numerous other factors at the family and neighborhood level. Since we know that these social factors and environmental conditions are causally related to academic performance, trying to ameliorate their impact on low-income children makes all the sense in the world. But, at the same time, I have yet to find (after extensive searching) a whole lot of evidence that we've been able to successfully do this in ways that rigorous research has found subsequently improved academic performance. And Russ Whitehurst argues the point even more strongly, writing in a recent report that "There is no compelling evidence that investments in parenting classes, health services, nutritional programs, and community improvement in general have appreciable effects on student achievement in schools in the U.S."
Let's take a look at the few programs they do mention in the piece. When I search Google Scholar for research on the programs they name, this is about all I can find on the East Durham Children's Initiative, Syracuse's Say Yes to Education program, Omaha's Building Bright Futures, and Boston's Citizen Schools. Only the last one links the program to any educational outcomes, and it appears to be an internal report. If there's evidence in peer-reviewed academic journals that these programs have improved students' academic performance, I've yet to see it (note: this is not to say that any of these four aren't working, just that we don't yet have really good evidence that they are).
At this point, some of you may be saying "you forgot about the Harlem Chidren's Zone!". That's certainly the most-cited example of social policy impacting academics. But there's a funny thing about that. As far as I can tell, only one study has linked HCZ to academic outcomes. And one thing that recently caught me eye is a chapter by Roland Fryer and others in the new Duncan/Murnane book on inequality and schools (highly recommended, btw). In particular, I find it interesting how they've changed their tune on HCZ the past couple years.
In 2009, Fryer put out an NBER working paper with PhD student Will Dobbie arguing that the HCZ had effectively closed the black-white achievement gap. The paper got all sorts of play in the press, with David Brooks claiming it proved once and for all that the "no excuses" schools were all that we needed and some of the Broader, Bolder folks replying that, no, it proved once and for all that community resources made the difference.
Shortly thereafter, I asked Geoffrey Canada which it was when he visited Vanderbilt -- he said that we needed both and that it was a "terrible, phony debate" to try and separate them. Nor could Dobbie and Fryer definitively separate them; in the introduction, they write (emphasis theirs) "We cannot, however, disentangle whether communities coupled with high-quality schools drive our results, or whether the high-quality schools alone are enough to do the trick." (p. 4)
But now they've updated the paper and, according to Fryer's Harvard info page, it's been accepted at the American Economic Journal: Applied Economics. This is from the abstract: "We conclude with evidence that suggests high-quality schools are enough to significantly increase academic achievement among the poor. Community programs appear neither necessary nor sufficient."
This would go nicely with the new book chapter (here's a slightly different version) in which they write, on the first page:
The evaluation of the Harlem Children's Zone allows us to conclude that a high-quality school coupled with community-based interventions does not produce better results than a high-quality school alone, offering further evidence that school investments offer higher social returns than community-based interventions.
That seems like a rather sweeping statement to make based on one preliminary estimate of one program's effects but, nonetheless, their findings do put the burden of proof back on those supporting the Broader, Bolder position.
The closest thing I've seen to a collection research citations indicating that we do have evidence that community-based interventions can work is David Kirp's recent book, but even that involved a good deal of cherry-picking and mostly discussed small programs not explicitly linked with local schools.
So, where does this leave us? As I wrote above, we have plenty of evidence that a wide range of experiences associated with living in poverty negatively impact kids' academic performance. And we have plenty of reason to believe that altering these experiences could, potentially, improve kids' academic performance. But I, and others, would argue that we have precious little empirical evidence that social policy has (or will) alter kids' lives in ways that will subsequently improve their grades, test scores, graduation rate, attainment, etc. So I find it a bit odd that Ladd and Fiske conclude by writing
But let’s not pretend that family background does not matter and can be overlooked. Let’s agree that we know a lot about how to address the ways in which poverty undermines student learning. Whether we choose to face up to that reality is ultimately a moral question.
I'd make a different pitch if I were they. I'd write something more along the lines of this: Let's not pretend that family background and living conditions don't matter and can or should be overlooked. Let's agree that we know a lot about how poverty undermines student learning and how large this impact is. And let's agree that we urgently need more research on ways to address the links between poverty and education. The Promise Neighborhoods and other initiatives deserve our full attention and support in the short-run and can potentially provide that will help us better address the problem in the long run.
Of course, twice as many words with half the certainty is a really bad formula for an op-ed. And there's no quicker way to frustrate policymakers than to write "more research is needed."
But, at the same time, I'm not sure it's helping their cause to claim that we know how to solve the problem. If I'm in charge of a new Promise Neighborhood, my immediate reaction would be "We do? Great!" Quickly followed by asking "which factors should I aim to address and which programs do we know are best to address these?" I don't know the answer to that, and I've yet to hear from anyone who does.
So, in the end, I'd say there's about as much empirical evidence that social policy will close the achievement gap as there is that charter schools, merit pay, and vouchers will close the gap. That is, very little. So if we insist on arguing for an either/or approach, this leaves us at a standstill. Both sides can yell that the other side's evidence is weak. Which doesn't seem particularly productive to me.
As a researcher, this seems like an excellent argument to conduct a lot more research on the links between social policy and academic performance (as well as on in-school interventions). Were I a policymaker, I'd want to avoid putting all my eggs in one basket. We know the status quo doesn't work, but we can't really say for sure what else would be better. That seems like a golden opportunity for policymakers and researchers to work together and experiment (literally) with a wide variety of reforms -- the former would get to hedge their bets and look prudent and open-minded while the latter would get to conduct groundbreaking research on a crucial issue.
In sum: Do we have conclusive evidence that a particular set of non-school interventions will close the achievement gap? No, we don't. So let's not claim we do. But, let's also vow to keep searching for it.
7 comments:
Helen Ladd is a wonderful person but I fear she has the same cure for depression that most academics do, "It would be terrible if there was nothing we can do, so there MUST be something we can do, and by 'we can do' I mean government programs administered by graduates of this and similar universities."
(The depression in this case is, "I feel so awful. Poor people do poorly in school, and they keep doing poorly in school, and if they don't do better in school, they're screwed." An alternative approach might try to lessen how much one is screwed by lack of academic success. Not surprisingly, this is not the first thing that comes to the academic mind.)
P.S. Congrats on getting published. It looks like an interesting article.
I think the "cure for depression" you note is a very real phenomenon, but I think it's actually less prevalent among academics than others. I generally find (with many exceptions, of course) researchers to be more skeptical than others when it comes to the efficacy of various policies (or, at the very least, the efficacy of policies other than the ones they study).
I think you raise an interesting idea about essentially reducing the penalty incurred by those with a lack of education. A prof I know recently raised a similar idea and it's had me thinking. On the one hand, it makes sense to focus on creating opportunities for the less educated since we're never going to have 100% of the population composed of college graduates (in fact, it's unlikely to approach even half in my lifetime). On the other hand, more education is strongly associated with so many positive factors (not just high test scores, but improved self-control, lower teen pregnancy, better health, and a million other things), that it also seems dangerous to risk de-emphasizing education in any way.
p.s. Thank you . . . It's no longer interesting to me because I've read it so many times.
On the other hand, more education is strongly associated with so many positive factors (not just high test scores, but improved self-control, lower teen pregnancy, better health, and a million other things), that it also seems dangerous to risk de-emphasizing education in any way.
And playing on a high school basketball team is strongly associated with being tall. That doesn't mean the causation runs from being on the team to being tall.
People who start off with better self-control, ability to defer gratification, and a million other things are the people who will do better at anything, including school.
Take another social state that is associated with a lot of good things: higher income, greater happiness, better health, and so on--marriage. Some conservatives say that this means the government should do a lot to encourage marriage. Most liberals disagree. They think that people who have positive qualities are more likely to attract someone willing to spend their whole life with them. Encouraging--or even forcing--marriage is not going to magically give people those qualities.
Is encouraging--or forcing--people to go to school really going to give them greater self-control, better health, etc?
Roger, two responses:
1.) Yes, raising the education levels of the population will result in people with better self-control, better health, etc. (in part as a direct result of their education and in part because they will find better jobs and lead a less stressful life).
2.) Even if this weren't the case, improving self-control and health and decreasing teen pregnancy and stress (and altering a hundred other factors in desirable ways) will result in students who learn more, score higher on tests, and are more likely to attain degrees.
So, yes, I think that increasing academic performance and educational attainment will result in future positive lifestyle changes for the population. But even if this isn't the case, changing kids' lifestyles, attributes, etc. in ways that will make them better citizens and job candidates will also result in better academic performance and educational attainment.
So, either way, trying to improve our country and the economic prospects of our citizens involves (either directly or indirectly) improving the performance of students in schools.
1) Why should "raising the education levels of the population ... result in people with better self-control, better health" any more than raising the participation on high school basketball teams result in a generally taller population?
In some very broad sense ("a sound mind in a sound body") becoming educated involves better self-control, taking better care of oneself, etc. However, I am not at all sure that increased years of schooling leads to that. A truly intrusive school, that seriously attempted to "educate the whole person" might do that. Perhaps a military academy or a strongly religious school. However, given the way public schools work, I am skeptical.
2) There's the same direction of causality problem here. Getting kids to spend extra years in school doesn't magically increase their self-control, reduce teen pregnancy, etc.--any more than forcing people to get married would.
Like so many people in the ed business, you have a touching faith in the ability of schools to fix things. But "correlation is not causation." If the people who go in least broken come out least broken and the people who come in most broken come out most broken, that suggests that the schools are not responsible for either the good outcomes or the bad.
I'm sorry; that was snotty. But we're talking years of kids' lives and dollars of adults' money. I think faith is an inadequate reason to take either one.
(It is certainly true that people with more schooling get better jobs but a large part of that is just employers using education as a filter. Lots of jobs that once required a high school diploma now require a college degree, not because the jobs have gotten any harder but because so many more applicants now have high school diplomas. You have to read a lot fewer applications, and do a lot less interviewing if you say, "college degree required." And since the people who graduate college are the people who started with better self-control, etc., you're still getting the top of the applicant pool.)
Roger,
You're not wrong to question the influence of schools in the sense that those with the most supports at home tend to both enter and leave as the highest achievers.
But you're taking it a step too far by asking whether schooling really has any effect on students at all. Causation and correlation are certainly difficult to untangle, but we have decades of research that consistently find that additional years of schooling lead to all sorts of positive outcomes in a variety of ways.
Additional schooling increases the knowledge and aptitude levels of students, gives them more exposure to potential mentors, makes them more informed citizens, gives them something productive on which to spend their time, makes them more attractive candidates on the job market (both due to the "sheepskin effects" of the diploma and due to the increased skills), and a million other things.
Yes, attending school develops self-control and self-discipline -- without which one cannot sit quietly in a classroom or complete a term paper. Yes, attending school improves health outcomes when a student is able to make better and more informed choices and their occupation affords them better health care. Yes, attending school decreases teen pregnancy rates as students both have less leisure time and more concrete immediate plans. And we could discuss another hundred examples as well.
Now, I'll be the first to argue that other areas of one's life often impact these factors more than do schools -- but that's a different argument than saying that schooling does affect any of these. Certainly schools can't do everything -- and what they do do, they can do better -- but there's really no question that more schooling positively impacts the average student in myriad ways.
I have no doubt that school affects people in various ways. What I'm not sure of is exactly what, and how much. I suspect that for many things the answer is, "not much."
Since school takes thousands of hours from young people's lives, and since many of them find it painful, I think it is important to be clear on exactly what it is accomplishing.
(Now, for many people the "take thousands of hours" is a feature, not a bug. This is part of the "day care" function of schools. Give kids a safe, structured place to go for 7 hours a day, where they won't be exposed to bad ideas and temptations, and where they will engage in pro-social activities that will help them grow up into better people. For example, as you mention, it's hard to get pregnant if there's an adult watching you all the time.
To some of us who aren't in academia, that raises the question of whether there are ways to get the good day care stuff without subjecting lots of people to frustration and failure, and others to seemingly endless cycles of memorizing and forgetting in the name of learning. Are there less "academic" alternatives to traditional school?)
Nobody doubts that more years of school are correlated with all the things you say they are. But correlation is only a starting point. Which way does the direction of causation run? To the extent that causation runs from schooling to good outcomes, how important it it (e.g. does it reduce the teen pregnancy rate from 10 to 5 or from 10 to 9.9?) How much does some third factor cause both the additional years and the good outcomes? (I haven't read it but I gather that The Bell Curve argued that lots of good things are caused by IQ.)
If you put sea weed on your indoor English Ivy and leave it off your outdoor marigolds and then find that the ivy grows considerably better, it is NOT legitimate to say, "People should put sea weed on all their plants because it helps them grow."
A well-controlled experiment would require lots of pairs of plants, where both members of each pair got the same sun, water, amount of room, etc.--where the only difference between the members was getting or not getting seaweed.
In any non-experimental study, you try to "control" for differences by statistically manipulating the data you have about your subjects. Unfortunately, you never have data on all the things you should control for. In fact, you never even know what all the things you should control for are.
If you do some simple statistical analysis, you find out that overweight people have a lower death rate than people who are not overweight. Does this mean that overweight people are healthier? Probably not. Most people who die have been declining for a while, and part of that decline often involves losing weight.
I have very little contact with the primary literature that purports to show that schooling does all sorts of wonderful things. What I have seen does a poor job of controlling for all the other things that affect life outcomes. So perhaps I am complaining out of ignorance. Perhaps it does control nicely.
(I was about to hit publish when the question occurred to me, "Have any studies tried to untangle the effects of academic knowledge and the "day care" effects of school? How much does a better life outcome come from things like knowing the law of conservation of mass, and how much from the structure school provides, the reward for following directions and such?" Again, I am ignorant.)
Post a Comment