Why Does Massachusetts Rank Highly?

In last week's debate, Mitt Romney took credit for Massachusetts' position atop some education rankings. And, yes, it's generally true that Massachusetts ranks at or near the top.  More specifically, the state has frequently had the highest average score on the National Assessment of Educational Progress (NAEP).

But the more important question is why Massachusetts ranks so highly.  Was it something that Romney did while Governor, or are there other factors at play?

The second question is really quite easy to answer.  It's almost certainly something other than Romney's actions.  For two reasons:

1.) Children in Massachusetts earned really high test scores both before and after Romney was Governor:

	2002	2011
1.)	Vermont (272)	Massachusetts (275)
2.)	Massachusetts (271)	New Jersey (275)
3.)	Montana (270)	Connecticut (275)
4.)	Nebraska (270)	Vermont (274)
5.)	Maine (270)	Montana (273)

2.) We know from decades of research that non-school factors influence achievement far more than in-school factors.  So it's exceedingly unlikely that a few state-level policy tweaks, implemented for a mere four years, could impact student performance dramatically enough to boost Massachusetts to the top of the nation.

We can also argue to what extent the high test scores mean the state's schools are a success.  We could certainly measure student and school performance in myriad other ways.  And even if we look only at test scores, we can go beyond the averages.  Massachusetts has one of the largest gaps in achievement between upper- and lower-income students, for example.  Though, again, that likely has little to do with Romney -- the state ranked 5th in 2002 and 6th in 2011 (measured as the average 8th grade reading score of those not eligible for free/reduced price lunch minus the average score of eligible students).

That said, I wanted to explore this a little more in-depth, so I went to the NAEP website and delved into the 8th grade reading scores.  The first thing you'll notice on the site is the map of state results (below) which shows striking regional disparities in test scores.  If we assume that Governors are almost solely responsible for the average test scores in their states, we could only conclude that virtually all Northern governors are education geniuses and almost all Southern Governors education dunces.  Which, of course, is preposterous -- there are clearly larger issues at play here (issues out of the hands of the various Governors).

What are these issues?  The socio-economic status of the states' residents would be at the top of the list (certainly, a Governor would have some power to influence that over the course of one or more terms -- but that change would be both slight and slow).  To examine this, I downloaded the state NAEP scores from the NAEP data webpage and demographic data from census website so that I could compare the two.

Unsurprisingly, a fairly strong correlation exists between a state's average 8th grade reading score and a state's median household income (r = .43).  When we plot all the states' average test scores and median household incomes on the graph below, we see a few outliers -- Montana, Kentucky, and Vermont score much higher than we'd expect given their average incomes while California, Hawaii, Nevada, and Alaska score much lower -- but Massachusetts is right about where we'd expect it to be (note that the best fit line would run right through Massachusetts if we deleted the outliers).  Massachusetts has relatively wealthy residents and high-scoring students.  Not a surprise.

Massachusetts stands out even more if we look at the education levels of the population.  The percentage of adults with a bachelor's degree in a state is very strongly correlated (r = .65) with the average NAEP score in that state, and Massachusetts ranks at the top of both categories.  Once again, we see some outliers -- both positive (Kentucky, Wyoming, and Montana stick out) and negative (Mississippi, Louisiana, New Mexico, California, and Hawaii don't look too good), but find Massachusetts right about where we'd expect.

So, yes, Romney was correct when he said that Massachusetts ranks at the top.  But it's exceedingly unlikely he had much to do with that.  Massachusetts' residents were and are wealthy and highly educated relative to the residents of other states, and that mostly explains why their children perform so well on tests.


cross-posted on Blog of the Century

It's All About Vocabulary?

The edusphere is abuzz about this NY Times piece on early vocabulary growth that ran over the weekend. Though the piece focuses on the current controversy surrounding test-based admissions to the top high schools in NYC, it's mostly based on the famous Hart and Risley book in which the authors conclude that children from families on welfare hear 32 million fewer words and 560,000 fewer encouragements than children of professional families between birth and age 4 -- and that these differences lead to subsequent differences in vocabulary and achievement.

To reinforce the importance of this early vocabulary growth, the article quotes a charter school principal saying that the "word deficit" is the greatest challenge the school faces and quotes E.D. Hirsch saying that "there is strong evidence that increasing the general knowledge and vocabulary of a child before age six is the single highest correlate with later success".

This all leads Robert Pondiscio to write that "Demography isn't destiny. Vocabulary is destiny".

Ok, stop.  Just stop.

Yes.  Vocabulary is important.  And vocabulary growth in the early years is crucially important.  Every person or organization responsible for raising young kids should aim to use and explain as many words words and concepts as possible.

But, c'mon.  Let's not get carried away.

The second we identify something -- anything -- as the "single" most important, we do ourselves and our nation's children a disservice.  I understand the allure of boiling everything down to the simplest solution possible, but life just doesn't work that way.

And arguing that vocabulary -- rather than demography -- is destiny?  That's just silly.

For starters, we have what economists would call an "endogeneity problem" in that statement.  An awful lot of what's driving the vocabulary of a child entering kindergarten is also driving the success of that kid later in school: parenting, myriad environmental conditions and social factors in the child's home and neighborhood, health, peers, genetics, and a thousand other things.  In other words: a child with a large vocabulary at age 4 is likely to succeed in school partly b/c of that vocabulary, but more so because the conditions that created that vocabulary will almost certainly continue to foster intellectual growth and social development throughout his/her school years.

Second, an awful lot of what drives vocabulary growth is demography.  The education level of one's parent(s) and other adult supervisors, the amount of stimulation available in one's surroundings (including the number of different objects one can learn about), the noise level inside and outside of one's home, the levels of stress to which a child and his/her family are exposed, the number of books available, and hundreds of other home, neighborhood, and family factors correlated with socio-economic status all result in a child learning more or fewer words.  So arguing that vocabulary is more important than demography in school is like arguing that strength is more important than weightlifting in football.

So, please, let's stop trying to reduce everything to the one, most important factor (which is surely more important than the factor the last person discussed).  The fewer things we focus on, the more distorted those measures become.  And the simpler we make the problem seem, the more simplistic our solutions.  Vocabulary certainly deserves some of our attention.  Now let's discuss what else deserves our attention instead of how much less they deserve it.

The Differences Between High- and Low-Performing Schools: Not What You Might Expect

Last Thursday, the National Center on Scaling Up Effective Schools (a research center led by Vanderbilt's Peabody College with partners at UNC-Chapel Hill, Florida State, Wisconsin-Madison, Georgia State, and the Educational Development Center, funded by a five-year, $13.6 million federal grant, which aims to identify and then explore ways to scale up, characteristics of effective high schools) released a new report examining the differences between high-and low-performing high schools in Broward County, Florida.

As only one small piece of the puzzle, we shouldn't get carried away with the findings.  But I was struck with what was -- and was not -- included in their list of differences between the schools.  Below is the Executive Summary's list of differences:

We identified one major theme that cut across all ten components: personalization for academic and social learning. In the area of personalization, our findings show that the higher value-added (VA) schools made deliberate efforts through systematic structures to promote strong relationships between adults and students as well as to personalize the learning experience of students. In addition, the higher VA schools maintained strong and reliable disciplinary systems that, in turn, engendered feelings of caring and, implicitly, trust among both students and teachers. Leaders at the higher VA schools talked explicitly about looking for student engagement in classroom walkthroughs as well as in their interactions with students. Teachers at the higher VA schools were more likely to discuss instructional activities that drew on students’ experiences and interests. The higher VA schools also encouraged stronger linkages with parents (p. 5).

Included: "soft" factors like trust and relationships.

Not included: virtually everything currently discussed in ed policy circles (school choice, teacher evaluations, merit pay, data-driven decision-making,  etc.)

Now, to be fair, many of factors were off the table because the study examined four schools located in the same county which had much in common (no differences in merit pay or district leadership, for example).  And there's always the possibility that implementing some of these reforms could change the factors included in the list even if they're not currently present in the schools.

Nonetheless, even when the measuring stick is value-added scores -- the latest, greatest measure being pushed on schools -- many of the factors emphasized by those pushing for its use don't seem to be drivers of the differences.

Most interestingly, the two low-scoring schools had higher scores on some measures of teaching practices and instructional quality than the two high-scoring schools.  Here's the summary from the research team on this topic:

Taken together, our indicators of the quality and nature of instruction across the schools -- CLASS-S*, course matrices, student shadowing, and interviews with multiple school stakeholders -- reveals no major differences in instructional quality across the four schools.  We cannot turn to evidence in the area of Quality Instruction to explain the differences in value-added achievement between our high- and low-VA schools" (p. 32).

While we certainly shouldn't base our policy decisions on one study examining four schools in one county, I do think it's fair to say that this confirms what we should've known all along -- that high-quality instruction (like every aspect of schools) is not sufficient when trying to create high-performing schools.  I should also note that, in many areas, larger differences existed between the honors and regular tracks within the schools than between the high- and low-scoring schools themselves.

Again, I don't want to get carried away with the results of one small-scale study (and I'll refrain from addressing the other 50 or so topics covered by the report at the moment), but I do think that, at this point, we can take two important lessons from this ongoing research:

1.) Regardless of the amount of press coverage, foundation money, or policy directed toward a particular aspect of school reform, not a single factor is sufficient to create a high-quality school.

2.) Even though it's easy (and, arguably, practical) to focus on the simplest, starkest issues, the most subtle, nuanced, and complicated ones are often at least as important.

From a 10,000 foot vantage point, the potential benefits of creating more charter schools, or implementing a merit pay plan or new curriculum are easy to see.  But, on the ground, it probably matters more how than whether those things are implemented -- without strong relationships, trust, and commitment, it's unlikely any reform will turn around a school or district.

That fact is really difficult for policymakers to swallow because there's no easy way to change those types of things: what is Congress supposed to do in order to make make teachers at the local elementary school get along better with their students?  The relationship between policy and the factors discussed in the report is so indirect that it's easy to just ignore them and focus on simpler solutions.  We should all try to resist that temptation.



*"the Classroom Assessment Scoring System for Secondary classrooms (CLASS-S), [is] an observational tool developed by researchers at the University of Virginia, to observe and assess the quality of teacher-student interactions in classrooms. Based on development theory and research suggesting that interactions between students and adults are the primary mechanism of student development and learning (Greenberg, Domitrovich, & Bumbarger, 2001; Hamre & Pianta, 2006; Morrison & Connor, 2002; Pianta, 2006; Rutter & Maughan, 2002), the CLASS-S focuses not on the presence of materials, the physical environment, or the adoption of a specific curriculum but on what teachers do with the materials they have and on the interactions teachers have with their students. The observation tool looks specifically at interactions between teachers and students across four domains: Emotional Support, Classroom Organization, Instructional Support, and Student Engagement" (p. 12).

Schools Rallying Communities

I have strongly mixed feelings about charter schools, but my biggest concern is one I almost never see mentioned by charter proponents, detractors, or neutral observers.  We hear a lot about how communities affect schools, but almost nothing about the reverse.

I grew up in a suburban district where people routinely headed to the local high school for football games, basketball games, school plays, and scads of other events.  And, to a lesser extent, the elementary and junior high schools brought in community members for fairs, concerts, etc.  All in all, the schools brought the community together quite often for various reasons.  And that's not uncommon.  Or at least, historically, it hasn't been uncommon.

But that might be changing.  If we imagine a world where schools and neighborhoods are completely decoupled and people from one town go to scads of different schools all over the place, that relationship almost ceases to exist.  We won't read stories like this piece in the NY Times about a small-town HS football team that's rallying the community.

Granted, it might be worth the trade-off if the new non-neighborhood schools dramatically outperformed our traditional school system, but it's important to recognize that there is a trade-off involved here.  And that schools have larger ripple effects on society beyond the academic performance of their current students.

Frustrating Work Conditions and Schools

It seems like a growing number of people give lip service to working conditions in school, but without many specifics.  If teachers are frustrated by the working conditions in their schools, how would we expect their behavior to change?

We're using Bolman and Deal's textbook in the Organizational Theory course I'm teaching this semester, which includes a large section on human resources in organizations.  Part of that section discusses Chris Argyris' work on the differences between human personality and management practices.  Argyris contends that workers have six options when trying to escape frustrating working conditions (pp. 128-130).  See how many of these seem familiar to you when thinking about schools:

They withdraw -- through chronic absenteeism or simply by quitting
This certainly happened at my school -- working conditions were so bad that the vast majority of teachers took all 10 of their sick/personal days each year (which compounded the problem, since we usually couldn't find any subs to come into the building).  I'm not sure what's been published on the topic, but I do know that if one looks through the NYC School Report cards that a lot of schools average a lot fewer teacher absences.

They stay on the job but withdraw psychologically, becoming indifferent, passive, and apathetic
This is the quintessential "bad teacher" right here.  The tenured burn-out who can't be bothered to do much of anything anymore.

They resist by restricting output, deception, featherbedding*, or sabotage
Sounds just like the legion of obstinate teachers who refuse to implement the latest, greatest curriculum or other reform handed down to them from above.

They try to climb the hierarchy to better jobs
As teachers in my school used to say: "those who can, do; those who can't, teach; those who can't teach, become principals" (I should note that there's some emerging evidence that many principals had above-average VAM scores when they were teaching).  Either way, it's pretty clear that a lot of teachers try to escape the classroom to become coaches, coordinators, and administrators of all types.  In my school, the most veteran teachers who hadn't moved into one of those types of positions all taught in positions that got them out of the classroom (e.g. "resource room," in which they'd pull out a couple kids at a time).

They form alliances (such as labor unions) to redress the power imbalance
Unions certainly play a large role in many schools.  What we often forget, though, is why the unions came about.  If teachers aren't frustrated and don't distrust their supervisors, they don't usually form (or utilize) unions.

They teach their children to believe that work is unrewarding and hopes for advancement are slim
I haven't seen any evidence of this happening with teachers . . . hopefully it doesn't get that bad.


I definitely see evidence of five out of these six behaviors, though it's unclear whether any of these are currently increasing.  I'd argue, though, that ameliorating the conditions that lead to these types of behavior should be one important goal in our quest to raise teacher quality and turn around low-performing schools.

If we instead go the opposite direction (sterner management, scripted curricula, etc.), we risk turning our schools into highly organized, poorly performing factories.  Taken to the extreme, teachers essentially become mindless drones.  The authors quote Ben Hamper (a former factory worker who then wrote about his experiences) saying that "Working the Rivet Line was like being paid to flunk high school the rest of your life" (p. 131).  Work like that certainly won't inspire anybody to become the high-quality teachers we all agree we need.


*"Featherbedding is a colloquial term for giving people jobs that involve little or no work" (p. 138).

Guest Post: Teacher Opposition to VAM

by Kerri Tobin, PhD

Teacher opposition to value-added modeling gets portrayed in the media as teachers refusing to take any kind of responsibility for student learning or academic growth. But most teachers do not balk at the idea that they should be accountable for advancing their students’ knowledge. What they oppose is value-added modeling, or VAM, the highly-imprecise tool that is being used to measure teachers' impact on students' learning.. When the New York Times won the right to publishteacher VAM scores this spring, only the tiny print noted the inaccuracy of the data, including some standard errors larger than the purported effect sizes. Researchers familiar with value-added modeling have repeatedly voiced their concern about its use in high-stakes decisions like teacher evaluations (for example: here, here, and here).

Often, opponents of current uses of standardized testing fall back on attempts to draw parallels between teaching and medicine: “It’s like grading doctors on how many of their patients die!” But these analogies are imprecise; they do not do the problem justice. John Ewing’s fertilizer analogy is fascinating but perhaps too long for the common person. What we need is an “elevator speech” – one that can be delivered in the time it takes to get from the ground to the 3rd floor – to explain why teachers oppose VAM. A better analogy is this: using VAM to evaluate teachers is akin to evaluating chemotherapists based on how much their patients’ tumors shrink every year. The expected rate of tumor shrinkage is calculated based on patients’ race and socioeconomic status. So if a patient’s tumor shrinks more, over that year, than expected, the doctor gets a positive score. If it shrinks less than it “should,” the doctor gets a negative score. The average of these patients’ scores becomes the doctor’s overall VAM score. Sound reasonable? Maybe, until you consider that a) different kinds of tumors respond differently to chemotherapy; b) the doctor has no control over what patients do outside the office (for example, lung cancer sufferers who continue to smoke); c) his patients saw a different chemotherapist last year and will see another one next year; and, perhaps most importantly, d) the doctor is not allowed to treat any co-morbid conditions (for example, a cancer patient with diabetes gets no treatment to manage his blood sugar) – even if he wanted to, and even in cases where the patient or his family might prefer that other conditions be treated instead of the cancer (e.g., when parents value social or self-confidence issues more than test scores), there simply aren’t enough hours in his day. Factors like the overall health of the patient, his lifestyle, eating habits, substance use, weight, and blood pressure might impact the effectiveness of the chemo. But the doctor cannot control these, in much the same way teachers cannot control where students live, if they have enough to eat and get regular medical care, whether anyone reads to them at home, how much TV they watch, or what time they go to bed at night. And in the same way our VAM-assessed doctor would be powerless to decide that a patient dying of AIDS needs antiretroviral therapy before he can tolerate chemo for a concurrent cancer, teachers have neither the time, resources, nor training to solve the problems in their students’ lives – emotional problems, health challenges, family issues, etc. – that impact their academic growth. This is how VAM works, and why teachers oppose it.

-------

Kerri Tobin is Assistant Professor of Education at Marywood University in Scranton, PA, where she researches the educational needs of students living in poverty and prepares teachers and school leaders to meet those needs.

The Two Types of TFA Alums

Almost ten years ago now (where has the time gone?), I figured out what I wanted to do after college: I wanted to join Teach For America.  Eight years ago, I began teaching in The Bronx through a sister program, the NYC Teaching Fellows (one of many organizations under the umbrella of The New Teacher Project, started by a TFA alum -- the interviews for TFA and the three TNTP programs I applied to were almost exactly identical).  While in NYC, many of my closest friends were NYCTF or TFA members and I left for grad school with a rosy picture in my mind of how we would change the world.

I think it's precisely because these types of programs are so near and dear to my heart that I've grown so frustrated with TFA.  For over 20 years now, they've done a world of good in countless ways.  But I've always believed that their greatest impact would be the actions of their alumni.  The recruitment materials I pored over my junior year in college told us that we could see how the system was failing from the inside and then go out and fix it -- whether that be by remaining in the field of education or as a lawyer, politician, school board member, concerned citizen, or whatever other route we chose.  The idea was that, over time, an army of active citizens with elite credentials and experience in our failing inner-city and rural schools would wield enough influence to finally fix what ails our educational system.

That's starting to come true.  Countless TFA alums have become principals, begun charter schools, and so on.  One became the head of the whole DC school system.  And now we're beginning to see TFA alums entering politics (my local school board race in Nashville, for example, featured two TFA alums both vying to unseat the chair of the board).

For most of the past decade I've waited for this day with bated breath.  Finally, I thought, we'd start to see some change.  The conditions that I couldn't believe our society would tolerate while I was teaching would finally start to be addressed.  But now I'm worried.

While I don't doubt that TFA alums will have an outsized influence on our educational policy in the coming decades, I'm no longer convinced that the results of this influence will be all good.

Why?  Because too many TFA alums took the wrong lesson from their experiences in the classroom.  I've interacted with (both personally and professionally), heard, and read countless TFA alums over the past decade, and I now generally lump them into one of two groups: the humbled and the hubristic.  It's a crass generalization, and many alums I know don't neatly fall into one group or the other, but I still think it's helpful in thinking through what changes we should expect as TFAers gain clout.  So, without further ado, here are the two types of TFA alums:

1.) The Humbled - If I chose one word to describe my classroom experience, it would be "humbling" ("frustrating" would be a close second).  I went in believing that I could change the world in one fell swoop, that I would surely be the world's greatest teacher, and that we could easily fix most of our problems if only we could find more miracle workers like myself.  By day two I realized that I wasn't the world's greatest teacher on that day.  And by day five I started to think it might not happen for at least a couple more weeks.  What followed was two years in which I valiantly fought losing battle after losing battle until I was utterly exhausted.  During those two years, I saw the underbelly of one of the lowest performing middle schools in NYC (its closure was announced the spring of my second year), and formed quite a few opinions regarding its failure.

But one notable item missing from my list was the quality of the veteran teachers in the building.  I knew I was smarter than some of them.  I knew I worked harder than all of them.  But I'll be damned if most didn't teach circles around me -- and many found a way to do it for decades while I lasted all of two years before I became a statistic.

My major takeaway from that experience was that fixing the problems that look so simple from the outside is really hard.

And I know a lot of TFA and TNTP alums who will tell you something similar.  Some are disillusioned.  Some are frustrated.  Some are neither.  But all came to realize that they can't do this on their own, and that it's not going to be easy.

This group of humbled alums are more likely to push for educational and societal reforms and policies that change the context in which schools operate.  They know that if you can address poverty at the family and neighborhood level, then school will go a lot smoother; that if you can change the attitudes and outlooks of students they'll learn a heck of a lot more regardless of the teacher; that if teachers are treated as professionals they'll rise to the occasion; that if teachers are given resources and support they'll both stick around longer and teach more effectively while they're there; and that schools, in general, need our help.

2.) The Hubristic - Many TFAers had very different experiences than I.  Research on TFA generally finds that their teachers' students make gains equivalent to or slightly higher than other teachers (sometimes even higher than veteran teachers).  Which means that a lot of corps members receive results each year telling them that they are the world's greatest teacher (or at least one of the best in their school).  Some of these are flat-out better teachers than I, some simply found the right fit, landed a position at a top-performing school, or received oodles of support.

Some, though not all, of these teachers see their school -- and its veteran teachers -- very differently from how I saw mine (or, in the case of TFA members who teach at high-flying charters, see other schools and their veteran teachers differently).  They think to themselves "this isn't so hard; if we could find more people like me, we could lick this problem in no time".  I hear stories from them of how mind blowingly lazy, stupid, and/or incompetent the veteran teachers at their school were.  And they leave their school with a distinct sense that they could fix our schools if somebody would let them -- and if everybody would listen to them.

As a result, hubristic alums are more likely to push for educational reforms and policies that aim to separate the wheat from the chaff.  They know that if we can recruit better people into teaching, get rid of the dead weight (or at least get them to fall in line), stop making excuses, and give the superstars the reins that our schools will shine in no time; in short, that schools need to be shaken up.

It's good that different people bring different perspectives and ideas to the table, but I get the distinct sense right now that the latter group is winning in a rout.  And I'm not sure that's going to be good for our schools down the road.  I don't doubt that improving teacher quality would yield positive results, but I do worry that our narrow focus on such a nebulous trait will prevent us from addressing other, more serious problems.

If we focus solely on the human capital of our teaching force, we will fail to address the home lives or emotions of our students, the competence of our school leaders, the quality of our curriculum, or any number of other challenges our system faces on any given day.

Where will that leave us?  Best case scenario, we're left with a whole bunch of superteachers who miraculously and dramatically raise achievement regardless of any outside challenges or distractions.  Worst-case scenario, we're left with a crumbling system full of disenfranchised teachers who are unable to overcome the shortcomings of their school context, school leaders, curriculum, and other factors and either give up or leave rather than take the blame (or just get fired).

I don't doubt that TFA alums are well intentioned, earnest, and sincere, but right now we're closer to the latter than the former.  And they're not helping.


cross-posted on Blog of the Century

More Liberal Arts for the Least Affluent

I've disagreed with Peter Meyer multiple times in the past, bot in posts (here, here, and here), and in comments on his blog posts (which I'm not going to take the time to dredge up).  So I think it's fair that I point out that he recently wrote what I think is an outstanding post last week on college attendance and poverty.  Also, digging up those old posts just made me realize I've been misspelling his name; my sincere apologies.

Anyway, Meyer makes a strong case regarding why, in an ideal world, we should want everybody to attend college -- and how obtaining a broad, liberal education particularly advantages the most disadvantaged.  Among other things, he points out that:

-exposure to new ideas, new institutions, and new styles of thinking is particularly beneficial for those who were exposed to the fewest of these in their childhood

-a college education opens more options for students compared to limitations placed on them by hyper-specific vocational training

-underemployed college grads still make far more than non-college grads in the same field (a college educated dishwasher makes 83% more, for example)

-increasing college attainment hardly solves our problems but not sending more kids to college creates more

While zillions of logistical hurdles stand in the way of all students procuring a top-notch liberal college education, Meyer concludes by arguing that:

I personally don’t care if a kid decides not to go to college. I would, however, demand that every high school graduate at least be capable of reading (and understanding) David Leonhardt’s story—i.e., your options are probably pretty constrained if you don’t go to college—and that every district superintendent be judged by the number of his or her truly college-ready graduates. If a student decides not to go to college, fine. But at least he or she would have, I would hope, the option of going if he or she wanted to—which is better, I would assume, than not having that option after twelve years of schooling.

I can only find two small points of contention in the post:

1.) the argument that teaching poor kids "a new kind of thinking -- reflection" is the key to getting them out of poverty is either inartfully expressed or demonstrates a lack of understanding.  I'm leaning toward the former, since he also wrote a pretty good piece explaining the genesis of that quote.  At first glance, it might look like Meyer is arguing that kids are poor because they think wrong.  I think, though I could be mistaken, that this was actually was a way of saying that exposing kids to more culture, society, and ideas (e.g. plays, museums, concerts, lectures, etc.) will benefit those who previously had the least exposure.  Indeed, the program driven by this notion was the result of the suggestion of an impoverished prisoner who said that kids needed to get more involved with the what was happening downtown in order to interact in new ways with government, society, etc.

2.) While I agree with Meyer that, ultimately, we shouldn't force every kid to get a high-quality college education: that giving every kid both the option to obtain it and the understanding of how it will benefit them is a better policy goal, I do hope that he personally does care which path any given student chooses.  Given that he argues that more students obtaining high-quality educations improves the lot of our entire society, I'd certainly hope he would then wish that all students chose to obtain that type of education.

Non-Academic Skills and Enterprise Rent-A-Car

One of my principal complaints about current educational policy debates and the educational blogosphere as a whole has been the myopic focus by many on academic skills when so many people (students, parents, employers, and society at large) care at least as much about a long list of other things (e.g. sports, drama, arts, critical thinking, physical health, social skills, sleep, commute distance, peers, and on and on).

So this one tiny example caught my eye last week.  This is from the textbook we're using in the Organizational Theory class I'm teaching (it's on the first page listed in this search):

A focus on customer service enabled Enterprise Rent-a-Car to overtake Hertz and become the biggest firm in its industry. Enterprise wooed its midmarket clientele by deliberately hiring "from the half of the class that makes the top half possible" -- college graduates more successful in sports and socializing than in class. Enterprise wanted people skills more than "book smarts" (Pfeffer, 1998, p. 71) 

I wasn't surprised to read that a firm considered more than academic record when hiring employees, but I was surprised to read that a firm deliberately hired large numbers of recent grads with less-than-stellar academic records specifically because they were heavily involved in their college's social scene.  So I did some more digging and found the source of that little blurb.

 This is from page 71 of Pfeffer's book:

Simply hiring the "best and the brightest" may not make sense in all circumstances. Enterprise Rent-A-Car is today the largest car rental company in the United States . . . In a low wage, often unionized, and seemingly low employee skill industry, virtually all of Enterprise's people are college graduates. But these people are hired primarily for their sales skills and personality and for their willingness to provide good service, not for their academic performance. Dennis Ross, the chief operating officer commented "We hire from the half of the college class that makes the upper half possible . . . We want athletes, fraternity types . . . people people." Brian O'Reilly interpolates Enterprise's reasoning:

The social directors make good sales people, able to chat up service managers and calm down someone who has just been in a car wreck.

Granted, this is almost 15 years old at this point, but I have no reason to suspect that this is no longer true (and anecdotal experience that indicates it is), but it would appear to be evidence that, popular to conventional wisdom, at least one firm does care about that blurb about your fraternity on your resume.

To be fair, O'Reilly goes on to assert that "The Enterprise employees hired from the caboose end of the class have something else going for them . . . a chilling realization of how unforgiving the job market can be".  So one could argue that this hiring practice is based on the desire for cost-savings as well.  But I'd argue that if that were truly the main motivation, then Enterprise wouldn't hire college grads.

Indeed, Pffefer then continues to assert that "organizations should screen primarily on important attributes that are difficult to change through training and should emphasize qualities that actually differentiate among those in the applicant pool." -- One could make a strong argument that the types of social skills referenced above fit those criteria.

In the end, I suspect we all know -- at some level -- that other factors matter more than academic skills (in choosing a school, in gauging a student's/school's success, and in life), but the education commentariat sure doesn't seem to like to act on that knowledge.

Mad Libs for Writing about Strikes

The commentariat and blogosphere sure love a showdown between a union chief and schools chief -- as evidenced by the flood of articles and posts this week on the strike in Chicago.  Unfortunately, about 90% of what's written is entirely predictable and, in the end, utterly useless.

Why?  Because almost every article is written using a secret mad libs cheat sheet that gets rolled out every time there's any controversy involving unions.  The result is that we could replace a few words in just about any piece to fit any strike or labor dispute anywhere in the country.  Regardless of the location, context, or actual issues at stake.  I've posted the cheat sheets below so that you can follow along:


For those opposed to teachers' unions

The [insert controversy] in [insert city] could be a turning point for educational reform in this country.  The union's [insert mildly derogatory adjective] leader, [insert name], has decided to place the needs of adults over the needs of children and [insert local leader] has finally stood up to make sure these children aren't forgotten.  While teachers and unions in most other cities have gotten the memo, the teachers in [city] are stubbornly clinging to the past; the evidence that value-added scores matter is overwhelming, and yet they're standing firm against the district's plan to use VAM scores to account for [X]% of a teacher's evaluation.  It looked like the union might see the light, but after [leader] said [quote taken out of context], it's clear that he/she has his/her head buried in the sand.  This is, indeed, a sad day for the children.  All hope of rescuing our abysmal education system now lie with [district/city leader], who seems to be embracing this challenge, saying [quote taken out of context].  We should all wait with bated breath while this most important of events plays out.


For those who defend teachers' unions

The [insert controversy] in [insert city] could be a turning point for educational reform in this country.  The union's [insert synonym for "heroic"] leader, [insert name], has decided to stand up for the rights of teachers and unions everywhere while [insert local leader] refuses to treat teachers like human beings and is determined to bust the union.  While teachers and unions in most other cities have caved and accepted the anti-union agenda, the teachers in [city] are standing up for their rights and what's best for children; the evidence that value-added scores are unreliable is overwhelming, and yet the district refuses to budge from their plan to use VAM scores to account for [X]% of a teacher's evaluation.  It looked like the union might cave, but after [city/district leader] said [quote taken out of context], it became clear that the union must finally decide that enough is enough.  This is, indeed, a glorious day for unions, the dying middle class, and potentially our entire country.  All hope of preventing the gutting of our indispensable public education system now lie with [union leader], who seems to be embracing this challenge, saying [quote taken out of context].  We should all wait with bated breath while this most important of events plays out.

Why Not Blame Principals and Superintendents?

I'm teaching a course in organizational theory this semester, and our textbook lays out some basics right off the bat.  Among these is that "The most common improvement strategy is upgrading management." (p. 9)

Which struck me because few in ed policy seem terribly concerned with it.  Sure, we have some mayoral takeovers and some turnaround schools and new charters of course need to hire new leaders.  But, in the grand scheme of things, I see little to no effort to seriously upgrade the management of schools -- at either the district or school levels.

Our attention, instead, focuses disproportionately on teachers.  I don't want to suggest that blaming somebody else would be particularly productive, but I genuinely understand why more people aren't blaming educational leaders.

The authors continue "Modern mythology promises that organizations will work splendidly if well managed."  Which also struck me, since I rarely here about a school needing only a great principal to succeed.  And, actually, outside of a few success stories (e.g. the KIPPs), I don't hear much of anything about great school leaders.

So I guess the flip side would be that people don't particularly credit school and district leaders for success either.

I should add one caveat to both of those conjectures, though: at the very local level (think of your child's school or your local district), I think educational leaders tend to receive a fair amount of scrutiny and both the flak and praise that go along with that.

At the national level, though, I hear almost nothing.  Concerted efforts to recruit or train more talented leaders are few and far between.  New evaluation and merit pay schemes for leaders have been quietly implemented some places, but received little attention.  And op-ed pages are silent about the need to upgrade our school management if we want to solve the civil rights problem of our generation.  Puzzling.

Instability: Deal With It, or Address It?

One of the facts of life for many high-poverty schools and districts is constant flux and turnover among district leaders, school leaders, teachers, families, and students.  A typical high-poverty school, for example, sees about 20-25% of its teachers leave each and every year (and that's just at a typical school: imagine what happens at the thousands of atypical schools out there).

I got to thinking about this while listening to a presentation last month on the failure of a curricular reform.  The research that was being presented identified this turnover as the main reason for said failure.  And I buy that explanation: almost every teacher they trained left before the new curriculum was fully established (as did the Superintendent, and the new one wanted to push a different curriculum).

But the larger question, to me, is what to do about this instability and turnover.  The presentation concluded with the argument that future reforms should take this instability into account and essentially be turnover-proof.  That's certainly a pragmatic approach.  But it always makes me squirm when people propose what are essentially dumbed-down reforms for the most troubled schools while others get to do the real thing.

So I had a different thought: why not attack that instability directly?  If a new curriculum can't be successfully implemented because teachers, principals, and district leaders are constantly in flux, why not try to stabilize those three?  For example, a group of teachers and administrators at a school are given a bonus if they stay in place for, say, 5 years.  And the district could lock-in the new curriculum for the same period of time so that it won't change even if the district's leadership does.

Maybe that would be too difficult to do on a large scale, but somebody could certainly try it on a smaller one -- a single charter school might be a good place to start.  After all, charters were started to serve as incubators and laboratories for ideas that traditional public schools hesitate to try.

I'm Back . . .

Hello again, everybody.  The last six months have been busy and eventful, but things are settling down enough now for me to resume writing in this space.

The major update in my life is that I am now an Assistant Professor of Leadership and Policy at Niagara University (just up the road from the eponymous falls).  I'm excited to work with their new PhD program in Leadership and Policy, and I'm particularly excited to work at a college with such a deep commitment to serving the community -- especially the poor and the oppressed.  The city of Niagara Falls has fallen on hard times, and the University's tireless work to help revitalize the community is one of the major reasons that it's among only 24 schools in the country to be recognized as a "Best College" for service learning.

(And it doesn't hurt that the school consistently appears on the honor roll of The Chronicle of Higher Education's "Great Colleges to Work For".)

Anyway, I have a number of great posts in store for you as I face new challenges and embrace new opportunities.  I hope you'll come along for the ride.

update: we, once again, were named as one of 27 schools with exemplary programs in service learning

Education and the Declining Median Class

A column a while back by David Brooks and numerous other reviews of Charles Murray's new book raise an issue I've been meaning to write about: the growing gulf between different classes (or "social tribes," as Brooks labels them) in the U.S. -- not just in earnings, which we hear a lot about, but in both achievement and a number of behaviors related to achievement.  As Brooks writes about the White population Murray discusses:

There are vast behavioral gaps between the educated upper tribe (20 percent of the country) and the lower tribe (30 percent of the country) . . . Roughly 7 percent of the white kids in the upper tribe are born out of wedlock, compared with roughly 45 percent of the kids in the lower tribe. In the upper tribe, nearly every man aged 30 to 49 is in the labor force. In the lower tribe, men in their prime working ages have been steadily dropping out of the labor force, in good times and bad. People in the lower tribe are much less likely to get married, less likely to go to church, less likely to be active in their communities, more likely to watch TV excessively, more likely to be obese.

The NY Times followed up with a story about the achievement gap growing between rich and poor students in the United States.  The article was based largely on two chapters from a book that was released last fall by the Russell Sage Foundation (which, as I've previously mentioned, I highly recommend).

In one chapter, Sean Reardon from Stanford finds that "the achievement gap between children from high- and low-income families is roughly 30 to 40 percent larger among children born in 2001 than among those born twenty-five years earlier" and that the gap between students from families in the 90th and 10th percentiles of income is now "nearly twice as large" as the gap between blacks and whites (the opposite was true 50 years ago).

Since I've spent my entire post-college life either trying to teach low-income students or researching low-income students, the chapter jumped out at me when I first saw a preview of it last summer.  The fact that the achievement gap between rich and poor was actually growing (while the black-white achievement gap has essentially plateaued the past 20 years), jibed with what I've read about, and seen in, both our schools and our society.

But then, when I was preparing to discuss the chapter with my students last fall, I took a closer look at the trends.  And this is what I discovered: the real growth in the gap between rich and poor isn't actually between the richest and the poorest, it's between the richest and those in the middle.

I've pasted one chart from the Reardon chapter to show what I mean.  The gap between those at the 50th percentile and those at the 10th percentile -- the middle-poor gap, if you will (represented by the dashed line) has been fairly steady.  The gap between those at the 90th percentile and those at the 10th percentile, meanwhile -- the rich-middle gap, if you will (represented by the solid line) -- has grown rapidly, from under a quarter of a standard deviation for children born in the 1940's to almost three-quarters of a standard deviation for children born in the 21st century.

I actually missed that point when I first skimmed the chapter, but to me it's maybe the most pressing policy issue of the next few decades: how will middle-income Americans work and live?

As somebody who studies the effects of poverty, I'm pre-disposed to believe that poverty, and the low performance of children living in poverty, poses the largest educational problem.  And there's no doubt that that problem is large.  Even if the poorest kids aren't falling further behind the middle-income students, they're still far behind where they ought to be and farther behind the wealthiest kids than before.

But if the middle of the income distribution falls far behind as well, that could leave us in really serious trouble.  This chart of wage growth that EPI offered in response to Brooks' op-ed is one way of looking at the issue.  The wages of the working poor are awful, but it's the hourly wages of the middle-income folks that have seen the lowest growth in the last 20 years.  Indeed, the median household made only slightly more in 2010 than it did in 1978 ($49K vs. $46K -- Table H-6) when adjusting for inflation.

As the table below indicates, from 1991-2010, median income by education level declined considerably for the nearly 60% of the population with a high school diploma, some college, or an associates degree, held steady for the 20% or so with a bachelor's degree and increased for the 10% or so with a master's or professional degree (to reduce visual clutter, I leave off the 13% or so of Americans who did not reach, or did graduate from, high school and the 1% who earned a doctorate degree*).  Now, education levels increased a bit during the time -- so part of the explanation may be that there were more people in each of the higher groups -- but not by enough to change the fact that Americans with median levels of education are earning less now than they were 20 years ago.

Median income by education level.  Source: US Census, Table H-13

It's hard to imagine a burgeoning economy in any country that sees no real income growth for those at the middle of the income distribution.  But Murray, Brooks, and others (including liberals as well) also note other worrying trends concerning health, marriage, childbirth, etc.

Indeed, another recent NY Times article reported that the majority of babies born to women under 30 are now born out of wedlock.  But more important are the different rates by social class: "About 92 percent of college-educated women are married when they give birth, compared with 62 percent of women with some post-secondary schooling and 43 percent of women with a high school diploma or less".  And while the number of babies born out of wedlock has risen for all three groups, it's the middle group that's seen the starkest increase: in 1990 only 11% of children born to women in their 20's with some college were born out of wedlock, but by 2009 that number had more than tripled to 34%.

Again, the problem is more pervasive among the least-educated women (the comparable number is 51%), but the largest change is in the middle of the income distribution.

Now, you may have noticed that I titled the post "Median Class" instead of "Middle Class," and that's because what a lot of people define as "middle class" isn't actually comprised of people in the middle of the income distribution.  People often talk about college-educated adults belonging to the middle class, but fewer than 30% of adults have a 4-year degree.  Depending on which model of social class one uses, those in the middle of the income distribution (those who fall right around median) are usually classified as lower-middle-class or working class.  And I want to be clear that I'm talking about those who fall right around the median of the income distribution.

It seems to me that a large part of the challenge is economic.  Jobs that pay high wages to employees without high levels of educational attainment are fast disappearing (one could write "high wage/low skill" jobs as shorthand, but I don't possess the skills for most of these jobs so that seems inaccurate to me).  We still have a not-insignificant number of jobs in construction, trucking, the trades, manufacturing, and so forth that pay fairly well, but the number of those types of jobs has declined dramatically -- largely due to globalization and/or technology -- in recent decades.  And it seems unlikely that this trend will dramatically reverse.  In other words, it seems unrealistic to expect anywhere near 70% of the population to find stable employment with decent wages without a 4-year degree.

The result, it seems to me, is that instability and low wages are no longer the domain of only the poorest Americans.  And it seems reasonable to assume that the growth in the rich-middle achievement gap is due, at least in part, to the spread of this job instability to the median earners.

In short, it seems like there's a growing bulge in the middle of the income and education distributions that is lost.  Fewer and fewer can make a good living without a college degree.  They're falling further behind the wealthy academically.  And to make matters worse (to be intentionally and melodramatically blunt), recent reports say they're increasingly divorced, fat, and lazy as well.

That's worrying.  But what really worries me is that I don't see an easy solution. Brooks' integration idea may be a small step in the right direction, and improving our educational system would certainly be another. But neither seems likely to prevent the problem from getting worse in the next 20 or so years.  It seems unlikely we can do any, yet alone all, of the following in a short period of time:

1.) dramatically increase the number of stable, high-paying jobs for those without college degrees
2.) dramatically** increase the number of college-educated adults while also increasing the number of stable, well-paying jobs for those with degrees accordingly
3.) reverse social trends and encourage more two-parent households, more civic engagement, less obesity, etc.

But I hope I'm wrong.  I hope 20 years from now I'm writing about the resurgence of the median class and not about the spread of poverty to children of middle-income households.  Either way, I think recent data indicate we need to adjust our focus when we discuss ways to boost performance of the lowest achievers.  If we want to focus research and policy on those lagging behind, we need to broaden our scope beyond just the 10 or 20% lowest-income Americans.  Those in the middle aren't doing that much better.



*median income increased slightly from 1991-2010 for those with less than a 9th grade education --  from $20,640 to $21,254, decreased from $27,375 to $24,787 for those who attended, but did not graduate from, high school, and also slightly decreased for those with a doctorate as well -- from $121,693 to $119,825.


**"dramatically," in this case, does not mean the 10 percentage point increase we've seen over the past 25 years for adults over age 25 and especially does not the 10 percentage point increase we've seen over the past 35 years for those aged 25-29

Thoughts on "Educational Productivity"

Last week, Matthew Ladner produced a stunning chart showing an "implosion" in our nation's educational productivity.  Productivity here seems to be defined as ratio of per-pupil expenditures on public education to average NAEP test scores.  The former has tripled since 1970 while the latter has essentially remained flat for the upper grades.  I'm not sure of the impetus behind that particular post, but the Bush Center has written a similar post that makes all the same mistakes.

Before I delve into those mistakes, I'll point out that they've also created a nifty website that allows people to compare students' standardized test scores in any district's to the scores of other students in the state, nation, and world.*

So, what's wrong with comparing spending to achievement?  Seems straightforward.  And the graph is certainly compelling.  But, alas, the statistics that seem the most straightforward are often the least useful.  Among other issues:

1.) Spending and test scores are on different scales.  Spending can multiply almost infinitely while the test scores have a ceiling.  In the chart on the site, the average 17 year-old scored 306 out of 500 on the NAEP math test in 2008.  Which means that even if every kid in the country earned a perfect score the next time around, the average score would only increase about 63%.  Since school spending has tripled, the ratio of spending to achievement would still be far greater now than it was 40 years ago.

2.) Why would we assume that it takes the same level of effort for a school to get a student to earn a certain score now as it did in 1970?  A zillion factors other than education spending influence achievement levels.  If parenting ability, economic circumstances, living conditions, and such increased dramatically then we shouldn't think it's miraculous if scores increase with no additional school spending.  Similarly, if societal conditions worsen in some way, then it would necessarily mean that more effort is necessary to achieve the same scores.  I have no idea whether it's now harder or easier to get the average 17 year-old to score 306 on the NAEP now than it was in 1970, but we'd need to know that answer to accurately measure educational productivity.

3.) Why would we assume that the same level of spending is commensurate with the same level of effort on behalf of districts now as it was in 1970?  The economic and social context of schooling is dramatically different.  Perhaps most importantly, the number of women in the workforce -- particularly in fields outside of education -- has exploded.  Simple economics dictates that it must cost more to buy the services of an equally qualified teacher.

4.) Test scores were not the sole goal of that increased spending.  Surely, we also aimed to increase the number of high school and college graduates (I don't have the HS stats handy, but almost twice as many 25-29 year olds have bachelor's degrees now as did in 1970 (though it's increased by only about a third since 1975)).  I think a reasonable argument could be made that it's increasingly costly to get each additional student to graduate (i.e. moving the HS graduation rate from 50 to 60% is easier than moving it from 80 to 90%), so we might not expect the same returns per dollar on those measures.  And, surely, we also aimed to improve many other skills (e.g. critical thinking, physical/emotional health, social skills, art appreciation, etc.) that aren't measured by the math and reading tests listed.

So, should Ladner's alarming chart worry us?  I wouldn't dismiss it out of hand.  I wouldn't be the least bit shocked if our returns to effort and spending have decreased the past 40 years.  But we can't tell whether productivity has decreased, remained steady, or increased by looking at that chart.  The most compelling figures he presents are the large increases in non-teacher staff in schools, but some unknown number of support staff are certainly invaluable so even that doesn't prove all that much.

And, by the way, those same four problems apply to any international comparison of a simple spending : test score ratio.  Were we to completely eliminate schools, culture, society, and a myriad of other contextual factors would still produce kids who scored much higher and lower on tests in different countries; it would be harder and easier and cheaper and more expensive to change that in different countries; and each country emphasizes different outcomes to a different degree.

I'll be the last person to argue that our nation's schools are just fine -- we face countless problems with a nearly infinite number of solutions -- so please don't interpret my criticism as an argument for the status quo. I hate the status quo.  But also realize that Ladner's chart gives us exactly zero information about what ails our schools.

*The "world" here is 25 developed nations.  On another note, here's a fun game: they don't seem to have compiled the list of the top-performing districts, so go see which ones you think might rank highest.  So far, I've found:

-The average student in Chappaqua, NY outscores 89% of students internationally in reading and 82% in math

-The average student in Chatham, NJ outscores 88% of students internationally in reading

-The average student in Brookline, MA outscores 77% in math.

Is Teacher Quality Really Causing the Achievement Gap?

Yesterday, the NY Times released the value-added scores of thousands of teachers over the past five years.  Before and immediately after the release, people seemed to mostly argue the merit of the decision to release the data.  But I have a substantive question about the data.

What caught my eye was that, according to the NYT analysis of this particular set of scores, good teachers are evenly distributed between high-poverty and low-poverty schools.  From the article:

there was no relationship between a school’s demographics and its number of high- or low-performing teachers: 26 percent of math teachers serving the poorest of students had high scores, as did 27 percent of teachers of the wealthiest.

The LA Times reported a roughly a similar situation in LA when they released teachers' scores a couple years ago.  Which is really quite shocking in a number of ways.  Most notably, researchers and practitioners have long assumed that lower-poverty schools had worse teachers than higher-poverty schools -- past studies have repeatedly found that teachers in high-poverty schools are less experienced, turn over at a much higher rate, score lower on achievement tests, attend less selective colleges, etc.  Accordingly, at least part of the theory of action behind the teacher quality movement has been that giving low-income students teachers who are as good as or better than those in higher-income schools would significantly narrow the achievement gap.

But these two measures of teacher quality indicate there may be no major differences between low- and high-poverty schools, while we know that large gaps in achievement still exist between low-income and high-income students.  Which means at least one of two things.

1.) Differences in teacher quality are not a major driver of the achievement gap.

2.) These value-added scores are not a good measure of teacher quality.

I don't think anybody seriously doubts -- or at least that anybody serious doubts -- that some teachers are much better than others and that the best teachers can make a large difference.  But if quality teachers, according to these value-added measure, are roughly evenly distributed between high- and low-poverty schools in LA at the same time that we see differences between high- and low-income students growing, then  improving the quality of teachers (again,as indicated by these value-added measures) in high-poverty schools seems unlikely to close the achievement gap.  Either other factors influence achievement far more, the effects of quality teachers on students are much less direct than many assume, or what we're measuring isn't what matters.

In short, these data indicate that we need to broaden our focus beyond teacher quality and/or re-evaluate the way we're currently measuring teacher quality.

How Education Research is like Football Research

In the fall, I wrote about an instance in which outsiders may be needed in education reform.  Today I'll give you an example of how outsiders can also be dangerous (though this one pertains more to research).

Perhaps the largest change in educational research over the past decade or so has been the sizable increase in large-scale quantitative research, a fair amount of which is conducted by researchers outside of ed schools.  Like any change, this has resulted in both positives and negatives.  But one thing that worries me is that I consistently notice the people who are most worried about statistical rigor in quantitative analyses (both inside and outside of ed schools) tend to be less concerned with understanding the context and processes of schooling.

And that's incredibly dangerous.

Methodology, statistics, and technical skills are very, very important in the development of good research.  But without a proper understanding of how schools work and what is actually happening on the ground, one can't expect to ask the right questions.  And if one fails to ask the right questions, it really doesn't matter how complex and rigorous their analysis is because the answers to those questions are meaningless.

Here's one example of how such a process can unfold -- it's completely unrelated to ed policy, but I still think it's illustrative.  The Freakonomics Blog posted a brief discussion yesterday of the ending to the Super Bowl.  The post said two things (paraphrasing, of course):

1.) Isn't it amazing that the coaches of both teams realized that the Giants scoring a touchdown with about a minute left was actually a better outcome for the Patriots?  The Patriots' coaches tried to let the Giants run the ball into the endzone while the Giants' coaches instructed their players not to score a TD.  These counter-intuitive behaviors are an excellent example of game theory properly implemented.

2.) But then the Giants failed to take game theory into account when attempting their two point conversion.  Wouldn't it have been much better for them to run time off the clock instead of trying to score to go up 6 points instead of 4?  They might've been able to kill 20 seconds by running the ball 95 yards backwards and around in circles, and certainly being up 4 with 40 seconds left is better than being up 6 with a minute left.  Why didn't the coaches think of this?

There's some clever thinking going on here.  Yes, this is an interesting application of game theory.  And, yes, running 20 seconds off the clock would've been a better strategy.  So the application of economic theory to the situation is exemplary.  In a short space, there's a cogent analysis and a provocative question.  But there are two fatal flaws.

1.) Both coaches did not apply game theory.  Tom Coughlin, the Giants' coach, said he preferred that the team take the guaranteed six points to running down the clock.  So let's hold off on patting him on the back for correctly applying game theory.

2.) More importantly, the clock doesn't run during two-point conversions.  The Giants could've run around in circles for ten minutes, and there still would've been exactly 59 seconds left on the clock.

So, what we have here, is a smart professor who's well-trained in economic theory and statistics.  This training has allowed him to make an important insight about a football game and ask an interesting question.  Except that he doesn't actually seem to know much about the rules of football or the context of the situation.  Which  has rendered his question moot.

And I see the same thing (in a much less dramatic and much less foolish) way happening in education research.  Smart people with training in other fields and disciplines and serious methodological credentials come into the field and find some low-hanging fruit ripe for picking.  At first, this seems like a great idea.  We can never have too many smart, well-trained researchers in education.  And the eye of the outsider can be sharp.  But then the research starts and we realize that somebody can be smart and well-trained but, at the same time, fail to truly understand how schools work and the contexts under which students, teachers, principals, schools, etc. operate.  And then we get smart, well-trained people asking the wrong questions (or interpreting their findings in silly ways).  And that neither advances the field nor helps us improve our educational system.

Let's bring the analogy back to football.  Let's say that Football was a field in many Universities.  Grad students train under faculty who work for Schools of Football and/or Departments of Football Policy, Football Leadership, Football Teaching & Learning, Football Evaluation, Football Foundations, Football Studies, and so on.  And most of the research on football is conducted by faculty and grad students from these schools and departments.  There's no reason why an economist shouldn't do a study on the costs and benefits of attending school on a football scholarship; why a psychologist shouldn't conduct a study on the impact of playing football on one's personality; or a sociologist shouldn't conduct a study of the impact of playing football on one's social capital.  But in order to do these studies well, they first need to understand how the game of football is played, what a player does on the field, how much he practices, and so on.  Otherwise they're just chucking their theories against a wall and hoping one sticks somewhere.

So, to all the smart economists, psychologists, sociologists, etc. out there who wish to conduct the research on education: Welcome, we'd love to hear your insights and figure out if we can apply your theories and methods to help us advance our field and improve our schools.  But before claiming that you've solved a problem none of us have been able to for the last 100 years, take some time to learn how schools operate.  Read a massive and wide-ranging stack of literature.  Go visit some schools.  Talk to people who work in schools and education departments.  Talk to people who study those who work in schools and education departments.  Then begin your research.

At the very least, that should save you the embarrassment of asking students how many touchdowns they need to score in order to hit a home run on their fourth grade reading test.

The Logistics of "Thinning Out" Bad Teachers

Nick Kristof recently wrote another column calling for more high-quality teachers based on the latest paper on value-added measures of teacher quality.  There's a whole lot to discuss about both the column and the research paper, but let me focus for a minute on one small part of it.

Near the end of the column, Kristof writes that "If we want to recruit and retain the best teachers, we simply have to pay more — while also more aggressively thinning out those who don’t succeed. It’s worth it."  Recruiting, retaining, paying (and training, which is left out of this sentence) are all complex endeavors, but the "thinning out" part of the equation is often taken for granted.

Here's my question for Kristof: even if (and that's a big if) we can find a fair, accurate, and agreeable way to identify and dismiss the worst teachers, how many teachers are we actually going to dismiss in such a scenario?

The first question would obviously be whether we need to fire the bottom 5%, 10%, 25% or some other number.  That's up for discussion.

But the logistical question then, is how many teachers among the bottom X% 1.) can be readily identified and 2.) are planning on teaching again next year.  This will differ greatly by school and district, but in some places, this is going to be a very small number.Why?  Let's take a look at what the research says.

First, research consistently finds that it takes 3-5 years for a teacher to reach their potential.  So a good number of the lowest-performing teachers are simply going to be novices who will be better teachers next year.  We don't want to fire a first-year teacher who was in the bottom X% if we have reason to believe they'll be a really good teacher in a couple of years.  That would be incredibly counterproductive.

Second, research has consistently found that value-added measures of teacher effectiveness bounce around considerably from year to year -- particularly for teachers who teach a small number of students (e.g. a 4th grade reading teacher with 18 students versus an 8th grade math teacher with 150 students).  At least one paper has found that averaging scores over three years provides a much better, and more stable, estimate of teacher performance than does any single-year estimate.

Third, a number of recent papers have found that, at least in the first few years, many of the least successful teachers exit teaching.  This makes sense -- if you start a new career and find yourself completely overwhelmed, you're not likely stay very long.

Fourth, teacher attrition is exceedingly high in many high-poverty schools.  The general consensus is that about half of urban teachers leave the field within their first 3 years.

So, what does this mean?  We probably don't want to fire a whole lot of teachers in the first 3-5 years of their career because a.) they're still learning and improving; b.) we can't be that sure who the worst teachers are anyway; and c.) a good portion of the catastrophically bad teachers are self-selecting out of the field anyway.  If we give discount the first two years, when teachers are still learning their craft, and then take three more years to compute accurate value-added scores, it would only be teachers who'd taught for 5+ years who would really be ripe for firing due to low value-added scores.

Which means that the main herd we're trying to thin is the teachers who've made it through those first few years, reached their potential, and for whom we have accurate value-added estimates.  But how many teachers is that?  When I looked at high-poverty NYC middle schools a few years, I found that in the average school, only 1/3 of teachers had 5 or more years of experience.

Let's say that we're very confident in our ability to recruit and retain teachers who are better than our current teaching force and so we decide to fire all below average teachers (a full 50%) -- which would be a far more aggressive plan than any I've seen proposed.  First, the majority of these below average teachers are novices who are still improving and for whom we don't have particularly good estimates of ability.  Given that the majority of struggling beginning teachers either improve or self-select out of the profession, let's estimate that 2/3 of all teachers in their first 5 years are identified as below average teachers.  This would mean that only 1/6 of all teachers in their sixth year and beyond are below average teachers.  And since only 1/3 of teachers are in their sixth year or beyond, this would mean that only 1/18 of all teachers would both have 5+ years of experience and be rated below average.  This is a little under 6% of all teachers.

The average school in my sample had 72 teachers.  So, that's the equivalent of firing four teachers.  And that's under an extremely aggressive scenario.  Besides, now that you've rid your school of the chaff, who, exactly, do you want to fire next year?  And if you want to argue that we could be more aggressive and fire some of the novice teachers, that would mean there'd be fewer low-performing experienced teachers (since teachers tend to be roughly equally effective pre- and post-tenure).  So, for now, let's stick with the assumption that, under an aggressive plan, we'd fire four teachers this year in the average school.

Now, other districts have far more experienced teachers.  And it might make more of a dent there.  But a good number of our poorest-performing schools and districts are quickly churning through teachers too fast for firing low performers to make much of an impact.  Certainly, we should make every effort to rid our schools of the worst teachers (by increasing the performance of, and/or dismissing the lowest performers) -- I don't think anybody seriously disputes that notion.  Or, at the very least, I don't think anybody serious disputes that notion.  But will firing the lowest performing 6% of teachers in high-poverty NYC schools make a difference?  It's possible.  But let's be reasonable -- it's not going to make much of a difference.

So, yes, let's work harder to rid our schools of the worst teachers.  But let's not pretend it will be easy to do.  And, perhaps more importantly, let's not hold our breath while we wait to see if that bullet is actually silver.  In most places, other problems loom far larger.

Evaluating the Evidence on Non-School Interventions

I've been meaning to finish writing this piece for six weeks, and now I finally have.  Enjoy.

One of the most e-mailed articles in the NY Times shortly before Christmas was this piece by Helen Ladd and Edward Fiske on social class and educational achievement, in which the authors call for more non-school interventions ("education policy makers should try to provide poor students with the social support and experiences that middle-class students enjoy as a matter of course"). Overall, I thought it was a pretty good piece, but two things in particular struck me.

1.) That they build an argument for focusing on what happens outside of schools and then their first recommendation is to expand pre-schools.

2.) The recommendations after the pre-school discussion are fairly vague.

While the first is interesting, I'm more intrigued by the second -- and I wonder to what extent it's because they want to recommend that we change 30 things they can't possibly list in the limited space and to what extent it's because they're not sure exactly what to address.

Which begs the question: what do we know about which non-school programs will make a difference?  One particularly promising young scholar has argued that we don't yet know enough (you'll get the joke if you click on the link) to draw many conclusions on the topic.

The authors are certainly right that "Large bodies of research have shown how poor health and nutrition inhibit child development and learning" and they could've included numerous other factors at the family and neighborhood level.  Since we know that these social factors and environmental conditions are causally related to academic performance, trying to ameliorate their impact on low-income children makes all the sense in the world.  But, at the same time, I have yet to find (after extensive searching) a whole lot of evidence that we've been able to successfully do this in ways that rigorous research has found subsequently improved academic performance.  And Russ Whitehurst argues the point even more strongly, writing in a recent report that "There is no compelling evidence that investments in parenting classes, health services, nutritional programs, and community improvement in general have appreciable effects on student achievement in schools in the U.S."

Let's take a look at the few programs they do mention in the piece.  When I search Google Scholar for research on the programs they name, this is about all I can find on the East Durham Children's Initiative, Syracuse's Say Yes to Education program, Omaha's Building Bright Futures, and Boston's Citizen Schools.  Only the last one links the program to any educational outcomes, and it appears to be an internal report.  If there's evidence in peer-reviewed academic journals that these programs have improved students' academic performance, I've yet to see it (note: this is not to say that any of these four aren't working, just that we don't yet have really good evidence that they are).

At this point, some of you may be saying "you forgot about the Harlem Chidren's Zone!".  That's certainly the most-cited example of social policy impacting academics.  But there's a funny thing about that.  As far as I can tell, only one study has linked HCZ to academic outcomes.  And one thing that recently caught me eye is a chapter by Roland Fryer and others in the new Duncan/Murnane book on inequality and schools (highly recommended, btw).  In particular, I find it interesting how they've changed their tune on HCZ the past couple years.

In 2009, Fryer put out an NBER working paper with PhD student Will Dobbie arguing that the HCZ had effectively closed the black-white achievement gap.  The paper got all sorts of play in the press, with David Brooks claiming it proved once and for all that the "no excuses" schools were all that we needed and some of the Broader, Bolder folks replying that, no, it proved once and for all that community resources made the difference.

Shortly thereafter, I asked Geoffrey Canada which it was when he visited Vanderbilt -- he said that we needed both and that it was a "terrible, phony debate" to try and separate them.  Nor could Dobbie and Fryer definitively separate them; in the introduction, they write (emphasis theirs) "We cannot, however, disentangle whether communities coupled with high-quality schools drive our results, or whether the high-quality schools alone are enough to do the trick." (p. 4)

But now they've updated the paper and, according to Fryer's Harvard info page, it's been accepted at the American Economic Journal: Applied Economics. This is from the abstract: "We conclude with evidence that suggests high-quality schools are enough to significantly increase academic achievement among the poor. Community programs appear neither necessary nor sufficient."

This would go nicely with the new book chapter (here's a slightly different version) in which they write, on the first page:

The evaluation of the Harlem Children's Zone allows us to conclude that a high-quality school coupled with community-based interventions does not produce better results than a high-quality school alone, offering further evidence that school investments offer higher social returns than community-based interventions.

That seems like a rather sweeping statement to make based on one preliminary estimate of one program's effects but, nonetheless, their findings do put the burden of proof back on those supporting the Broader, Bolder position.

The closest thing I've seen to a collection research citations indicating that we do have evidence that community-based interventions can work is David Kirp's recent book, but even that involved a good deal of cherry-picking and mostly discussed small programs not explicitly linked with local schools.

So, where does this leave us?  As I wrote above, we have plenty of evidence that a wide range of experiences associated with living in poverty negatively impact kids' academic performance.  And we have plenty of reason to believe that altering these experiences could, potentially, improve kids' academic performance.  But I, and others, would argue that we have precious little empirical evidence that social policy has (or will) alter kids' lives in ways that will subsequently improve their grades, test scores, graduation rate, attainment, etc.  So I find it a bit odd that Ladd and Fiske conclude by writing

But let’s not pretend that family background does not matter and can be overlooked. Let’s agree that we know a lot about how to address the ways in which poverty undermines student learning. Whether we choose to face up to that reality is ultimately a moral question.

I'd make a different pitch if I were they.  I'd write something more along the lines of this: Let's not pretend that family background and living conditions don't matter and can or should be overlooked.  Let's agree that we know a lot about how poverty undermines student learning and how large this impact is.  And let's agree that we urgently need more research on ways to address the links between poverty and education.  The Promise Neighborhoods and other initiatives deserve our full attention and support in the short-run and can potentially provide that will help us better address the problem in the long run.

Of course, twice as many words with half the certainty is a really bad formula for an op-ed.  And there's no quicker way to frustrate policymakers than to write "more research is needed."

But, at the same time, I'm not sure it's helping their cause to claim that we know how to solve the problem.  If I'm in charge of a new Promise Neighborhood, my immediate reaction would be "We do? Great!"  Quickly followed by asking "which factors should I aim to address and which programs do we know are best to address these?"  I don't know the answer to that, and I've yet to hear from anyone who does.

So, in the end, I'd say there's about as much empirical evidence that social policy will close the achievement gap as there is that charter schools, merit pay, and vouchers will close the gap.  That is, very little.  So if we insist on arguing for an either/or approach, this leaves us at a standstill.  Both sides can yell that the other side's evidence is weak.  Which doesn't seem particularly productive to me.

As a researcher, this seems like an excellent argument to conduct a lot more research on the links between social policy and academic performance (as well as on in-school interventions).  Were I a policymaker, I'd want to avoid putting all my eggs in one basket.  We know the status quo doesn't work, but we can't really say for sure what else would be better.  That seems like a golden opportunity for policymakers and researchers to work together and experiment (literally) with a wide variety of reforms -- the former would get to hedge their bets and look prudent and open-minded while the latter would get to conduct groundbreaking research on a crucial issue.

In sum: Do we have conclusive evidence that a particular set of non-school interventions will close the achievement gap?  No, we don't.  So let's not claim we do.  But, let's also vow to keep searching for it.