Why Does Massachusetts Rank Highly?

In last week's debate, Mitt Romney took credit for Massachusetts' position atop some education rankings. And, yes, it's generally true that Massachusetts ranks at or near the top.  More specifically, the state has frequently had the highest average score on the National Assessment of Educational Progress (NAEP).

But the more important question is why Massachusetts ranks so highly.  Was it something that Romney did while Governor, or are there other factors at play?

The second question is really quite easy to answer.  It's almost certainly something other than Romney's actions.  For two reasons:

1.) Children in Massachusetts earned really high test scores both before and after Romney was Governor:

	2002	2011
1.)	Vermont (272)	Massachusetts (275)
2.)	Massachusetts (271)	New Jersey (275)
3.)	Montana (270)	Connecticut (275)
4.)	Nebraska (270)	Vermont (274)
5.)	Maine (270)	Montana (273)

2.) We know from decades of research that non-school factors influence achievement far more than in-school factors.  So it's exceedingly unlikely that a few state-level policy tweaks, implemented for a mere four years, could impact student performance dramatically enough to boost Massachusetts to the top of the nation.

We can also argue to what extent the high test scores mean the state's schools are a success.  We could certainly measure student and school performance in myriad other ways.  And even if we look only at test scores, we can go beyond the averages.  Massachusetts has one of the largest gaps in achievement between upper- and lower-income students, for example.  Though, again, that likely has little to do with Romney -- the state ranked 5th in 2002 and 6th in 2011 (measured as the average 8th grade reading score of those not eligible for free/reduced price lunch minus the average score of eligible students).

That said, I wanted to explore this a little more in-depth, so I went to the NAEP website and delved into the 8th grade reading scores.  The first thing you'll notice on the site is the map of state results (below) which shows striking regional disparities in test scores.  If we assume that Governors are almost solely responsible for the average test scores in their states, we could only conclude that virtually all Northern governors are education geniuses and almost all Southern Governors education dunces.  Which, of course, is preposterous -- there are clearly larger issues at play here (issues out of the hands of the various Governors).

What are these issues?  The socio-economic status of the states' residents would be at the top of the list (certainly, a Governor would have some power to influence that over the course of one or more terms -- but that change would be both slight and slow).  To examine this, I downloaded the state NAEP scores from the NAEP data webpage and demographic data from census website so that I could compare the two.

Unsurprisingly, a fairly strong correlation exists between a state's average 8th grade reading score and a state's median household income (r = .43).  When we plot all the states' average test scores and median household incomes on the graph below, we see a few outliers -- Montana, Kentucky, and Vermont score much higher than we'd expect given their average incomes while California, Hawaii, Nevada, and Alaska score much lower -- but Massachusetts is right about where we'd expect it to be (note that the best fit line would run right through Massachusetts if we deleted the outliers).  Massachusetts has relatively wealthy residents and high-scoring students.  Not a surprise.

Massachusetts stands out even more if we look at the education levels of the population.  The percentage of adults with a bachelor's degree in a state is very strongly correlated (r = .65) with the average NAEP score in that state, and Massachusetts ranks at the top of both categories.  Once again, we see some outliers -- both positive (Kentucky, Wyoming, and Montana stick out) and negative (Mississippi, Louisiana, New Mexico, California, and Hawaii don't look too good), but find Massachusetts right about where we'd expect.

So, yes, Romney was correct when he said that Massachusetts ranks at the top.  But it's exceedingly unlikely he had much to do with that.  Massachusetts' residents were and are wealthy and highly educated relative to the residents of other states, and that mostly explains why their children perform so well on tests.


cross-posted on Blog of the Century

It's All About Vocabulary?

The edusphere is abuzz about this NY Times piece on early vocabulary growth that ran over the weekend. Though the piece focuses on the current controversy surrounding test-based admissions to the top high schools in NYC, it's mostly based on the famous Hart and Risley book in which the authors conclude that children from families on welfare hear 32 million fewer words and 560,000 fewer encouragements than children of professional families between birth and age 4 -- and that these differences lead to subsequent differences in vocabulary and achievement.

To reinforce the importance of this early vocabulary growth, the article quotes a charter school principal saying that the "word deficit" is the greatest challenge the school faces and quotes E.D. Hirsch saying that "there is strong evidence that increasing the general knowledge and vocabulary of a child before age six is the single highest correlate with later success".

This all leads Robert Pondiscio to write that "Demography isn't destiny. Vocabulary is destiny".

Ok, stop.  Just stop.

Yes.  Vocabulary is important.  And vocabulary growth in the early years is crucially important.  Every person or organization responsible for raising young kids should aim to use and explain as many words words and concepts as possible.

But, c'mon.  Let's not get carried away.

The second we identify something -- anything -- as the "single" most important, we do ourselves and our nation's children a disservice.  I understand the allure of boiling everything down to the simplest solution possible, but life just doesn't work that way.

And arguing that vocabulary -- rather than demography -- is destiny?  That's just silly.

For starters, we have what economists would call an "endogeneity problem" in that statement.  An awful lot of what's driving the vocabulary of a child entering kindergarten is also driving the success of that kid later in school: parenting, myriad environmental conditions and social factors in the child's home and neighborhood, health, peers, genetics, and a thousand other things.  In other words: a child with a large vocabulary at age 4 is likely to succeed in school partly b/c of that vocabulary, but more so because the conditions that created that vocabulary will almost certainly continue to foster intellectual growth and social development throughout his/her school years.

Second, an awful lot of what drives vocabulary growth is demography.  The education level of one's parent(s) and other adult supervisors, the amount of stimulation available in one's surroundings (including the number of different objects one can learn about), the noise level inside and outside of one's home, the levels of stress to which a child and his/her family are exposed, the number of books available, and hundreds of other home, neighborhood, and family factors correlated with socio-economic status all result in a child learning more or fewer words.  So arguing that vocabulary is more important than demography in school is like arguing that strength is more important than weightlifting in football.

So, please, let's stop trying to reduce everything to the one, most important factor (which is surely more important than the factor the last person discussed).  The fewer things we focus on, the more distorted those measures become.  And the simpler we make the problem seem, the more simplistic our solutions.  Vocabulary certainly deserves some of our attention.  Now let's discuss what else deserves our attention instead of how much less they deserve it.

The Differences Between High- and Low-Performing Schools: Not What You Might Expect

Last Thursday, the National Center on Scaling Up Effective Schools (a research center led by Vanderbilt's Peabody College with partners at UNC-Chapel Hill, Florida State, Wisconsin-Madison, Georgia State, and the Educational Development Center, funded by a five-year, $13.6 million federal grant, which aims to identify and then explore ways to scale up, characteristics of effective high schools) released a new report examining the differences between high-and low-performing high schools in Broward County, Florida.

As only one small piece of the puzzle, we shouldn't get carried away with the findings.  But I was struck with what was -- and was not -- included in their list of differences between the schools.  Below is the Executive Summary's list of differences:

We identified one major theme that cut across all ten components: personalization for academic and social learning. In the area of personalization, our findings show that the higher value-added (VA) schools made deliberate efforts through systematic structures to promote strong relationships between adults and students as well as to personalize the learning experience of students. In addition, the higher VA schools maintained strong and reliable disciplinary systems that, in turn, engendered feelings of caring and, implicitly, trust among both students and teachers. Leaders at the higher VA schools talked explicitly about looking for student engagement in classroom walkthroughs as well as in their interactions with students. Teachers at the higher VA schools were more likely to discuss instructional activities that drew on students’ experiences and interests. The higher VA schools also encouraged stronger linkages with parents (p. 5).

Included: "soft" factors like trust and relationships.

Not included: virtually everything currently discussed in ed policy circles (school choice, teacher evaluations, merit pay, data-driven decision-making,  etc.)

Now, to be fair, many of factors were off the table because the study examined four schools located in the same county which had much in common (no differences in merit pay or district leadership, for example).  And there's always the possibility that implementing some of these reforms could change the factors included in the list even if they're not currently present in the schools.

Nonetheless, even when the measuring stick is value-added scores -- the latest, greatest measure being pushed on schools -- many of the factors emphasized by those pushing for its use don't seem to be drivers of the differences.

Most interestingly, the two low-scoring schools had higher scores on some measures of teaching practices and instructional quality than the two high-scoring schools.  Here's the summary from the research team on this topic:

Taken together, our indicators of the quality and nature of instruction across the schools -- CLASS-S*, course matrices, student shadowing, and interviews with multiple school stakeholders -- reveals no major differences in instructional quality across the four schools.  We cannot turn to evidence in the area of Quality Instruction to explain the differences in value-added achievement between our high- and low-VA schools" (p. 32).

While we certainly shouldn't base our policy decisions on one study examining four schools in one county, I do think it's fair to say that this confirms what we should've known all along -- that high-quality instruction (like every aspect of schools) is not sufficient when trying to create high-performing schools.  I should also note that, in many areas, larger differences existed between the honors and regular tracks within the schools than between the high- and low-scoring schools themselves.

Again, I don't want to get carried away with the results of one small-scale study (and I'll refrain from addressing the other 50 or so topics covered by the report at the moment), but I do think that, at this point, we can take two important lessons from this ongoing research:

1.) Regardless of the amount of press coverage, foundation money, or policy directed toward a particular aspect of school reform, not a single factor is sufficient to create a high-quality school.

2.) Even though it's easy (and, arguably, practical) to focus on the simplest, starkest issues, the most subtle, nuanced, and complicated ones are often at least as important.

From a 10,000 foot vantage point, the potential benefits of creating more charter schools, or implementing a merit pay plan or new curriculum are easy to see.  But, on the ground, it probably matters more how than whether those things are implemented -- without strong relationships, trust, and commitment, it's unlikely any reform will turn around a school or district.

That fact is really difficult for policymakers to swallow because there's no easy way to change those types of things: what is Congress supposed to do in order to make make teachers at the local elementary school get along better with their students?  The relationship between policy and the factors discussed in the report is so indirect that it's easy to just ignore them and focus on simpler solutions.  We should all try to resist that temptation.



*"the Classroom Assessment Scoring System for Secondary classrooms (CLASS-S), [is] an observational tool developed by researchers at the University of Virginia, to observe and assess the quality of teacher-student interactions in classrooms. Based on development theory and research suggesting that interactions between students and adults are the primary mechanism of student development and learning (Greenberg, Domitrovich, & Bumbarger, 2001; Hamre & Pianta, 2006; Morrison & Connor, 2002; Pianta, 2006; Rutter & Maughan, 2002), the CLASS-S focuses not on the presence of materials, the physical environment, or the adoption of a specific curriculum but on what teachers do with the materials they have and on the interactions teachers have with their students. The observation tool looks specifically at interactions between teachers and students across four domains: Emotional Support, Classroom Organization, Instructional Support, and Student Engagement" (p. 12).

Schools Rallying Communities

I have strongly mixed feelings about charter schools, but my biggest concern is one I almost never see mentioned by charter proponents, detractors, or neutral observers.  We hear a lot about how communities affect schools, but almost nothing about the reverse.

I grew up in a suburban district where people routinely headed to the local high school for football games, basketball games, school plays, and scads of other events.  And, to a lesser extent, the elementary and junior high schools brought in community members for fairs, concerts, etc.  All in all, the schools brought the community together quite often for various reasons.  And that's not uncommon.  Or at least, historically, it hasn't been uncommon.

But that might be changing.  If we imagine a world where schools and neighborhoods are completely decoupled and people from one town go to scads of different schools all over the place, that relationship almost ceases to exist.  We won't read stories like this piece in the NY Times about a small-town HS football team that's rallying the community.

Granted, it might be worth the trade-off if the new non-neighborhood schools dramatically outperformed our traditional school system, but it's important to recognize that there is a trade-off involved here.  And that schools have larger ripple effects on society beyond the academic performance of their current students.

Frustrating Work Conditions and Schools

It seems like a growing number of people give lip service to working conditions in school, but without many specifics.  If teachers are frustrated by the working conditions in their schools, how would we expect their behavior to change?

We're using Bolman and Deal's textbook in the Organizational Theory course I'm teaching this semester, which includes a large section on human resources in organizations.  Part of that section discusses Chris Argyris' work on the differences between human personality and management practices.  Argyris contends that workers have six options when trying to escape frustrating working conditions (pp. 128-130).  See how many of these seem familiar to you when thinking about schools:

They withdraw -- through chronic absenteeism or simply by quitting
This certainly happened at my school -- working conditions were so bad that the vast majority of teachers took all 10 of their sick/personal days each year (which compounded the problem, since we usually couldn't find any subs to come into the building).  I'm not sure what's been published on the topic, but I do know that if one looks through the NYC School Report cards that a lot of schools average a lot fewer teacher absences.

They stay on the job but withdraw psychologically, becoming indifferent, passive, and apathetic
This is the quintessential "bad teacher" right here.  The tenured burn-out who can't be bothered to do much of anything anymore.

They resist by restricting output, deception, featherbedding*, or sabotage
Sounds just like the legion of obstinate teachers who refuse to implement the latest, greatest curriculum or other reform handed down to them from above.

They try to climb the hierarchy to better jobs
As teachers in my school used to say: "those who can, do; those who can't, teach; those who can't teach, become principals" (I should note that there's some emerging evidence that many principals had above-average VAM scores when they were teaching).  Either way, it's pretty clear that a lot of teachers try to escape the classroom to become coaches, coordinators, and administrators of all types.  In my school, the most veteran teachers who hadn't moved into one of those types of positions all taught in positions that got them out of the classroom (e.g. "resource room," in which they'd pull out a couple kids at a time).

They form alliances (such as labor unions) to redress the power imbalance
Unions certainly play a large role in many schools.  What we often forget, though, is why the unions came about.  If teachers aren't frustrated and don't distrust their supervisors, they don't usually form (or utilize) unions.

They teach their children to believe that work is unrewarding and hopes for advancement are slim
I haven't seen any evidence of this happening with teachers . . . hopefully it doesn't get that bad.


I definitely see evidence of five out of these six behaviors, though it's unclear whether any of these are currently increasing.  I'd argue, though, that ameliorating the conditions that lead to these types of behavior should be one important goal in our quest to raise teacher quality and turn around low-performing schools.

If we instead go the opposite direction (sterner management, scripted curricula, etc.), we risk turning our schools into highly organized, poorly performing factories.  Taken to the extreme, teachers essentially become mindless drones.  The authors quote Ben Hamper (a former factory worker who then wrote about his experiences) saying that "Working the Rivet Line was like being paid to flunk high school the rest of your life" (p. 131).  Work like that certainly won't inspire anybody to become the high-quality teachers we all agree we need.


*"Featherbedding is a colloquial term for giving people jobs that involve little or no work" (p. 138).