Why Does Massachusetts Rank Highly?

In last week's debate, Mitt Romney took credit for Massachusetts' position atop some education rankings. And, yes, it's generally true that Massachusetts ranks at or near the top.  More specifically, the state has frequently had the highest average score on the National Assessment of Educational Progress (NAEP).

But the more important question is why Massachusetts ranks so highly.  Was it something that Romney did while Governor, or are there other factors at play?

The second question is really quite easy to answer.  It's almost certainly something other than Romney's actions.  For two reasons:

1.) Children in Massachusetts earned really high test scores both before and after Romney was Governor:

	2002	2011
1.)	Vermont (272)	Massachusetts (275)
2.)	Massachusetts (271)	New Jersey (275)
3.)	Montana (270)	Connecticut (275)
4.)	Nebraska (270)	Vermont (274)
5.)	Maine (270)	Montana (273)

2.) We know from decades of research that non-school factors influence achievement far more than in-school factors.  So it's exceedingly unlikely that a few state-level policy tweaks, implemented for a mere four years, could impact student performance dramatically enough to boost Massachusetts to the top of the nation.

We can also argue to what extent the high test scores mean the state's schools are a success.  We could certainly measure student and school performance in myriad other ways.  And even if we look only at test scores, we can go beyond the averages.  Massachusetts has one of the largest gaps in achievement between upper- and lower-income students, for example.  Though, again, that likely has little to do with Romney -- the state ranked 5th in 2002 and 6th in 2011 (measured as the average 8th grade reading score of those not eligible for free/reduced price lunch minus the average score of eligible students).

That said, I wanted to explore this a little more in-depth, so I went to the NAEP website and delved into the 8th grade reading scores.  The first thing you'll notice on the site is the map of state results (below) which shows striking regional disparities in test scores.  If we assume that Governors are almost solely responsible for the average test scores in their states, we could only conclude that virtually all Northern governors are education geniuses and almost all Southern Governors education dunces.  Which, of course, is preposterous -- there are clearly larger issues at play here (issues out of the hands of the various Governors).

What are these issues?  The socio-economic status of the states' residents would be at the top of the list (certainly, a Governor would have some power to influence that over the course of one or more terms -- but that change would be both slight and slow).  To examine this, I downloaded the state NAEP scores from the NAEP data webpage and demographic data from census website so that I could compare the two.

Unsurprisingly, a fairly strong correlation exists between a state's average 8th grade reading score and a state's median household income (r = .43).  When we plot all the states' average test scores and median household incomes on the graph below, we see a few outliers -- Montana, Kentucky, and Vermont score much higher than we'd expect given their average incomes while California, Hawaii, Nevada, and Alaska score much lower -- but Massachusetts is right about where we'd expect it to be (note that the best fit line would run right through Massachusetts if we deleted the outliers).  Massachusetts has relatively wealthy residents and high-scoring students.  Not a surprise.

Massachusetts stands out even more if we look at the education levels of the population.  The percentage of adults with a bachelor's degree in a state is very strongly correlated (r = .65) with the average NAEP score in that state, and Massachusetts ranks at the top of both categories.  Once again, we see some outliers -- both positive (Kentucky, Wyoming, and Montana stick out) and negative (Mississippi, Louisiana, New Mexico, California, and Hawaii don't look too good), but find Massachusetts right about where we'd expect.

So, yes, Romney was correct when he said that Massachusetts ranks at the top.  But it's exceedingly unlikely he had much to do with that.  Massachusetts' residents were and are wealthy and highly educated relative to the residents of other states, and that mostly explains why their children perform so well on tests.


cross-posted on Blog of the Century