The study involved 296 middle school math teachers in Nashville who were assigned to either a treatment group (eligible for bonuses of 5, 10, or $15,000) or a control group and then tracked for three years.
The main result was that students assigned to bonus-eligible teachers did not perform any better than students assigned to treatment group teachers. The lone exception were 5th grade students in years 2 and 3 of the study, but the gains did not persist through the end of 6th grade. The main portion of the executive summary reads as follows:
POINT was focused on the notion that a significant problem in American education is the absence of appropriate incentives, and that correcting the incentive structure would, in and of itself, constitute an effective intervention that improved student outcomes.
By and large, results did not confirm this hypothesis. While the general trend in middle school mathematics performance was upward over the period of the project, students of teachers randomly assigned to the treatment group (eligible for bonuses) did not outperform students whose teachers were assigned to the control group (not eligible for bonuses).
Prior to implementation, researchers calculated that 55% of teachers would be able to obtain bonuses if their students answered only an additional 2-3 questions correct on the state test. Across the three years of the study, 33.6% of bonus-eligible teachers earned a bonus in one or more years.
This means that most teachers did not dramatically improve, even for one year. It also means that the tendency of test scores to bounce around significantly did not result in different random groups of teachers receiving a bonus each year. About two-thirds of teachers in the treatment group never received a bonus. And about two-thirds of the teachers who received a bonus earned one in multiple years.
Of potentially more import are the results from the teacher interviews and surveys. These will continue to be analyzed in the coming months, but right now the main takeaway point is that teachers in the treatment group really didn't report changing much at all. About 80% of teachers reported that they were already working as hard as they could before the incentives were implemented and were therefore unable to work any harder afterward.
There are two potential bright spots, however, for merit pay proponents:
1.) The project ran smoothly (e.g. the right teachers received the right bonuses at the right time) and didn't suffer any major backlash. That this was truly a partnership between union, university, district, and other groups probably helped in this regard.
2.) It's unclear right now, but the bonuses may have had a small effect on the patterns of teacher attrition for those in the bonus group. 27 teachers in the control group left to teach in another district while only 15 teachers in the treatment group did. The numbers are small, so conclusions are hard to draw, but the most popular criticism of the study by performance pay advocates seems to be that it didn't shed any light on how performance pay might affect teacher recruitment and retention.
There was also no evidence that treatment group teachers were successful in gaming the system by having more kids (or lower performing kids) removed from their class, preventing new students from transferring into their classes, focusing more efforts on math instruction at the expense of other subjects, or helping their students cheat on the state tests.
What does it mean?
I wrote before that this was one of the most important studies this decade. That said, it's still just one study. And we have to be careful about how we extrapolate from the results of any one study. The main question the study was set up to answer was whether offering performance incentives to individual teachers would result in their students performing better on standardized tests. The study offered no evidence that this was the case, and little reason to believe that this would be the case for similarly designed incentive systems.
Why is this so? The main reason seems to be the lack of any major changes by teachers. But there are a couple of other possibilities. Just because teachers reported that they didn't change anything doesn't necessarily mean that they didn't change anything. So it's not out of the realm of possibility that at least some of the teachers made changes but that these changes didn't yield subsequent results. It's also possible that, despite the fact that teachers received the project fairly well, they wanted to prove the working hypothesis wrong -- that is, that teachers eligible for a bonus, at least on some level, resented the fact that somebody thought they'd teach better if they were offered a carrot, and decided to not work harder despite the carrot looking awfully delicious. At the same time, the control group teachers could've worked a little harder to prove that they were plenty motivated to teach solely because they wanted their students to succeed. While the answer is likely a little of all of the above, I tend to think the most likely scenario is that teachers simply weren't all that much more motivated by the prospect of a bonus fairly far down the road (teachers were paid the following November).
What it doesn't mean is that all merit pay schemes forevermore are doomed to abject failure. We don't know if different types of bonuses awarded in different ways (shorter time spans, group awards, non-monetary awards, etc.) might have a larger effect, and we know little about how performance pay affects the long-term make-up of the teacher labor force. At the same time, it does call into serious question the application of the overly simplistic homo economicus model used by economists. My gut feeling is that economists tend to view teacher motivation and the teaching profession in an overly simplistic manner guided too much by basic economic theories and not enough by the literature on the sociology of teaching or the psychology of motivation.
Where do we go from here?
NCPI has another study utilizing team-based incentives that should be out at some point over the next couple of years. In addition, other non-randomized studies of the myriad incentive systems that have sprouted all over the country the past couple years are certainly underway as well. I'm not sure why the NCPI researchers chose to claim that this was "the first scientific study of performance pay ever conducted in the United States," since the definition of "scientific" is hotly debated, but the literature base will certainly continue to grow in the coming years regardless.
In addition to the continuing analysis of the data from this study, there are plenty of opportunities for other researchers and funding agencies to examine questions surrounding the impact of merit pay on the teacher labor force, on school culture, on student outcomes other than test scores, and many other areas.
In the meantime, it's important that merit pay opponents not claim that this study proves once and for all that merit pay does not, and will not ever, work in schools. If nothing else, it appears likely that better teachers got paid more money than worse teachers -- which is arguably an improvement on the current system. And, at the same time, it's important that merit pay proponents not claim that this study is meaningless and that we should recklessly proceed with merit pay in schools at breakneck speed.
Merit pay has grown considerably more popular among teachers over the past decade, so it's eminently possible that districts and unions can work together to design and implement better, more nuanced merit pay systems that might have a better chance of success. I don't think anybody would argue that the status quo is the perfect system, so simply saying that merit pay won't work isn't a solution. But these results indicate that we should proceed with caution. The assumption that teachers will work harder for financial incentives is now a dangerous one to make and should be made with caution. As such, the performance pay systems that will undoubtedly continue to emerge should begin with a more nuanced and informed understanding of the practices and motivations of current and prospective teachers. We can only hope that an informed and open-minded approach from both sides will eventually result in compensation systems that attract and reward good teachers in ways that current teachers find meaningful and fair.
For more information, see my previous posts on the experiment:
Part 1: Background Info
Part 2: What to Look For
Part 3: Why it Matters
Part 4: What We Can Learn
Live-Blog of Results
No comments:
Post a Comment