Last year around this time a new study released by the Urban Institute looking at the effectiveness of Teach for America teachers was all the rage. In the paper, the researchers found that high school TFA teachers in NC were slightly more effective than other teachers -- a finding that has been trumpeted who knows how many times since.
I pointed out at the the time that the study, though well executed, was far from perfect. Well, the researchers recently released an updated version of the study -- something that, surprisingly, I've only seen Debra Viadero blog about (hat tip: GothamSchools).
If I have time, I might do a full evaluation at a later date. But until then, here are a few things to keep in mind when you read about the results:
-It's only high school teachers and it's only in NC. Who knows how it would play out if it were elementary or middle school students in other states.
-In the data, there was no absolutely conclusive way to determine which students were assigned to which teachers. Usually, the teacher that proctored the test was also the teacher for the students, but not always. Using some sophisticated methods, the researchers decided they were pretty sure which students belonged to 84% of the teachers and moved on from there. When analyzing a smaller sample of teachers about whom they were more sure, the results differed slightly -- indicating that figuring out which students had which teacher is somewhat problematic for their estimates.
-The sample consists of 98 teachers from 23 school districts observed 150 times over the course of 7 years. Which means that the mean number of teachers analyzed per district in a given year was 0.93.
-The return to TFA teachers in most models is about .1-.18 standard deviations -- a very modest effect size. The authors argue that this is significant because the return to a teacher with 3-5 years of experience compared to one with 0-3 years of experience is about .05 SD -- in other words, they argue thata TFA teacher is many times better than an experienced teacher.
-But my main worry about the study is that they have too large a sample in the comparison group (as far as I can tell it's every teacher from those 23 districts that they could match to students). I maintain that not every teacher does the same job -- teaching an AP class in a wealthy school is very different from teaching a low-track class in a high-poverty school. In the latest version of the paper the authors sort of make an attempt to look at this and compare TFA teachers to other teachers teaching students with similar test scores. TFA teachers who taught students in the top quartile did quite well. But those teaching students in the bottom quartile (which, I assume, is most of them (they don't provide a number for this)) did not. When narrowing the sample to teachers who taught kids in the bottom quartile, TFA teachers had an advantage of .061 standard deviations -- virtually identical to the advantage of .054 SD that teachers with 3-5 years of experience had. Given that the main goal of TFA is to help the neediest, I find this troubling.
2 comments:
interesting thoughts. my opinion on this whole alternative teacher certification debate:
if alternatively prepared teachers are doing no worse than regular teachers, it makes no sense to bar them from entering the profession. even if the effect sizes are small (but positive!), opening up teaching (particularly given the shortages in hard to staff schools and subjects) to a wider pool of people will only do good things for the children.
I wasn't taking a position on the alt cert debate, I was simply pointing out a few of the limitations of the paper -- which a lot of people seem to ignore.
Indeed, it seems pretty clear that TFA teachers are, at the very least, doing no worse than would teachers the district might hire or assign were they not there -- though this paper only made a half-hearted attempt at measuring that as well.
Post a Comment