Tests Aren't Omniscient

I like David Brooks, but sometimes he goes off the reservation when he writes about education. Today's column isn't too bad -- it's worth a read.

He makes a number of points, but I found his remarks surrounding measurement of teacher quality most notable -- two sections in particular:

Most important, it would increase merit pay for good teachers (the ones who develop emotional bonds with students) and dismiss bad teachers (the ones who treat students like cattle to be processed).

Developing bonds with students should be encouraged; treating students like cattle should not. But Brooks mostly discusses the use of standardized test scores and I'm unaware of any in use that measure this. If he's talking about making relationships with students a criteria in evaluating teachers, then I'm with him -- but if he thinks that this is what state tests are measuring then he needs to give this a little more thought.

Today, tests can tell you which students are on track and which aren’t. They can tell you which teachers are bringing their students’ achievement up by two grades in a single year and which are bringing their students’ levels up by only half a grade. They can tell you which education schools produce good teachers and which do not.

New York City Schools Chancellor Joel Klein has data showing that progress on tests between the third and eighth grades powerfully predicts high school graduation rates years later — a clear demonstration of the importance of these assessments.

The first paragraph represents a problem I have with a lot of proponents of merit pay and accountability -- it oversells what we can do with tests. Brooks says that tests can tell us these things. At this present moment, using the tests we currently use, they can do nothing of the sort. What they can do is inform us -- using the information we can make reasonable guesses, but teachers' results one year are only weakly to moderately correlated with their results the next year. In other words, if we base our evaluation of teachers on one year of test data then we're going to be wrong an awful lot of the time (not to mention the approximately 2/3 of teachers who don't teach a subject that's tested by a state test). I think there are a lot of solid arguments for better and more widespread use of data, particularly in evaluating teachers and schools, but paragraphs such as the one above aren't really truthful and don't do anybody any favors. Remember: overstating your case doesn't make it stronger.

As for the second paragraph, I hardly think it's a surprise that a child's growth in test scores over the course of six years of schooling is correlated with whether or not they graduate from school. All that means is that the tests aren't completely useless -- it's not any sort of demonstration of their omniscience.

By all means, let's find better ways to use the data we collect -- but let's start with a realistic understanding of what we can and cannot know from this usage.