Sunday Commentary: Of What Utility is Generalizing?

When judging research papers, one major criteria is generalizability -- the degree to which we can generalize the findings to other schools. Every now and then this strikes me as odd; throughout our lives we are told not to make generalizations, but researchers strive to make the largest generalizations possible.

When taken to the extreme, we see why this may be important -- a thousand separate research studies on each individual school, each emphasizing its uniqueness, would hardly advance our understanding of schools. But I fear we often take it too far. The size of one's sample and the degree to which findings can be generalized often seem more important than the quality of the research when determining what papers are published or publicized. And I question of what utility such generalizations are in the first place.

As one small example, let us examine the issue of teacher attrition. We can ask the following three basic questions:

1.) How many teachers quit each year?
3.) Why are they leaving?
3.) How is this affecting schools?

Were one to answer these questions based on a nationally representative sample of teachers and schools, we would assume that we could generalize to all schools in the country. But it simply would not be true. Teacher attrition varies widely across the country. Teachers leave at different rates, and for different reasons, across different regions, different grade levels, and different subjects just to name a few. In other words, we cannot possibly answer the three questions above for all schools at the same time.

Many argue that the idea that we have a "teacher shortage" is absurd -- across the country we have more teachers than positions. And we can cite anecdotal evidence in support of this notion -- numerous schools receive hundreds of applications for each job opening, for example. Depending on which study you read, teacher turnover averages about 11% per year -- not too different from many similar fields. Anybody making this argument stands on firm ground -- generally speaking, teacher retention is not the biggest problem in America's schools.

But we should all know to use caution when making generalizations. Teacher retention might not be the largest problems in most schools, but many differ quite dramatically from national averages. Take high-poverty urban schools -- teacher turnover averages around 22% per year. In New York City middle schools, close to 50% of teachers are in their first or second year at that particular school. And teachers in these schools leave for different reasons. In one study of teachers in Washington (state), those in high-poverty (>50% free/reduced price lunch) schools were more than five times as likely (53% to 10%) to report that discipline issues were a reason to leave their schools than those in low-poverty (below 20% free/reduced price lunch) schools. And we can imagine that teachers leaving at different rates and for different reasons would affect schools differently. Indeed, in the survey I referenced on Friday, 48% of teachers who reported teaching in urban schools answered that teacher retention is a serious problem in their school -- 19% of suburban teachers said the same.

In other words, generalizing about teacher retention is of limited utility. Saying "we have a teacher shortage" is just as incorrect as saying "there is no teacher shortage." The same can thing be said about trying to answer the three questions I posed earlier -- the answer to each question depends on what school one is examining. Any answer that does not begin with "it depends" is, to some degree, disingenuous. Why does this matter? Beyond nitpicking over the value and precision of particular studies, it has important ramifications for the education system as a whole. Once we answer the three questions above we would then follow up with two more:

1.) Is this number too high or too low?
2.) What (if anything) should we do about this?

And this is where generalizing becomes dangerous. If we take nationwide statistics at face value, we would conclude that serious interventions are not needed. And quite a few schools would potentially suffer as a result. Every study, of course, does not try to generalize to the entire country. If there is one area where most academics excel it is nuance. Nonetheless, I fear we put generalizations on a pedestal and that this is hurting both our research and our policies. We need to generalize in order to examine broader trends and propose broader solutions, but at some point we go too far. At some point generalizing should no longer be a goal nor should it be rewarded.

Before reacting to or publishing any study on education, we need to ask ourselves to what extent we should generalize rather to what extent we can. If generalizing to more schools is of little utility -- if it results in nothing more than less accurate findings -- then it needs to be avoided.

General statements about schools make life easier, but are oftentimes inaccurate and lead us in the wrong direction. Different schools have different problems, and they need different solutions. As such, we should greet any blanket statistics -- and subsequent solutions -- with skepticism.