How Education Research is like Football Research

In the fall, I wrote about an instance in which outsiders may be needed in education reform.  Today I'll give you an example of how outsiders can also be dangerous (though this one pertains more to research).

Perhaps the largest change in educational research over the past decade or so has been the sizable increase in large-scale quantitative research, a fair amount of which is conducted by researchers outside of ed schools.  Like any change, this has resulted in both positives and negatives.  But one thing that worries me is that I consistently notice the people who are most worried about statistical rigor in quantitative analyses (both inside and outside of ed schools) tend to be less concerned with understanding the context and processes of schooling.

And that's incredibly dangerous.

Methodology, statistics, and technical skills are very, very important in the development of good research.  But without a proper understanding of how schools work and what is actually happening on the ground, one can't expect to ask the right questions.  And if one fails to ask the right questions, it really doesn't matter how complex and rigorous their analysis is because the answers to those questions are meaningless.

Here's one example of how such a process can unfold -- it's completely unrelated to ed policy, but I still think it's illustrative.  The Freakonomics Blog posted a brief discussion yesterday of the ending to the Super Bowl.  The post said two things (paraphrasing, of course):

1.) Isn't it amazing that the coaches of both teams realized that the Giants scoring a touchdown with about a minute left was actually a better outcome for the Patriots?  The Patriots' coaches tried to let the Giants run the ball into the endzone while the Giants' coaches instructed their players not to score a TD.  These counter-intuitive behaviors are an excellent example of game theory properly implemented.

2.) But then the Giants failed to take game theory into account when attempting their two point conversion.  Wouldn't it have been much better for them to run time off the clock instead of trying to score to go up 6 points instead of 4?  They might've been able to kill 20 seconds by running the ball 95 yards backwards and around in circles, and certainly being up 4 with 40 seconds left is better than being up 6 with a minute left.  Why didn't the coaches think of this?

There's some clever thinking going on here.  Yes, this is an interesting application of game theory.  And, yes, running 20 seconds off the clock would've been a better strategy.  So the application of economic theory to the situation is exemplary.  In a short space, there's a cogent analysis and a provocative question.  But there are two fatal flaws.

1.) Both coaches did not apply game theory.  Tom Coughlin, the Giants' coach, said he preferred that the team take the guaranteed six points to running down the clock.  So let's hold off on patting him on the back for correctly applying game theory.

2.) More importantly, the clock doesn't run during two-point conversions.  The Giants could've run around in circles for ten minutes, and there still would've been exactly 59 seconds left on the clock.

So, what we have here, is a smart professor who's well-trained in economic theory and statistics.  This training has allowed him to make an important insight about a football game and ask an interesting question.  Except that he doesn't actually seem to know much about the rules of football or the context of the situation.  Which  has rendered his question moot.

And I see the same thing (in a much less dramatic and much less foolish) way happening in education research.  Smart people with training in other fields and disciplines and serious methodological credentials come into the field and find some low-hanging fruit ripe for picking.  At first, this seems like a great idea.  We can never have too many smart, well-trained researchers in education.  And the eye of the outsider can be sharp.  But then the research starts and we realize that somebody can be smart and well-trained but, at the same time, fail to truly understand how schools work and the contexts under which students, teachers, principals, schools, etc. operate.  And then we get smart, well-trained people asking the wrong questions (or interpreting their findings in silly ways).  And that neither advances the field nor helps us improve our educational system.

Let's bring the analogy back to football.  Let's say that Football was a field in many Universities.  Grad students train under faculty who work for Schools of Football and/or Departments of Football Policy, Football Leadership, Football Teaching & Learning, Football Evaluation, Football Foundations, Football Studies, and so on.  And most of the research on football is conducted by faculty and grad students from these schools and departments.  There's no reason why an economist shouldn't do a study on the costs and benefits of attending school on a football scholarship; why a psychologist shouldn't conduct a study on the impact of playing football on one's personality; or a sociologist shouldn't conduct a study of the impact of playing football on one's social capital.  But in order to do these studies well, they first need to understand how the game of football is played, what a player does on the field, how much he practices, and so on.  Otherwise they're just chucking their theories against a wall and hoping one sticks somewhere.

So, to all the smart economists, psychologists, sociologists, etc. out there who wish to conduct the research on education: Welcome, we'd love to hear your insights and figure out if we can apply your theories and methods to help us advance our field and improve our schools.  But before claiming that you've solved a problem none of us have been able to for the last 100 years, take some time to learn how schools operate.  Read a massive and wide-ranging stack of literature.  Go visit some schools.  Talk to people who work in schools and education departments.  Talk to people who study those who work in schools and education departments.  Then begin your research.

At the very least, that should save you the embarrassment of asking students how many touchdowns they need to score in order to hit a home run on their fourth grade reading test.