Guest Post: Teacher Opposition to VAM

by Kerri Tobin, PhD

Teacher opposition to value-added modeling gets portrayed in the media as teachers refusing to take any kind of responsibility for student learning or academic growth. But most teachers do not balk at the idea that they should be accountable for advancing their students’ knowledge. What they oppose is value-added modeling, or VAM, the highly-imprecise tool that is being used to measure teachers' impact on students' learning.. When the New York Times won the right to publishteacher VAM scores this spring, only the tiny print noted the inaccuracy of the data, including some standard errors larger than the purported effect sizes. Researchers familiar with value-added modeling have repeatedly voiced their concern about its use in high-stakes decisions like teacher evaluations (for example: here, here, and here).

Often, opponents of current uses of standardized testing fall back on attempts to draw parallels between teaching and medicine: “It’s like grading doctors on how many of their patients die!” But these analogies are imprecise; they do not do the problem justice. John Ewing’s fertilizer analogy is fascinating but perhaps too long for the common person. What we need is an “elevator speech” – one that can be delivered in the time it takes to get from the ground to the 3rd floor – to explain why teachers oppose VAM. A better analogy is this: using VAM to evaluate teachers is akin to evaluating chemotherapists based on how much their patients’ tumors shrink every year. The expected rate of tumor shrinkage is calculated based on patients’ race and socioeconomic status. So if a patient’s tumor shrinks more, over that year, than expected, the doctor gets a positive score. If it shrinks less than it “should,” the doctor gets a negative score. The average of these patients’ scores becomes the doctor’s overall VAM score. Sound reasonable? Maybe, until you consider that a) different kinds of tumors respond differently to chemotherapy; b) the doctor has no control over what patients do outside the office (for example, lung cancer sufferers who continue to smoke); c) his patients saw a different chemotherapist last year and will see another one next year; and, perhaps most importantly, d) the doctor is not allowed to treat any co-morbid conditions (for example, a cancer patient with diabetes gets no treatment to manage his blood sugar) – even if he wanted to, and even in cases where the patient or his family might prefer that other conditions be treated instead of the cancer (e.g., when parents value social or self-confidence issues more than test scores), there simply aren’t enough hours in his day. Factors like the overall health of the patient, his lifestyle, eating habits, substance use, weight, and blood pressure might impact the effectiveness of the chemo. But the doctor cannot control these, in much the same way teachers cannot control where students live, if they have enough to eat and get regular medical care, whether anyone reads to them at home, how much TV they watch, or what time they go to bed at night. And in the same way our VAM-assessed doctor would be powerless to decide that a patient dying of AIDS needs antiretroviral therapy before he can tolerate chemo for a concurrent cancer, teachers have neither the time, resources, nor training to solve the problems in their students’ lives – emotional problems, health challenges, family issues, etc. – that impact their academic growth. This is how VAM works, and why teachers oppose it.

-------

Kerri Tobin is Assistant Professor of Education at Marywood University in Scranton, PA, where she researches the educational needs of students living in poverty and prepares teachers and school leaders to meet those needs.

The Two Types of TFA Alums

Almost ten years ago now (where has the time gone?), I figured out what I wanted to do after college: I wanted to join Teach For America.  Eight years ago, I began teaching in The Bronx through a sister program, the NYC Teaching Fellows (one of many organizations under the umbrella of The New Teacher Project, started by a TFA alum -- the interviews for TFA and the three TNTP programs I applied to were almost exactly identical).  While in NYC, many of my closest friends were NYCTF or TFA members and I left for grad school with a rosy picture in my mind of how we would change the world.

I think it's precisely because these types of programs are so near and dear to my heart that I've grown so frustrated with TFA.  For over 20 years now, they've done a world of good in countless ways.  But I've always believed that their greatest impact would be the actions of their alumni.  The recruitment materials I pored over my junior year in college told us that we could see how the system was failing from the inside and then go out and fix it -- whether that be by remaining in the field of education or as a lawyer, politician, school board member, concerned citizen, or whatever other route we chose.  The idea was that, over time, an army of active citizens with elite credentials and experience in our failing inner-city and rural schools would wield enough influence to finally fix what ails our educational system.

That's starting to come true.  Countless TFA alums have become principals, begun charter schools, and so on.  One became the head of the whole DC school system.  And now we're beginning to see TFA alums entering politics (my local school board race in Nashville, for example, featured two TFA alums both vying to unseat the chair of the board).

For most of the past decade I've waited for this day with bated breath.  Finally, I thought, we'd start to see some change.  The conditions that I couldn't believe our society would tolerate while I was teaching would finally start to be addressed.  But now I'm worried.

While I don't doubt that TFA alums will have an outsized influence on our educational policy in the coming decades, I'm no longer convinced that the results of this influence will be all good.

Why?  Because too many TFA alums took the wrong lesson from their experiences in the classroom.  I've interacted with (both personally and professionally), heard, and read countless TFA alums over the past decade, and I now generally lump them into one of two groups: the humbled and the hubristic.  It's a crass generalization, and many alums I know don't neatly fall into one group or the other, but I still think it's helpful in thinking through what changes we should expect as TFAers gain clout.  So, without further ado, here are the two types of TFA alums:

1.) The Humbled - If I chose one word to describe my classroom experience, it would be "humbling" ("frustrating" would be a close second).  I went in believing that I could change the world in one fell swoop, that I would surely be the world's greatest teacher, and that we could easily fix most of our problems if only we could find more miracle workers like myself.  By day two I realized that I wasn't the world's greatest teacher on that day.  And by day five I started to think it might not happen for at least a couple more weeks.  What followed was two years in which I valiantly fought losing battle after losing battle until I was utterly exhausted.  During those two years, I saw the underbelly of one of the lowest performing middle schools in NYC (its closure was announced the spring of my second year), and formed quite a few opinions regarding its failure.

But one notable item missing from my list was the quality of the veteran teachers in the building.  I knew I was smarter than some of them.  I knew I worked harder than all of them.  But I'll be damned if most didn't teach circles around me -- and many found a way to do it for decades while I lasted all of two years before I became a statistic.

My major takeaway from that experience was that fixing the problems that look so simple from the outside is really hard.

And I know a lot of TFA and TNTP alums who will tell you something similar.  Some are disillusioned.  Some are frustrated.  Some are neither.  But all came to realize that they can't do this on their own, and that it's not going to be easy.

This group of humbled alums are more likely to push for educational and societal reforms and policies that change the context in which schools operate.  They know that if you can address poverty at the family and neighborhood level, then school will go a lot smoother; that if you can change the attitudes and outlooks of students they'll learn a heck of a lot more regardless of the teacher; that if teachers are treated as professionals they'll rise to the occasion; that if teachers are given resources and support they'll both stick around longer and teach more effectively while they're there; and that schools, in general, need our help.

2.) The Hubristic - Many TFAers had very different experiences than I.  Research on TFA generally finds that their teachers' students make gains equivalent to or slightly higher than other teachers (sometimes even higher than veteran teachers).  Which means that a lot of corps members receive results each year telling them that they are the world's greatest teacher (or at least one of the best in their school).  Some of these are flat-out better teachers than I, some simply found the right fit, landed a position at a top-performing school, or received oodles of support.

Some, though not all, of these teachers see their school -- and its veteran teachers -- very differently from how I saw mine (or, in the case of TFA members who teach at high-flying charters, see other schools and their veteran teachers differently).  They think to themselves "this isn't so hard; if we could find more people like me, we could lick this problem in no time".  I hear stories from them of how mind blowingly lazy, stupid, and/or incompetent the veteran teachers at their school were.  And they leave their school with a distinct sense that they could fix our schools if somebody would let them -- and if everybody would listen to them.

As a result, hubristic alums are more likely to push for educational reforms and policies that aim to separate the wheat from the chaff.  They know that if we can recruit better people into teaching, get rid of the dead weight (or at least get them to fall in line), stop making excuses, and give the superstars the reins that our schools will shine in no time; in short, that schools need to be shaken up.

It's good that different people bring different perspectives and ideas to the table, but I get the distinct sense right now that the latter group is winning in a rout.  And I'm not sure that's going to be good for our schools down the road.  I don't doubt that improving teacher quality would yield positive results, but I do worry that our narrow focus on such a nebulous trait will prevent us from addressing other, more serious problems.

If we focus solely on the human capital of our teaching force, we will fail to address the home lives or emotions of our students, the competence of our school leaders, the quality of our curriculum, or any number of other challenges our system faces on any given day.

Where will that leave us?  Best case scenario, we're left with a whole bunch of superteachers who miraculously and dramatically raise achievement regardless of any outside challenges or distractions.  Worst-case scenario, we're left with a crumbling system full of disenfranchised teachers who are unable to overcome the shortcomings of their school context, school leaders, curriculum, and other factors and either give up or leave rather than take the blame (or just get fired).

I don't doubt that TFA alums are well intentioned, earnest, and sincere, but right now we're closer to the latter than the former.  And they're not helping.


cross-posted on Blog of the Century

More Liberal Arts for the Least Affluent

I've disagreed with Peter Meyer multiple times in the past, bot in posts (here, here, and here), and in comments on his blog posts (which I'm not going to take the time to dredge up).  So I think it's fair that I point out that he recently wrote what I think is an outstanding post last week on college attendance and poverty.  Also, digging up those old posts just made me realize I've been misspelling his name; my sincere apologies.

Anyway, Meyer makes a strong case regarding why, in an ideal world, we should want everybody to attend college -- and how obtaining a broad, liberal education particularly advantages the most disadvantaged.  Among other things, he points out that:

-exposure to new ideas, new institutions, and new styles of thinking is particularly beneficial for those who were exposed to the fewest of these in their childhood

-a college education opens more options for students compared to limitations placed on them by hyper-specific vocational training

-underemployed college grads still make far more than non-college grads in the same field (a college educated dishwasher makes 83% more, for example)

-increasing college attainment hardly solves our problems but not sending more kids to college creates more

While zillions of logistical hurdles stand in the way of all students procuring a top-notch liberal college education, Meyer concludes by arguing that:

I personally don’t care if a kid decides not to go to college. I would, however, demand that every high school graduate at least be capable of reading (and understanding) David Leonhardt’s story—i.e., your options are probably pretty constrained if you don’t go to college—and that every district superintendent be judged by the number of his or her truly college-ready graduates. If a student decides not to go to college, fine. But at least he or she would have, I would hope, the option of going if he or she wanted to—which is better, I would assume, than not having that option after twelve years of schooling.

I can only find two small points of contention in the post:

1.) the argument that teaching poor kids "a new kind of thinking -- reflection" is the key to getting them out of poverty is either inartfully expressed or demonstrates a lack of understanding.  I'm leaning toward the former, since he also wrote a pretty good piece explaining the genesis of that quote.  At first glance, it might look like Meyer is arguing that kids are poor because they think wrong.  I think, though I could be mistaken, that this was actually was a way of saying that exposing kids to more culture, society, and ideas (e.g. plays, museums, concerts, lectures, etc.) will benefit those who previously had the least exposure.  Indeed, the program driven by this notion was the result of the suggestion of an impoverished prisoner who said that kids needed to get more involved with the what was happening downtown in order to interact in new ways with government, society, etc.

2.) While I agree with Meyer that, ultimately, we shouldn't force every kid to get a high-quality college education: that giving every kid both the option to obtain it and the understanding of how it will benefit them is a better policy goal, I do hope that he personally does care which path any given student chooses.  Given that he argues that more students obtaining high-quality educations improves the lot of our entire society, I'd certainly hope he would then wish that all students chose to obtain that type of education.

Non-Academic Skills and Enterprise Rent-A-Car

One of my principal complaints about current educational policy debates and the educational blogosphere as a whole has been the myopic focus by many on academic skills when so many people (students, parents, employers, and society at large) care at least as much about a long list of other things (e.g. sports, drama, arts, critical thinking, physical health, social skills, sleep, commute distance, peers, and on and on).

So this one tiny example caught my eye last week.  This is from the textbook we're using in the Organizational Theory class I'm teaching (it's on the first page listed in this search):

A focus on customer service enabled Enterprise Rent-a-Car to overtake Hertz and become the biggest firm in its industry. Enterprise wooed its midmarket clientele by deliberately hiring "from the half of the class that makes the top half possible" -- college graduates more successful in sports and socializing than in class. Enterprise wanted people skills more than "book smarts" (Pfeffer, 1998, p. 71) 

I wasn't surprised to read that a firm considered more than academic record when hiring employees, but I was surprised to read that a firm deliberately hired large numbers of recent grads with less-than-stellar academic records specifically because they were heavily involved in their college's social scene.  So I did some more digging and found the source of that little blurb.

 This is from page 71 of Pfeffer's book:

Simply hiring the "best and the brightest" may not make sense in all circumstances. Enterprise Rent-A-Car is today the largest car rental company in the United States . . . In a low wage, often unionized, and seemingly low employee skill industry, virtually all of Enterprise's people are college graduates. But these people are hired primarily for their sales skills and personality and for their willingness to provide good service, not for their academic performance. Dennis Ross, the chief operating officer commented "We hire from the half of the college class that makes the upper half possible . . . We want athletes, fraternity types . . . people people." Brian O'Reilly interpolates Enterprise's reasoning:

The social directors make good sales people, able to chat up service managers and calm down someone who has just been in a car wreck.

Granted, this is almost 15 years old at this point, but I have no reason to suspect that this is no longer true (and anecdotal experience that indicates it is), but it would appear to be evidence that, popular to conventional wisdom, at least one firm does care about that blurb about your fraternity on your resume.

To be fair, O'Reilly goes on to assert that "The Enterprise employees hired from the caboose end of the class have something else going for them . . . a chilling realization of how unforgiving the job market can be".  So one could argue that this hiring practice is based on the desire for cost-savings as well.  But I'd argue that if that were truly the main motivation, then Enterprise wouldn't hire college grads.

Indeed, Pffefer then continues to assert that "organizations should screen primarily on important attributes that are difficult to change through training and should emphasize qualities that actually differentiate among those in the applicant pool." -- One could make a strong argument that the types of social skills referenced above fit those criteria.

In the end, I suspect we all know -- at some level -- that other factors matter more than academic skills (in choosing a school, in gauging a student's/school's success, and in life), but the education commentariat sure doesn't seem to like to act on that knowledge.

Mad Libs for Writing about Strikes

The commentariat and blogosphere sure love a showdown between a union chief and schools chief -- as evidenced by the flood of articles and posts this week on the strike in Chicago.  Unfortunately, about 90% of what's written is entirely predictable and, in the end, utterly useless.

Why?  Because almost every article is written using a secret mad libs cheat sheet that gets rolled out every time there's any controversy involving unions.  The result is that we could replace a few words in just about any piece to fit any strike or labor dispute anywhere in the country.  Regardless of the location, context, or actual issues at stake.  I've posted the cheat sheets below so that you can follow along:


For those opposed to teachers' unions

The [insert controversy] in [insert city] could be a turning point for educational reform in this country.  The union's [insert mildly derogatory adjective] leader, [insert name], has decided to place the needs of adults over the needs of children and [insert local leader] has finally stood up to make sure these children aren't forgotten.  While teachers and unions in most other cities have gotten the memo, the teachers in [city] are stubbornly clinging to the past; the evidence that value-added scores matter is overwhelming, and yet they're standing firm against the district's plan to use VAM scores to account for [X]% of a teacher's evaluation.  It looked like the union might see the light, but after [leader] said [quote taken out of context], it's clear that he/she has his/her head buried in the sand.  This is, indeed, a sad day for the children.  All hope of rescuing our abysmal education system now lie with [district/city leader], who seems to be embracing this challenge, saying [quote taken out of context].  We should all wait with bated breath while this most important of events plays out.


For those who defend teachers' unions

The [insert controversy] in [insert city] could be a turning point for educational reform in this country.  The union's [insert synonym for "heroic"] leader, [insert name], has decided to stand up for the rights of teachers and unions everywhere while [insert local leader] refuses to treat teachers like human beings and is determined to bust the union.  While teachers and unions in most other cities have caved and accepted the anti-union agenda, the teachers in [city] are standing up for their rights and what's best for children; the evidence that value-added scores are unreliable is overwhelming, and yet the district refuses to budge from their plan to use VAM scores to account for [X]% of a teacher's evaluation.  It looked like the union might cave, but after [city/district leader] said [quote taken out of context], it became clear that the union must finally decide that enough is enough.  This is, indeed, a glorious day for unions, the dying middle class, and potentially our entire country.  All hope of preventing the gutting of our indispensable public education system now lie with [union leader], who seems to be embracing this challenge, saying [quote taken out of context].  We should all wait with bated breath while this most important of events plays out.

Why Not Blame Principals and Superintendents?

I'm teaching a course in organizational theory this semester, and our textbook lays out some basics right off the bat.  Among these is that "The most common improvement strategy is upgrading management." (p. 9)

Which struck me because few in ed policy seem terribly concerned with it.  Sure, we have some mayoral takeovers and some turnaround schools and new charters of course need to hire new leaders.  But, in the grand scheme of things, I see little to no effort to seriously upgrade the management of schools -- at either the district or school levels.

Our attention, instead, focuses disproportionately on teachers.  I don't want to suggest that blaming somebody else would be particularly productive, but I genuinely understand why more people aren't blaming educational leaders.

The authors continue "Modern mythology promises that organizations will work splendidly if well managed."  Which also struck me, since I rarely here about a school needing only a great principal to succeed.  And, actually, outside of a few success stories (e.g. the KIPPs), I don't hear much of anything about great school leaders.

So I guess the flip side would be that people don't particularly credit school and district leaders for success either.

I should add one caveat to both of those conjectures, though: at the very local level (think of your child's school or your local district), I think educational leaders tend to receive a fair amount of scrutiny and both the flak and praise that go along with that.

At the national level, though, I hear almost nothing.  Concerted efforts to recruit or train more talented leaders are few and far between.  New evaluation and merit pay schemes for leaders have been quietly implemented some places, but received little attention.  And op-ed pages are silent about the need to upgrade our school management if we want to solve the civil rights problem of our generation.  Puzzling.

Instability: Deal With It, or Address It?

One of the facts of life for many high-poverty schools and districts is constant flux and turnover among district leaders, school leaders, teachers, families, and students.  A typical high-poverty school, for example, sees about 20-25% of its teachers leave each and every year (and that's just at a typical school: imagine what happens at the thousands of atypical schools out there).

I got to thinking about this while listening to a presentation last month on the failure of a curricular reform.  The research that was being presented identified this turnover as the main reason for said failure.  And I buy that explanation: almost every teacher they trained left before the new curriculum was fully established (as did the Superintendent, and the new one wanted to push a different curriculum).

But the larger question, to me, is what to do about this instability and turnover.  The presentation concluded with the argument that future reforms should take this instability into account and essentially be turnover-proof.  That's certainly a pragmatic approach.  But it always makes me squirm when people propose what are essentially dumbed-down reforms for the most troubled schools while others get to do the real thing.

So I had a different thought: why not attack that instability directly?  If a new curriculum can't be successfully implemented because teachers, principals, and district leaders are constantly in flux, why not try to stabilize those three?  For example, a group of teachers and administrators at a school are given a bonus if they stay in place for, say, 5 years.  And the district could lock-in the new curriculum for the same period of time so that it won't change even if the district's leadership does.

Maybe that would be too difficult to do on a large scale, but somebody could certainly try it on a smaller one -- a single charter school might be a good place to start.  After all, charters were started to serve as incubators and laboratories for ideas that traditional public schools hesitate to try.

I'm Back . . .

Hello again, everybody.  The last six months have been busy and eventful, but things are settling down enough now for me to resume writing in this space.

The major update in my life is that I am now an Assistant Professor of Leadership and Policy at Niagara University (just up the road from the eponymous falls).  I'm excited to work with their new PhD program in Leadership and Policy, and I'm particularly excited to work at a college with such a deep commitment to serving the community -- especially the poor and the oppressed.  The city of Niagara Falls has fallen on hard times, and the University's tireless work to help revitalize the community is one of the major reasons that it's among only 24 schools in the country to be recognized as a "Best College" for service learning.

(And it doesn't hurt that the school consistently appears on the honor roll of The Chronicle of Higher Education's "Great Colleges to Work For".)

Anyway, I have a number of great posts in store for you as I face new challenges and embrace new opportunities.  I hope you'll come along for the ride.

update: we, once again, were named as one of 27 schools with exemplary programs in service learning