Maybe we should call it an “unconfidence interval”
Sharon Otterman wrote over the weekend in the New York Times about the complications of value-added scores for teachers, including teacher-of-record issues and a confidence interval that may give observers more pause than confidence. A commenter to my previous post on teacher-of-record noted the confidence interval issue, which is clearly demarcated in the Times story: A city teacher ranked in the 63rd percentile could actually stand anywhere from 46th to 80th. The metric may do okay identifying the worst and best teachers, but in the middle it is somewhat of a muddle, and it is important if you are writing about such systems to be clear about what they do and do not show.
“As a general rule, you should be worried when the people who are producing something are the ones who are most worried about using it.” This was from Douglas Harris, whose University of Wisconsin colleagues produced the New York measure.
“As a general rule, you should be worried when the people who are producing something are the ones who are most worried about using it.” This was from Douglas Harris, whose University of Wisconsin colleagues produced the New York measure.
posted by Linda Perlstein at
9:21 AM
8 Comments:
Hi Linda,
Sharon Otterman is quite right about the relative inability of VAM to make distinctions of teachers in the middle of the distribution. Bill Sanders has written on this very point (see his chapter with June Rivers in Ted Hershberg and Claire Robertson-Kraft's "The Grand Bargain"). However, as Sanders and Rivers also point out, the measure is extremely accurate at identifying teachers in the very highest and lowest parts of the distributions.
The debate over VAM continues to be clouded on both sides by zealots who would want to use it as a single measure to evaluate teacher effectiveness, and those trying so hard to discredit the method so as to avoid having ANY kind of objective measure of educator quality.
It is unfortunate we have come to this place. VAM is a powerful measure, but it has limitations. We should use it with prudence and good judgment, but we should use it.
Jason Glass
Somewhere in Indiana
This is what I find puzzling: "Everyone," including the journalists who are supportive of VAM, seems to agree that this method should not be used as the only measure to evaluate teacher effectiveness; and yet in Los Angeles teachers were publicly labeled BY THE NEWSPAPER as "least effective" based on the test scores of their students. To this day I can't imagine why the courts allowed this to happen. Publishing the scores and the names of teachers is one thing, but labeling them as "effective" or "ineffective" based SOLELY on those scores is in a completely different category. I predict this will go down as one of journalism's most shameful episodes. Was this an unintended consequence of the Great Recession (desperate effort to rescue a newspaper in bankruptcy)?
Also, reporters seem to accept these test scores as valid without questioning the test itself. Is the test very different from year to year? How long is it in school offices and/or classrooms before it is administered? Are teachers encouraged to review them ahead of time? Who administers the test? Who collects them? Where do they go after they are collected? How long do they remain at the school site? Is the principal (or anyone else) alone with them for extended periods of time after administration? Are teachers encouraged to drill the students on exact test items?
As a retired teacher, I see nothing wrong with evaluating teachers on the basis of test scores, provided these tests are:
Designed for that purpose
Professionally administered (no peeking!)
Because of my many years of experience, I feel confident in saying that these tests, mostly because of the lack of security surrounding them, are almost totally invalid. If they are to be used for high stakes purposes, they'll have to be administered by outside agencies in the same way that the SAT is administered. Teachers should not accept less.
This oversimplification by Jason Glass is false and invalid:
"The debate over VAM continues to be clouded on both sides by zealots who would want to use it as a single measure to evaluate teacher effectiveness, and those trying so hard to discredit the method so as to avoid having ANY kind of objective measure of educator quality."
I agree that value added methodology *warrants consideration* as a component in evaluating teachers. The Los Angeles Times used it as THE gauge with which to judge teachers on a "most effective" to "least effective" scale. That is what was wrong.
I am not saying that value added methodology should never be used at all, and my view is in line with others who share my philosophy, which incorporates deep skepticism about the current "education reform" trend and dismay about the current teacher-bashing trend. So Jason Glass mischaracterizes our view in a manner intended to discredit it.
In fact, I'm not qualified to make the determination as to whether value-added methodology should carry weight and if so, how much weight -- which is why I say it "warrants consideration." But I am every bit as qualified as the Los Angeles Times reporters and editors who DID make that determination, to their disgrace and the disgrace of the struggling, dying profession.
Hi Caroline,
I fear in your rush to nail me as "false and invalid," you missed the point of my post, which is that there is danger in the extremes on both sides of this debate. I would think truth of this point would be fairly difficult to argue.
I am glad we have found common ground in saying that value added warrants consideration and I appreciate your response to my post.
Jason Glass
Jason Glass, I didn't blast YOU as false and invalid but your statement that anyone who opposed the L.A. Times' wrongheaded and harmful use of value-added methodology is "trying ... hard to discredit the method so as to avoid having ANY kind of objective measure of educator quality." I don't know of anyone trying hard to discredit the method so as to avoid having any kind of objective measure of educator quality. That's a straw-man argument, and it IS dishonest and invalid.
If anyone were trying hard to discredit value-added methodology so as to avoid having any kind of objective measure of educator quality, I would oppose them too.
I will take this opportunity to once again emphasis that the Los Angeles Times committed an unforgivable sin against journalistic standards and ethics and needs to renounce and apologize.
I mean emphasize -- sorry. And happy New Year.
CarolineSF:
"The Los Angeles Times committed an unforgivable sin against journalistic standards and ethics and needs to remounce and apologize."
Very well said. I agree completely. For those who do not agree, think of it this way: How many dedicated and excellent teachers were falsely and publicly labeled as "least effective" by L.A. Times journalists? This sort of thing usually is not tolerated in our country and for good reason. It's frightening to think what can happen to a nation's principles once economic desperation has manifested itself. Let's hope for a complete renunciation of this type of scapegoating in 2011.
Previously, quite a number of newspaper editorial boards have de facto anointed themselves the arbiters of what charter schools should be approved and how charter schools should be held accountable (or not, NOT being the universal decree). Those include the editorial boards of my hometown paper and my husband's longtime employer, the San Francisco Chronicle, and my former employer, the San Jose Mercury News.
Appropriate or out of bounds for journalists? Discuss among yourselves.
So perhaps it was the next logical step for newsroom staff to anoint themselves the arbiters of how teacher quality should be gauged, and to proceed to do the gauging.
Post a Comment
Considerate comments are welcome. Uncivil remarks will be deleted. Anonymous comments -- including those unaccompanied by the author's first and last name -- are not permitted.
Subscribe to Post Comments [Atom]
Links to this post:
Create a Link
<< Home