New York City public-school kids may be dreading the end of summer, but schools chancellor Joel Klein is the one who’ll really be tested when classes begin again. Last spring, Klein was bragging about the extraordinary upswing in scores during his tenure: a 31-point rise in the percentage of students who passed state reading tests, a 41-point increase in math. That was before state authorities admitted that they’d been progressively more lenient in scoring the tests, and decided to grade more strictly.
The new stringency resulted in the elimination of most of the miraculous gains of the Bloomberg years, and an administration that had lived by the numbers is getting clobbered by them. Klein told parents that the state “now holds students to a considerably higher bar.” This would make sense only if the state hadn’t previously been lowering that bar.
Last year, NYU professor and Klein antagonist Diane Ravitch said exactly that in a Times op-ed, an assertion that Klein claimed was “without evidence.” But the fact that New York students’ scores on the National Assessment of Educational Progress had moved only marginally, even as state scores skyrocketed, was manifest then and is inescapable now.
Steve Koss, a former math teacher at Manhattan’s Lab School, paints the city’s relationship with stats as an example of Campbell’s Law at work. Named for sociologist Donald T. Campbell, the precept holds, essentially, that the more that numbers are used for political purposes, the more they will be manipulated—and distort the decisions they were supposed to inform.
The very measures that get bandied about most often—like those stellar test gains—turn out to be the most suspect, because their main purpose all along was to promote policy decisions that were already made. And where there’s one set of bad numbers, there will be others.
Take the Harlem Village Academies, a group of two middle schools and a high school that Bloomberg called “the poster child” for reform. Both middle schools boast a 100 percent proficiency rate in math for their eighth-graders. But when Koss and others sifted the data, they discovered that far fewer students take the test in eighth grade than the number who start in the school’s fifth grade.
One middle school in 2006–7 had 66 students in its fifth-grade class. But only nineteen students took (and passed) the eighth-grade state tests this year. So less than 30 percent of the fifth-graders actually got to the eighth-grade tests. What happens to the rest? They are held back, or leave the school before finishing—or both.
Deborah Kenny, HVA’s chief executive, says that the schools, which choose students by lottery, start largely with fifth-graders still reading at a first- or second-grade level. “Many of them will take five years instead of four to graduate,” she notes.
Those spending extra years getting up to speed, though, aren’t just fifth-graders who come in unprepared. In one year, as GothamSchools.org blogger Kim Gittleson discovered, 12 of 34 seventh-graders at one school were held back. Kenny blames ineffective teaching, and also notes that it’s not only failing students who leave the school (“We had a valedictorian leave one year”). But 100 percent proficiency is a somewhat different achievement when students take extra years to get there. Or when those who don’t seem to be getting there go elsewhere and no longer get counted.
Whether testing effectively measures schools and teachers is now almost beside the point. As Ravitch notes, “Despite the debacle in New York state and evidence of cheating on tests elsewhere, testing will be more important than ever in the brave new world of data-driven education.” Which brings us back to Campbell’s Law. If it’s impossible to prevent the politicization of testing, the one consolation is that when numbers blow up, they can take the politicians with them.
Have good intel? Send tips to intel@nymag.com.