Let’s assume I were to tell you to stop what you are doing, get up from where you are, go to the nearest outside door, open it and go outside, and look straight up at the sky… Then, return and tell me in 5 sentences what it was you saw, and I would rate you against my own personal preconceived notion of what you should have seen…
Oh, did I add,… if you didn’t guess the same as I, you’d flunk, possibly retake a year of school, you would not graduate, it would cause your teacher to get fired, it would close your school, and cause your district to lose funding… It’s all on you, babe.. You had better guess right!
I apologize for not thinking of this sooner… and getting it into the House and Senate committees before the voting of SB51… Those creative neurons just didn’t fire. It was afterwards that the thought occurred to me to find out where the tests were graded, search the local papers there, and find some exposé interviewing people who grade standardized tests.
The tests are graded in Minnesota, and in the Minneapolis City Pages, there it was… Written back in February 2011 by Jessica Lussenhop. .
Here are some excerpts:
DiMaggio had good reason to worry. His score could determine whether the school was deemed adequate or failing—whether it received government funding or got shut down.
DiMaggio soon learned that his boss was a temp like him. In fact, the boss was only the team leader because he’d once managed a Target store.
DiMaggio found out that the human resources woman who’d hired them both was a temp. He realized that their office space—filled with long tables lined with several hundred computer monitors and generic office chairs—was rented.
Eventually, DiMaggio got used to not asking questions. He got used to skimming the essays as fast as possible, glancing over the responses for about two minutes apiece before clicking a score.
Every so often, though, his thoughts would drift to the school in Arkansas or Ohio or Pennsylvania. If they only knew what was going on behind the scenes.
“The legitimacy of testing is being taken for granted,” he says. “It’s a farce.”
In 2009, K-12 testing was estimated to be a $2.7 billion industry. Today, it has almost doubled to a $5.3 billion dollars. Today, tens of thousands of temporary scorers are employed to correct essay questions. This year, Maple Grove-based Data Recognition Corporation will take on 4,000 temporary scorers, Questar Assessment will hire 1,000, and Pearson will take on thousands more. From March through May, hundreds of thousands of standardized test essays will pour into the Twin Cities to be scored by summer. Now scorers from local companies are drawing back the curtain on the clandestine business of grading student essays, a process they say goes too fast; relies on cheap, inexperienced labor; and does not accurately assess student learning.
As part of their training, Indovino and her co-workers read through pre-graded examples out loud, then discussed why each had been scored the way it was. The process quickly divided the room into two camps—the young, unemployed kids who were just there for a paycheck, and the retired teachers. “The retired teachers would argue everything,” says Indovino. All over the room, the teachers were raising their hands and disputing the rubric. Indovino preferred to keep her head down and just score the way she was told to. “I was good at the bad system,” she says.
One student wrote, “Martin Luther King Jr. was a good leader.” With artfulness far beyond the student’s age, the essay delved into King’s history with the civil rights movement, pointing out the key moments that had shown his leadership.
There was just one problem: It didn’t fit the rubric. The rubric liked a longer essay, with multiple sentences lauding key qualities of leadership such as “honesty” and “inspires people.” This essay was incredibly concise, but got its point across. Nevertheless, the rubric said it was a 2. Puthoff knew it was a 2.
He hesitated the way he had been specifically trained not to. Then he hit, “3.”
It didn’t take long before a supervisor was in his face. He leaned down with a printout of the King essay.
There were the students who wrote extremely well but whose responses were too short—in his mind he saw them, bored with the essay topic, hurrying to finish. Or the essays where the handwriting got rushed and jumbled at the end, then cut off abruptly—he imagined the proctor telling the frantic student to lay down his pencil on a well-written but incomplete response.
And there were the kids who just did what they wanted. Like the boy from Arkansas who, instead of writing about the most fun thing to do in his town, instead wrote a hilarious essay on why his town is terrible and how he wanted to burn it down and pee on the ashes.
“I wanted the kid to get the score they deserved,” Puthoff says of his time in the business. “But they want to put them in boxes.”
Farley now understood the reasons why, when he’d been a scorer, his team leaders would tell the room he wanted to start seeing more 3s or 4s or whatever. Supervisors were expected to turn the test scores into a nice bell curve. If his room did not agree at least 80 percent of the time, the tests would be taken back and re-graded, wasting time and money. The supervisor would be put on probation or demoted. When Farley complained to a fellow supervisor about his problem, she smiled wryly and held up a pencil.
“I’ve got this eraser, see,” she told him. “I help them out.”
So Farley simply began changing Harry’s scores to agree with his peers’. The practice soon spread well beyond Harry. “I’d just change a bunch of answers to make it look like my group was doing a great job,” Farley says. “I wanted the stupid item to be done, and so did my bosses.”
That’s when the representative informed Farley that the rubric for her state’s scoring had suddenly changed.
“We can’t give this many 1s and 2s,” she told him firmly.
The scorers would not be going back to re-grade the hundreds of tests they’d already finished—there just wasn’t time. Instead, they were just going to give out more 3s.
No one objected—the customer was always right. “They get paid money to put scores on paper, not to put the right scores on papers,” he says. “They have a bottom line. Why anyone would expect anything else is beyond me.”
And I love this one… When Pearson was contacted for their take on this article, the spokespersons response was “why would anyone take a former employee’s view seriously.”
But then…. who would take the word of a spokesperson for a multi-billion dollar corporation whose profits mostly derive from its perceived legitimacy in honestly grading tests?
If you think testing scores are still real, you must read the whole article here. It crystallizes the problem perfectly, for in it you can see this is not the work of bad people. Just the efforts of a lot of people treading water desperately trying to keep their heads afloat….
But the point I am making and one that must be dear to every parent’s heart, … is this the best system to insure your child gets guided correctly through his life? Remember, this information will be shared and available to every future employer of his.
Is this adequate grounds for firing a teacher?
Is this adequate grounds for closing a school?
Is this adequate grounds for cutting funding to a district?
Is this adequate grounds for scrapping the 31st best(2012) educational teaching facility in the number one best teacher preparation country in the world?
According to Jack Markell, Mark Murphy, RTTT, Rodel Foundation, and 59 of all 63 state legislators… it is.