DaytonDailyNews.com

FLUNKING THE TEST

Flunking the Test: A look at 'No Child Left Behind'

By Mark Fisher and Scott Elliott

Dayton Daily News

The No Child Left Behind act will soon allow the government to withhold money from schools based solely on test scores and could block many students from meeting their goals left for school to take the practice Ohio Fourth Grade Proficiency Test, 9-year-old honor roll student Kylie Miller paused at the doorway of her Kettering home and looked back at her mother. "Mom, don't get mad if I fail again. I'm just dumb," Kylie said. At Meadowdale High School in Dayton, 18-year-old senior Tynisha Edmondson is struggling mightily to pass the final Ohio proficiency test - science - that blocks her from a high school diploma. She has come tantalizingly close, missing the cutoff by a question or two, during the twice-a-year opportunities she has had since her freshman year. But close isn't good enough, and if she doesn't pass her last-chance test she took May 7, she won't graduate with her classmates June 3, and her plans to attend Wright State University next fall will be in severe jeopardy. "Basically, I'm scared," Tynisha said. The state's achievement and proficiency tests already possess great power over Ohio schoolchildren, stopping some from graduating, threatening younger students with repeating or forcing others to enroll in summer school. Test results can brand schools - and entire school districts - as failing. Yet that impact pales in comparison to the power that achievement tests are about to gain through changes in federal and state law. "It's just dismissed by saying, 'Oh well, the student can take it again four or five times.' " Bowers said improvements in Ohio's school-testing program diminish the need for validity studies on Ohio's new achievement tests. The No Child Left Behind act will soon allow the federal government to sanction and withhold money from schools based solely on test scores. And a new and much more difficult statewide high school exit exam known as the The Dayton Daily News interviewed students, parents, teachers, school administrators, state education officials, testing researchers, testing industry officials and other experts, and reviewed more than 10,000 pages of Ohio Department of Education e-mail correspondence with testing contractors and related documents obtained through Ohio's open records law. Among the findings: * The rush for states to comply with the testing requirements of the federal No Child Left Behind act will overload an already strained school-test industry. Rapid expansion of school testing and test scoring has already spawned dozens of errors that penalized students unfairly. * Testing companies looking to trim labor costs have attempted to program computers to score students' essays - but they may have pushed the limits of technology too far. One computer program performed woefully in a Dayton Daily News experiment, awarding a perfect score to a reporter's intentionally bizarre, nonsensical essay and proclaiming, "This is effective writing." * Today's standardized tests cannot be counted on to accurately measure what students have learned. The exams are based on outdated science and often penalize students for making plausible, defensible choices. And Ohio officials instruct test makers to adjust the difficulty of the tests in ways that all but guarantee large numbers of students will initially fail. * Thousands more Ohio high school students could be denied diplomas beginning in 2007 because the new Ohio Graduation Test demands achievement a full two grade levels higher than existing exit exams. More than three out of four sophomores statewide flunked a trial version of the new math exam in 2003, and even after the Ohio Department of Education shortened the test, and the recommended passing score was lowered, nearly a third still failed this year. With the tests poised to grow even more powerful as the driving force in American education, these flaws may ultimately harm the very people the tests are supposed to help: children. "I think our era will be looked at as a new type of Spanish inquisition," said Doris Nell, chairwoman of Lebanon High School's English department. Nell is part of a team that has successfully tutored students and teaching colleagues - a program widely credited with helping the district meet 95 percent of the state's standards, up from 56 percent four years earlier. But Nell is concerned the tests themselves overwhelm classroom instruction. "I don't have trouble with accountability in a general sense, but I have a real problem with our being obsessed with testing," she said. "There are wonderful things that happen in a classroom spontaneously, but now there is not enough time and space for that." Federal law's requirements may have hidden costs The No Child Left Behind act has made testing the centerpiece of school accountability nationwide, with the noble goal of pushing every child to proficiency by 2012. But the goal comes with serious costs and, some say, unintended consequences. In Ohio alone, the annual budget for school testing quadrupled in five years, from $18 million to $75 million. The federal testing mandate is having a similar impact across the country. The General Accounting Office estimated last year that 45 states must develop or overhaul 433 achievement tests by 2008 to comply with the new, tougher requirements. The projected cost: $3.9 billion - and that could rise to $5.3 billion if states use more essays and open-ended questions that require more expensive scoring, according to a GAO report. But the real costs of the new law may be less visible. One question is whether the testing companies can keep up with the demand without compromising quality. Already, there have been embarrassing errors. In 2002, a computer scoring glitch failed 736 Nevada students who had passed a state graduation test. The state fined the test company and reimbursed students who hired tutors after flunking. A similar computer error in Indiana two years earlier resulted in 13 students being erroneously told they had failed a high school exit exam. In April, Connecticut finally released test scores for 125,000 students in fourth, sixth and eighth grades after a three-month delay for re-scoring because of errors on the reading, writing and math tests. Such mistakes will become more prevalent as states rush to meet the deadlines set by the new law, said W. James Popham, professor emeritus at UCLA who ran his own testing company and has authored more than 25 books and 200 journal articles related to testing issues. "But scoring mistakes can be corrected," he said. "What worries me more is the harm that will be done to children because of lousy tests." President Bush chose Hamilton, Ohio, to sign the No Child Left Behind act in January 2002 because U.S. Rep. John Boehner, R-West Chester, co-authored the bill. The law requires states to develop academic standards and test all third-grade through eighth-grade students in reading and math. All states also must have a high school exam in place by 2005. Also in the law: States must report student performance by race and other categories. Students in each group must show "annual yearly progress" on test scores or the school will risk being labeled as underperforming. Such schools start out receiving extra money, but if scores don't come up they can be sanctioned. Just last week, Dayton announced two elementary schools would be "redesigned" with new teachers and administrators because of the law's requirements. The full impact of the No Child Left Behind act has not yet been felt, said Ramsey Selden, vice president and director of the assessment program for the American Institutes for Research. His Washington, D.C.-based testing company is developing tests for Ohio students in kindergarten through eighth grade under a contract with the Ohio Department of Education. "I think there will be a cranking up of activity to meet the needs of the (No Child Left Behind) testing deadlines," he said. The mandates are squeezing the handful of companies that supply virtually the entire nation with standardized tests. Just seven companies account for 85 percent of the test-building market, with industry titans Harcourt Educational Measurement in Texas, Minnesota-based NCS/Pearson Educational Measurement and CTB/McGraw Hill in California handling two-thirds of the workload. "All of the companies are running at capacity or beyond it," Selden said. "Companies are bumping into each other and competing against each other for the same people." Are there enough test-question writers and exam scorers to go around? "Not really," Selden said. Other testing industry representatives are more optimistic. "If you look at the last 10 to 20 years an increase has occurred, the current companies have pretty much been able to handle that capacity with very few problems," said William G. Harris, executive director of the Association of Test Publishers, a non-profit industry trade group. The pressure to produce greater accountability in public schools - and quickly - comes from state legislators as well as from federal legislation. The Ohio General Assembly passed legislation in 2001 that called for a complete overhaul of Ohio's testing system, replacing the existing proficiency tests by 2006 with 15 new tests in third, fourth, fifth, seventh, eighth and 10th grades. The state is in the midst of that transition now, and had to make modifications to bring Ohio into compliance with the federal testing legislation. A September 2003 e-mail exchange illustrates the tension the demanding timelines have created between educators and the testing companies. In the exchange, Nancy Haefeli of the Ohio Department of Education complaied about missed deadlines to Paul Williams, who works at the American Institutes for Research firm that's designing Ohio's tests for kindergarten through eighth grade. "I do not have the greatest confidence in AIR to set timelines," Haefeli wrote. "We have asked for things in the past which have taken months to actually receive, all along being told that it was coming next week or the end of this week. As project manager, I am having a difficult time allowing this practice to continue." Williams' response provides a glimpse inside the pressure the companies are under. "I understand your need for speed and accuracy, and am pushing to obtain both the best I can," he wrote. Ohio's deputy state school superintendent, Bob Bowers, said the companies Ohio works with have been "very diligent" in meeting their obligations. Validity of school tests questioned along with reliability As states like Ohio churn out more tests to measure the proficiency of students, critics worry that the tests themselves are receiving little scrutiny. "State departments of education and testing contractors have a conflict of interest when it comes to asking tough questions of tests," said George Madaus, professor of education and a senior fellow with the National Board on Educational Testing and Public Policy at Boston College. The only time Ohio attempted to gauge how accurately a proficiency test predicted future student success, the test failed. Four years ago, the state education department hired San Antonio-based Assessment and Evaluation Services to assess the validity of the Ohio Fourth Grade Proficiency reading test. The company concluded that the exam's passing score was too high to be used to hold students back a grade, and that the test was an unreliable predictor of whether a child would succeed academically in fifth grade. Madaus said the validity of proficiency and achievement tests "does not get enough attention." "In medicine, we're concerned about false positives and false negatives. If a prostate cancer screening shows a number in the suspicious zone, the doctors tell you about it, but then they order other kinds of tests, CAT scans, biopsies. Seldom do they just go on immediately and cut everything out. They search for other methods to confirm what the initial test suggested to them," Madaus said. "In education, there's no such concern" if a student fails a high-stakes test such as a high school exit exam, he said. The new achievement tests are based on clearly defined academic standards and benchmarks, Bowers said, so there will be no doubt the tests are covering the appropriate material. And testing companies will track student performance on a yearly basis in grades 3 through 8 to determine whether the tests are reliable predictors of success from year to year, Bowers said. Nevertheless, a state board of education member, Carl Wick of Centerville, said the state board "intends to do a validity study" on the Ohio Graduation Test because of its power to determine who receives a high school diploma. Popham said he doesn't need a validity study to conclude that Ohio's new achievement tests are flawed. He said they blur the line between two different kinds of tests with distinct - and perhaps competing - purposes. Ohio's proficiency and achievement exams are designed to measure how well students have learned certain academic concepts and skills. Test questions are supposed to be tied to state standards for what should be taught in each subject. But state officials have told test-making companies they also want to spread out the scores, much as nationally normed tests do. By adjusting the difficulty of some questions, the state ensures that students' scores range from very low to very high, with a large number of kids in the middle - the so-called "bell-shaped curve." Nationally-normed tests such as the Stanford 9 and the Terra Nova are designed to compare students to one another. So if a third grader scores in the 90th percentile, that means that the results were comparably better than 90 percent of the third graders who took that test nationwide. By using the same techniques of normed tests, Ohio hopes to provide better information to teachers on which students excel, simply meet standards or fall behind. In theory, the students at the bottom will get the help they need to improve their scores and receive passing grades. "We put this in our contracts, that we want a test that challenges strong readers, and weaker readers don't bow out right away," said Mitchell Chester, the Ohio Department of Education's assistant superintendent who oversees the state's testing programs. If the scores are spread evenly across the spectrum, "then the test is doing its job," Chester said, adding that scores will improve as teachers and students adjust to the tests. But Popham said score spreading guarantees a certain proportion of students will fail, which strays from the very purpose of an achievement test - to measure learning. "That is wrong-headed and makes no educational sense," he said. Chester suggested that Popham, who once owned a testing company that contributed test questions to early Ohio proficiency tests, has an anti-testing bias and "represents one end of the spectrum." However, even some of the staunchest advocates for standards and testing acknowledge there are problems. Achieve Inc. was created in 1996 by the nation's governors and corporate leaders to help states raise academic standards, improve school testing programs and strengthen accountability. The organization's statement of principles proclaims that, "Tests play an integral role in a state's efforts to raise academic achievement. High-quality tests shine a spotlight on what is working in successful schools, and they sound an alarm when schools and students need help." Achieve officials said test quality needs to be higher. "There is room for improvement," said Joseph Garcia, Achieve's director of public leadership. Garcia, however, said critics have lost sight of the widely varying quality of instruction and grading inconsistencies that existed prior to the standards-and-testing movement of the 1990s. "We can say a lot about how these tests are developed," Garcia said. "We can say a lot less about teacher-developed tests." Some children unfairly labeled While testing experts continue to debate the merits of standardized testing, there is a potential human toll to the effort to ensure rigor in every classroom. Some deserving students may not graduate, while others could be prematurely labeled as failures. Randy Bennett, a researcher who studies new test-building ideas for the Educational Testing Service, makers of the SAT college entrance exam, said today's tests cannot tell the whole story of what a student has mastered in a subject, and therefore judgments based on test scores can be faulty. Some students simply don't test well. Meadowdale High School student Tynisha Edmondson was among the 2 to 3 percent of Ohio seniors who headed into May needing a passing mark on one or more proficiency tests before they can receive a diploma. Edmondson, who has earned As in science classes, hadn't passed her science proficiency exam. She is also part of another group of students: those whose chance to march with their classmates at commencement hinges on their answers to one or two proficiency test questions. Edmondson was so close, some testing specialists said she may have a valid argument that she already has passed the exam. She scored 198 on the proficiency test she took in March - just short of the 200 needed to pass. That's very likely within the standard error of measurement that applies to such tests. The standard error, similar to the margin of error in a political poll, means the results could be off by an equal percentage in either direction. Popham said Edmondson's 198 is statistically indistinguishable from the passing score of 200. "That's not fair at all," Edmondson said. "Why would you want to do that to somebody?" Fourth-grader Kylie Miller may be a long way from graduation, but she's already had a bad experience with Ohio's proficiency testing program. Kylie, who earned all As and Bs at Indian Riffle Elementary School in Kettering, failed all parts of a practice proficiency test last winter. School officials wanted her to attend summer school if she failed the real tests this spring, her mother said. Kacey Winhoven, Kylie's mother, said the exams send the wrong message. "The test has driven the confidence level of my daughter right into the dirt," Winhoven said. "As a parent, that upset me a great deal. I will not let a test destroy my daughter's confidence level." "My daughter's education has been put on hold so that they can now cram all the test materials into her head . . . The test is not fair, accurate or good for our children."