Home > Blogs > Get on the Bus > Archives > 2008 > August > 31 > Entry
Study: Ohio tests invalid for rating schools
A Youngstown State professor’s study of testing data suggests Ohio cannot validly claim schools are improving or slipping based on state ratings and says the achievement gap between black and white students is exaggerated.
Randy Hoover’s research showed Ohio has a large poverty gap in test performance between poor students and their wealthier classmates, regardless of race or ethnicity. Hoover said the correlation of non-school factors like family income with test performance was off the charts.
“This is an extremely high correlation for social science research,” he said. “I’ve never seen anything this high.”
Hoover’s findings support a Dayton Daily News 2006 study of test performance and poverty in Ohio’s 610 school districts that produced similar results. For that study, the newspaper’s computer analysis of the impact of several student characteristics on test scores found median income of the district had by far the most powerful impact on its test performance.
Hoover’s study went further. The three factors he found were most likely to predict school district test performance were the percentage of single parent wage earners, the percentage of poor children and the median family income in the district.
Combining those factors for what Hoover called the “lived experience index” he found they were responsible for at least 61 percent of a district’s test performance. Hoover studied about 60 variables to see which correlated best with test performance.
“On most of them I got no correlation whatsoever,” he said.
Karla Warren, an education department spokeswoman, said the study does not fairly reflect efforts to ensure tests treat students of all wealth levels and ethnicity evenly.
“The Ohio Department of Education doesn’t support the findings of this study, and we stand by our tests,” she said. “Our tests undergo a detailed review process.”
But Hoover argues the study shows Ohio draws invalid conclusions about the quality of school districts by using tests that largely measure how poverty impacts each district. In fact, when Hoover, an education professor and former classroom teacher, looked at school district performance after controlling for “lived experience” factors, he found a different range of school district test performance — far more high poverty districts scored well and more wealthy district scored badly.
“There are as many school districts with advantaged students significantly underperforming as there are school districts with disadvantaged populations,” Hoover wrote. “The stakeholders reading the Ohio school report cards have no way of knowing if the schools and districts are actually advancing academic achievement.”
For instance, the gap between white and black student scores — white students score much higher on average — also nearly disappears when you control for poverty and non-school factors.
“The dominant force in the lower district performance as percent black increases is lack of wealth, not race,” Hoover wrote. “When we controlled for the social-economic factors of lived experience, there was only a very slight relationship between the percent black and actual district performance. There is an achievement gap, but it is extremely small.”
Warren said two state committees review test questions for any hint of cultural, racial or ethnic bias.
Read Hoover’s study online here.
Update: Steve at the Daytonlocal blog adds some comments on this story.
Permalink | Comments (25) | Post your comment | Categories: Testing

E-mail
E-mail
E-mail
Comments
By Oldprof
September 3, 2008 10:17 PM | Link to this
School Supporter (not classic): you are becoming incoherent in your ad hominem. I suggest you browse over to Scott’s more recent blog entry, “Supporters launch drive to save Stivers art teachers (again)”. My post there clearly provides evidence that yes, more funding leads to success. If you have counter-evidence (more than a broken link to an irrelevant website) let’s have it; if not, I’d expect that few think your debate is compelling, and your name-calling tiresome.
By School Supporter
September 3, 2008 12:26 PM | Link to this
Translation: “We have your kids. Send money. Don’t be deceived by test scores. We’ll tell you when your kids are learning, but you need to send more money. We’re trained, certified, degreed, and accredited professionals. You are just parents and taxpayers. Trust us. We’ll let you know when you’ve sent enough money. Viva La Revolucion Bolivariana! Hasta La Victoria Siempre!” Hoover & Co. is an embarrassment to educators throughout Ohio. Professional development in Ohio is supposed to prepare teachers to “use data to improve student achievement.” This differs from using statistics to evade accountability (while denying that very intent). Maybe closing a few Ohio ed schools can drive that point home to ed faculty and deans. Yes, there are issues with testing—and responsible citizens have worked that issue for the last ten years. See here http://ohiorc.org/value-added/ and here: http://www.edexcellence.net/detail/news.cfm?news_id=735&id=130
By Scott Elliott
September 2, 2008 11:11 PM | Link to this
Wash Twp, right. When Hoover controls for socioeconomic issues, he finds different high and low achieving districts. The state system largely IDs wealthy districts as high performers and poor districts as low performers. The state could do two things. It could do a Hoover-style adjustment to show who outperforms and who underperforms and add that ranking to the report card. Or it could seek new tests that don’t correlate so strongly with non-school factors. Ohio does neither. If it did, stakeholders would have better info about how their local schools perform.
By Washington Township
September 2, 2008 10:56 PM | Link to this
Hoover is indeed using socioeconomic factors in constructing his lived experience index. The index permits one to say that the proficiency scores need to be adjusted, or the difficulties of teaching & learning in an environment with higher poverty and single parent families need to be taken account. It compensates for these factors.
By Scott Elliott
September 2, 2008 9:02 PM | Link to this
Wash Twp, Hoover argues just the opposite — that the tests give us little information about student academic performance, not that instruction can’t impact learning. It’s the measurement of largely non-school factors rather than in-school effects, that Hoover opposes. Ohio, he says, purports to rate districts for their academic achievement but uses tests that mostly tell you which districts are rich and poor. If Ohio used Hoover’s approach to estimate where districts should rank based on wealth compared to where they actually rank you could get useful information about which districts are over and under achievers.
By lou
September 2, 2008 5:16 PM | Link to this
As someone who studies the report cards looking for trends I noticed this a few years ago. I look at many districts report cards, not just Dayton’s and my son’s, but many others (OK I am a weird person who loves to work numbers). I am glad someone finally did a full study but the State will never read it or admit they might have made some errors. Same way no one noticed the 5th grade test scores were lower across the board. Fair or not, correct or not the State is not going to admit they are wrong or change anything.
By Washington Township
September 2, 2008 5:04 PM | Link to this
What this study implies is that that the problems with poor performance is a consequence of socioeconomic factors, not how kids are educated. If so, then why raise school taxes, since it won’t really effect performance?
By Scott Elliott
September 2, 2008 2:22 PM | Link to this
Some people are commenting both here and in e-mails to me that this study is old news. It’s not. While many studies have confirmed the strong connection between family income and test scores, that’s only part of the story here. Hoover’s study shows Ohio’s accountability system punishes districts for having poor kids and rewards others for having wealthy kids. That is the point of his research. Ohio could easily add report card measures that show districts scoring much higher than would be predicted by the wealth in the community as well as those scoring much lower than wealth would predict. That would be very useful information for various stakeholders to have.
By Lea
September 2, 2008 8:04 AM | Link to this
It took someone to STUDY this stuff? This is common sense. In a great part, those with a lower economic status a) don’t have time to help their children because they’re trying to make money to live and/or b) don’t care and have fallen into the entitlement (i.e., welfare) trap (what we call “baby mommas/daddies”). Single-parent households many times fall into these two categories. Now take this example: mom has custody in “emergency” district, children miss an average of 18 days per year but are passed through with a “D” average. Mom works 3rd shift and leaves children with babysitter a lot, refuses to let Dad see them. Then… Dad and stepmom get custody, live in “excellent” district, interact with kids, help them. Children miss 2 days each that year, grades jump through the year to “A” and “B” average. Now here’s the kicker. Dad and stepmom make about $10K less per year than mom and her boyfriend. But they are involved and care. It’s the parents who make the difference. Always has been and always will. We need to STOP teaching the tests; our children are learning nothing useful there.
By deb
September 2, 2008 5:09 AM | Link to this
Rick - one of the problems with suspending/expelling more students at DPS is that lowers the attendance rate, and lowering the funds from the State…These are based on student attendance. “When you do what you’ve always done, you get what you’ve always gotten.” DPS and other schools that educate our urban youth, need to do it different….and in a manner that will help our kids be successful in spite of parents who might not support the educational process. Unfortunately, I don’t know how that would be - but it is something to think about…
By Laura
September 1, 2008 8:45 PM | Link to this
What I would be interested in knowing is whether anyone at the Ohio Dept of Ed has bothered to take a serious look at this study to see if it has any validity. Instead, it sounds as if they are saying that the results of the study are irrelevant and they just don’t care what the results are. Ms. Warren says the study doesn’t take into consideration the efforts they make to ensure the tests are fair. Well, “efforts” are nice but if they aren’t sufficient someone needs to make more “effort” and maybe they need to at least take a good look at the results of the study before deciding that it doesn’t have any merit.
By Rick
September 1, 2008 4:10 PM | Link to this
It’s time we took action against “truant” parents. In addition, Dayton needs to expel and suspend more students.
By School Supporter
September 1, 2008 3:37 PM | Link to this
Oldprof writes, “evidence supporting my charge that the Ohio Board of Education is incompetent.” Feel free to list more Ohio college faculty who support the proposed Strickland/Obama education policy partnership (and oppose Dr. Zelman’s bipartisan support for civil rights). Consulting ratemyprofessors.com is both entertaining and informative; for Hoover: “A legend and icon. He’d be proud I teach his word to conservative right-wingers in Baptist Georgia now.” “someone needs to monitor his activities … he is full of himself and wothless … get rid of him” “The most arrogant person I have ever met. He spent the first week ranting about ‘fascist Republicans’” Of course, the real problem is Hoover’s apparent ignorance of the expectations Ohioans hold for school success and how to achieve it.
By Oldprof
September 1, 2008 2:10 PM | Link to this
School Supporter—(not “School Supporter Classic”, I presume?)—it is a mark of soft thinking to attack an academic study ad hominem. If you can’t find a flaw within the study itself, calling Hoover wrong because he is a “left-winger” is simply prejudicial and unfair. In fact, considering the mess that conservatives have made of public education through underfunding, overmandating, and the illusion of “choice”, the opposite conclusion might be more valid.
By Concerned Mom of 3
September 1, 2008 9:57 AM | Link to this
How hard would it be to apply the statistically signifant factors from Professor Hoover’s study to the state’s evaluations of school districts? The state should already have a lot of the information. (Tax forms, records of who is receiving public assistance, prison records, etc.)Student scores on the state tests are a reflection of parental success or failure. I am a perfect example of this… My children attend Dayton Public Schools. They are presented the same lessons as the other DPS students. They are able to pass the tests with flying colors. It is because they have parents who value education.
By School Supporter
September 1, 2008 9:34 AM | Link to this
Unbelievable. Hoover is the very sort of left-wing academic that Governor Strickland depends upon for crafting his education reforms. In 1999 Achieve, Inc. recommended the new State Superintendent (Dr. Zelman) not rely on Ohio’s ed schools to reform Ohio’s K-12; those efforts resulted in recognition from the Leadership Conference for Civil Rights (see: http://www.realizethedream.org/programs/title-i.html). Now Governor Strickland want to partner with Barack Obama to reform Ohio’s schools. Does he think we won’t notice the Hugo Chavez fan club among their education advisors? This bunch is sure to produce rhetoric about education and democracy while doing little to fix urban schools. Like this: “Today, the conservative political agendas of school reform and accountability have changed the very essence of public schooling. … However, the problem is that many of the most critical issues that need to be seen, understood, and addressed by those holding the vision of democratic schooling and democratic policy are counter intuitive to the dominant political ideology enfleshed by the citizenry of America.” Translation: “Left-wing academics know how to run public schools better than the public. Trust us with your money and your kids. Long live Hugo Chavez!” (Or did I mistranslate “Viva Presidente Chavez!”?) With friends like these, Governor Strickland won’t need enemies as he tries to promote public education!
By Oldprof
September 1, 2008 8:41 AM | Link to this
Once again, evidence supporting my charge that the Ohio Board of Education is incompetent. Perhaps Susan Haverkos and Carl Wick will now take a cue from Susan Tave Zelman and start finding a mode of service more suited to their skills.
By deb
September 1, 2008 8:15 AM | Link to this
I took a class a couple years ago in educational research. There was this one study that I read(& sorry, don’t remember the particulars) that did see scores improve with smaller classes AND the implementation of mental health services. The mental health professionals provided classes on anger management, peer mediation, developing positive relationships - all of the things that many parents do as part of their job of raising their children. It is when the children feel SAFE in school, that they begin to learn. Our children that are living in poverty ridden areas don’t feel safe because their basic needs(shelter, food, heat, clothes) are not being taken care of. I believe that if you swapped the teachers from Oakwood and DPS - the Oakwood teachers wouldn’t fair very well w/ the test scores either…
By School Supporter
September 1, 2008 8:06 AM | Link to this
Explain the rationale for ignoring Hoover’s work in Marc Dann’s complaint to close charter schools. That complaint is based solely on those suspect test scores and implicitly argues that concentration of at risk students should not prevent achievement of an excellent rating. (That is, a school which serves families in public housing and fails to do as well as Oakwood ought be closed as a failed public trust.) If the Ohio Education Association finds any merit in Hoover’s work, they sure kept quiet when coordinating with the Attorney General’s office. Also, correlation to SES is old news. Please note that Ohio now includes value-added in its assessment. Finally, Pogrow’s “Good Statistics About Bad Programs Tell Little” explains a good deal of the real problem.
By Rich
August 31, 2008 8:21 PM | Link to this
Just to save everyone else from too much time searching across the Internet, I’ve found Prof. Hoover’s website: http://www.cc.ysu.edu/~rlhoover/OAT-OGT/ It contains both his 2000 study of Ohio Achievement Tests and the current 2008 study of Ohio Proficiency Tests. Looks to be VERY in-depth, but well worth studying. An education blog on the Cleveland Plain Dealer included a comment that Gov. Strickland promised to read Hoover’s report, but I wouldn’t hold my breath for any real change in so-called accountability via the state report cards. Our political leaders find value in having such report cards, even if their conclusions are invalid or less than useful for real school improvement. As others have commented here, any experienced teacher would tell you that socio-economic and family issues far outweigh any other as a predictor for school success.
By Rich
August 31, 2008 7:54 PM | Link to this
Scott — PLEASE provide a link to this professor’s study. I’m going to Google until I find it for myself, but lots more readers would benefit if you would print it for all of us. Thanks!
By the obvious
August 31, 2008 4:55 PM | Link to this
The findings of this study are so readily apparent to any competent educator. The state committees can review the test questions for racial and ethnic bias. They should probably also review the questions for gender bias as well, but that’s not the point of this study. The dominant factor is socio-economic status. Still, in Ohio, our school funding system is arranged so that students born into poverty are most likely to attend schools without adequate funding. The “achievement gap” and socioeconomic gap between rich and poor grows wider. Sprinkle that with some racism, and it explains much about the apparent success…or lack of..in our school systems across the state. Simply post two maps, one showing the average socioeconomic status of a district to the average scores. The maps will match…and have for years.
By A Parent
August 31, 2008 4:26 PM | Link to this
We keep doing studies year after year when it has been demonstrated that money isn’t the real problem. The problem is parent involvement and peer group expectations. Income shows up because parents who are more likely to have a strong work ethic are more likely to be involved in their children’s education. Stop looking at money and start looking at the parents. How many children will continue to fail at school because we make excuses for them and their parents?
By KT
August 31, 2008 4:19 PM | Link to this
Well, I hate to say, “duh”, but we have known about the correlation between poverty and educational achievement for a long time. This is why Dayton Public Schools can’t get ahead. What, do all of you really believe that none of Dayton’s teachers can teach? DPS teaching staff, for the most part, deserve combat pay. Look at what happened at the fair last night…this is what the teaching staff battle everyday.
By null
August 31, 2008 3:32 PM | Link to this
Proof positive that the problem does not lie with teacher performance. It’s time to stop blaming and pressuring teachers to raise scores when the real problem is completely out of their control—unless (tongue-in-cheek) teachers will now be required to boost parent incomes in an attempt to show increases. Of course, we know full well that this study will never be mentioned by the administrative taskmasters; as you can see, ODE is already trying to refute repeatable test results with lame excuses and no real evidence to back it up. Just a side note: I wonder if one of the variables measured was conducting school during the dog days of summer without air conditioning.