OHIO DEPARTMENT OF EDUCATION

Emails show data scrubbing analyzed in ’08

Yet no sanctions were taken against urban school districts

3/11/2013
BY NOLAN ROSENKRANS
BLADE STAFF WRITER
  • CTY-kasich31p

    Pecko

    THE BLADE/AMY E. VOIGT
    Buy This Image


  • It is well-documented that the Ohio Department of Education long knew — or should have known — about extensive data scrubbing at urban Ohio school districts.

    Media reports in 2008 detailed how urban districts excluded thousands of test scores on state report cards. And emails recently released to The Blade by the department show the education department analyzed data scrubbing, discussed media reports about the practice, and held high-level meetings about it.

    Yet no sanctions were taken then against districts, education officials can’t point to any specific additional guidance provided to districts after the reports, and no formal investigation into the practice began until controversy broke out last year in Columbus. Department officials say they had no reason to believe at the time that district actions were improper.

    In 2012, Ohio districts came under scrutiny for a form of data scrubbing, the removal of students from attendance, and thus test-score rolls. In the scrutinized practice, districts broke enrollments for truant students, which under state reporting guidelines meant that their test scores wouldn’t count toward district report card grades.

    While a state auditor’s report says Toledo Public Schools and eight other Ohio districts improperly reported attendance data, an ongoing education department investigation into those practices will likely prove more significant.

    That’s because the education department has threatened possible heavy sanctions against districts that manipulated data. Districts could have their state report cards retroactively downgraded, they could lose state funding, and educators could have their licenses suspended or revoked.

    But while the Ohio Department of Education has taken a decidedly militant tone since new reports of attendance data manipulation arose last year, unclear still is whether the department once ignored — willfully or otherwise — the same data reporting practices for years. 

    Pecko
    Pecko

    TPS practices

    Toledo Public School officials have voluntarily admitted their own longstanding scrubbing practices after last year’s report about Columbus City Schools’ data manipulation. TPS leaders have repeatedly argued since last year that the state Department of Education’s inaction after the 2008 reports was to them a tacit approval of their reporting practices.

    District administrators say they thought they were making a good-faith effort to accurately report attendance and test scores, and say they even discussed their practices with state officials.

    When the state issued letters last month warning districts mentioned in the auditor’s report that they could face sanctions, TPS Superintendent Jerome Pecko said the state hasn’t been detached from the data reporting.

    “All we have to do is point to the 2008 attempt by ODE to look into what was going on in Cleveland,” Mr. Pecko said. “The subject of EMIS reporting was under scrutiny at that time, and they walked away from it.”

    The state’s school data reporting system, the Education Management Information System, is often referred to by its acronym, EMIS.

    In September, 2008, The Plain Dealer reported that the Cleveland Metropolitan School District excluded the test scores of thousands of students. The practice, under the Eugene Sanders administration, helped the district rise from academic emergency to continuous improvement. Cleveland officials admitted they unenrolled students who transferred or stopped coming to school.

    When data-scrubbing reports arose last summer, The Blade requested Ohio Department of Education internal communications for the six months after the 2008 Plain Dealer article. That request was finally fulfilled last week, and consisted of a string of emails between state education department data analysts and top administrators. Some of the requested documents were omitted because the department said they are protected by attorney-client privilege.

    The 2008 story prompted internal discussions in mid-September of that year by state education department staff, as they questioned where the newspaper’s data came from and how the analysis was done. Education department staff believed The Plain Dealer’s numbers were wrong because of a flawed data set, and that Cleveland actually had more scrubbed students than the newspaper reported.

    “I am trying to replicate the 2008 numbers [from The Plain Dealer], but for reading grades 4-10, my numbers are 1,299 tests higher than reported in The Plain Dealer article [3,062 tests],” Lindsey Ladd, then a state education department data analyst, wrote to Michael Carmack, director of Ohio Department of Education’s Office of Enterprise Applications. “The article reports that Cleveland attributed 552 tests to the state for 10th grade reading; I got 889.”

    Staff appeared to try to re-create an analysis done by Mitchell Chester, who in 2007 was an assistant state superintendent who since has left the department. That analysis showed how many students from each district had test scores count only toward state totals, not on district report cards. For example, of the 14,193 TPS students tested during the 2007-2008 school year, 2,706, or nearly 15 percent, did not count toward the district’s report-card scores.

    Staff arranged for a conference call after the analysis. It’s not clear who was on the call, but included on emails discussing the call were Marilyn Troyer, then deputy superintendent of public instruction; Karla Warren, then the education department’s press secretary; Matt Cohen, head of policy, research, and analysis, and Stan Heffner, then associate superintendent for the center for curriculum and assessment. Mr. Heffner would become state superintendent of public instruction, until he resigned last year under a cloud of ethics violations.

    All but Mr. Cohen are no longer with the education department.

    According to emails, Mr. Heffner in 2008 spoke to Mr. Chester, apparently about his 2007 analysis.

    Ms. Troyer, now chief of innovation, improvement, and human capital for New Albany-Plain Local Schools, said she doesn’t remember the call or the preceding conversations.

    The department declined Blade requests to speak directly with Mr. Cohen.

    Department spokesman John Charlton said the agency has long taken steps to address concerns about school data.

    In 2005, the department formed an internal working group to to deal with statewide data reporting issues. At the time, he said, the department did not have authority to address data integrity with districts.

    Legislation in 2007 gave the department power to sanction districts for incomplete or inaccurate data, including corrective action plans. Later that year, the department began an upgrade of EMIS that wasn’t completed until last year.

    The first phase, completed in 2010, gave the department the ability to view all the breaks of enrollment for a student throughout a school year. 

    Toledo Public Schools.
    Toledo Public Schools.

    Flagging districts

    Mr. Charlton said he was unsure if the state has ever implemented any corrective action for inaccurate data since 2008. But he said the department began in 2009 flagging districts with high numbers of students whose test scores only counted for state totals, and alerted districts when they were flagged.

    The Columbus Dispatch followed up on The Plain Dealer report in October, 2008, pointing to similar high rates of scrubbing in Columbus City Schools and subsequent improvements on state test scores.

    “We don’t have any indication that there’s anything improper,” Mr. Cohen told the Dispatch in 2008. “At the moment, we have no reason to say that there’s anything wrong.”

    Mr. Chester, by then Massachusetts’ commissioner of elementary and secondary education, even hypothetically presented the scenario to The Dispatch of schools unenrolling students and then re-enrolling them as a way to alter scores.

    That’s the same practice urban districts such as Toledo have either now acknowledged or been caught doing.

    Attempts to contact Mr. Chester were unsuccessful.

    The education department told The Dispatch in 2008 it checked scrubbing rates against mobility rates; districts that had percentages of students removed from data rolls well above mobility rates could draw further scrutiny.

    Mr. Charlton said that after the 2008 analysis of Cleveland’s scrubbing, the department decided against more analysis because the district had the highest mobility rate in the state and the largest percentage of students in charter schools. “Both of these factors justified why Cleveland had the highest percentage of student tests that did not count at the local level due to breaks in enrollment,” he said.

    Education department officials have argued most districts knew how to properly report data, and won’t take the blame for districts that provided inaccurate information.

    The scope of the current state investigation on data scrubbing will depend on what practices each district apparently used, Mr. Charlton said, as each district’s techniques varied somewhat.

    Under TPS’ practice, administrators withdrew, then re-enrolled, students who were habitually truant, a practice the auditor’s office found improper. “Every [investigation] will be different,” Mr. Charlton said.

    There is no time frame for the state’s investigation. But the past two years’ state report cards for districts named in the auditor’s report were watermarked by ODE, noting the inquiry and that they may be retroactively downgraded.

    Contact Nolan Rosenkrans at: nrosenkrans@theblade.com or 419-724-6086, or on Twitter @NolanRosenkrans