When he was hired as St. Paul Public Schools superintendent in 2017, Joe Gothard was determined to evaluate the district’s many programs, commitments and partnerships and put a stop to whatever wasn’t working.
Faced with falling enrollment and perennial budget deficits, he wanted to understand whether the district was “effective in everything we do, and if not, can we shift the focus? Those are difficult decisions, but we can’t do everything,” he said in an interview during his first weeks on the job.
But five years in, efforts to connect specific programs to student outcomes have yet to produce any real change.
From special school-day programs to college-access initiatives and after-school partnerships, the district has yet to quit one of its commitments based on data showing it doesn’t work. And nothing’s been scaled up because of data showing it does work.
“I think we had some really good momentum going,” but the coronavirus pandemic got in the way, Gothard said. “That’s really complicated things.”
59 PROGRAMS SCORED
The strategic plan Gothard released in late 2018 calls for “an ongoing process for collecting, analyzing and using information to assess program effectiveness.”
An initial attempt at deciding what to “stop, start and sustain” that school year “got nowhere,” Gothard admitted to the school board.
For the following year, his administration was more deliberate. They built a rubric with eight criteria, including whether a given program was effective, equitable and popular with stakeholders.
A team of five gathered information on around 90 different commitments in the district, ultimately focusing in on and scoring 59 of them. According to materials released in response to a Pioneer Press public records request:
- The team gave high marks to 39 programs, including History Day, Innocent Classroom, Queer Youth Prom and the college-focused AVID and College Possible. They said that group of 39 should be reviewed again after three years.
- Fourteen programs with middling grades should be reviewed after one year, the team said. Those included Youth Pride, after-school instrumental music, awards for bilingual high school seniors, the Sanneh Foundation’s Dreamline coaches and a whole-class model for parent-teacher conferences favored by the teachers union.
- Six programs were recommended to sunset. However, those either were smaller than the team first thought, were run by volunteers or otherwise didn’t really belong in the evaluation process, said Stacey Gray-Akyea, the district’s research director. She said she didn’t think the district was still doing any of those six, but they weren’t stopped because of the program review.
IMPLEMENTATION PROBLEMS
Gray-Akyea said the rubric was solid, but putting the recommendations into action proved difficult.
It’s tricky, she said, to evaluate so many disparate programs of varying sizes and scopes in the same way. Comparing competing college-access programs is easy enough, but can student test scores tell you whether a teacher preparation program is effective? How do you compare the Montessori schools to tutoring? Those hiccups limited the types and number of commitments they evaluated.
Figuring out an initiative’s true cost isn’t as straightforward as it sounds, either, Gray-Akyea said. The number on a budget line may not account for staff time or the space a program occupies.
Another challenge was the team’s timeline. By the time they got the recommendations to Gothard in spring 2020, it was too late to make an impact on the next year’s budget.
“Even if we wanted to stop something, that contract may have already been signed,” Gray-Akyea said.
The team’s 59 recommendations never were formally adopted.
The teachers went on strike that spring, followed closely by a pandemic virus that would kill a school board member and close the schools for about a year, bringing a temporary halt to nearly every commitment they evaluated.
AFTER THE PANDEMIC
Gray-Akyea said she’s now collecting a new round of data and looking ahead to making a new set of “stop, start, sustain” recommendations for the 2023-24 budget year.
That will be the last year the district will be able to spend its $207 million in federal coronavirus relief grants, which have enabled the district to add to its long list of ongoing initiatives.
Gothard said one pandemic-era innovation — a cohort of literacy teachers who work with young students — has shown “really outstanding” progress.
At the same time, the district has hired more support staff for a student body that was feeling especially stressed and anxious in the aftermath of the pandemic and the murder of George Floyd.
Another addition to the budget is providing free meals for all students — not just those who qualify based on low family income or because they attend a high-poverty school. The federal government paid for those meals during the last two school years but declined to extend it, so the St. Paul district has decided to cover those costs from its general fund next year.
“The hardest thing is our students in the last two years have shown us things that aren’t working in our educational system that we didn’t see in the same way before,” Gothard said. “We’ve got these moving priorities.”
The district plans to create progress-monitoring systems with public dashboards that show whether its grant-funded strategies have succeeded. That work will inform district decisions about what they’re willing to pay for once the federal funds are gone.
“We have to make sure that on the other side of this, we are funding the priorities that are both in line with our strategic plan, but more importantly, priorities that are proven to be successful for children,” Gothard said.