Schools embracing business principles to measure performance

As most public schools struggle to recover the profound learning loss tied to the pandemic, students in Colorado’s Boulder Valley School District are racing ahead, leaving the dark days of COVID behind.

Boulder Valley’s notable 2024 test scores, which easily surpassed its math and reading marks in 2019, are not the result of a newfangled education theory but instead involve a bedrock principle of American business: Return on Investment.

Academic ROI (AROI) is a data-driven approach that analyzes the costs and outcomes of programs to identify those that deliver the most educational bang for the buck. Hidebound educators might bristle at the use of AROI, conjuring alarming images of assembly-line learning. But Boulder Valley is one of a small but growing number of districts that are adopting the business practice to target funding for the biggest effect on student performance.

What’s a more cost-effective way to help elementary students who are behind in reading: one-on-one versus four-on-one tutoring? AROI helped Boulder Valley figure that out.

It was a no-brainer that tutoring students together rather than individually was less costly. When the district’s research found that small groups were also more effective in helping students learn to read, it shifted its tutoring resources to the four-on-one program as the most cost-effective option.

“The results of four-on-one were a little bit better at a quarter of the cost, so we can now do four times as much literacy intervention,” said Bill Sutter, the chief financial officer at Boulder Valley.

Advocates say other districts need to follow Boulder Valley’s lead, now more than ever, to make up performance declines that the pandemic amplified. School funding has steadily climbed for decades, recently reaching more than $750 billion a year and comprising about 20% of all state and local spending. But falling math and reading test scores in the last decade among the nation’s 50 million students, particularly in low-income communities, show that districts have gotten a poor return on their sizable investment, according to researchers at Georgetown University’s Edunomics Lab, which focuses on education finance.

“We often say it would be great to have more money,” said Professor Marguerite Roza, who heads the lab and promotes AROI. “But we owe it to students to regularly scrutinize the funding we do have to see if we can do more to maximize the outcomes for them.”

Smaller Classes Don’t Always Pay Off

District spending is often driven by what’s trendy in education or in the interest of staff, without examining if it’s also the best use of resources for student achievement. New York State, for example, recently mandated that New York City Public Schools shrink class sizes in all grades by 2028 even though there’s plenty of evidence that the multi-billion-dollar program will not deliver significant academic gains across K-12, according to researchers who have studied similar efforts elsewhere.

The Los Angeles Unified School District’s $6 million artificial intelligence deal with a tech firm is also causing a public stir. The deal blew up in August when the company filed for bankruptcy, highlighting one of the biggest areas of wasteful spending in education – technology. A study by Glimpse K12, which focuses on AROI, found that 67% of educational software licenses were not used by the 275 schools examined, costing districts a total of $2 million a year.

“Our IT folks would go to conferences and were told by salespeople that some software was the best thing on the planet,” said Carol MacLeod, then the chief financial officer of the Lake County School District in Florida and now an AROI advocate. “But we found that some of these software programs were never used. They were probably sitting in a closet.”

AROI aims to ensure that each education dollar is more wisely spent. While the business practice has spread to other fields, including law enforcement, social services, and nonprofit groups, it can face resistance among teachers who are wed to their programs.

The practice emphasizes data-driven accountability, a touchy subject among educators since the controversial federal No Child Left Behind Act of 2002 imposed performance reporting on districts. Even though AROI is controlled by districts, escaping the heavy hand of federal and state overseers, some educators still criticize data-based evaluations as not holistic enough to capture the full benefits of programs.

“The challenge with the AROI process is the pushback that comes with publicly saying that a particular instructional program is not as cost effective as another,” said Sutter of Boulder Valley, which enrolls about 30,000 students. “Educators and parents can be passionate about specific educational programs and the AROI process can get sidetracked quickly if you don’t bring them along carefully.”

This month, the end of $190 billion in emergency federal funding presents districts with a golden opportunity to use AROI to manage steep budget cuts facing many of them. Students have regained only one-third of the learning lost in math and one-quarter in reading, threatening their college and career prospects, according to recent studies. Rather than resort to the common practice of saving jobs and sacred cow programs, advocates say, districts should focus on preserving the most cost-effective programs that will help students recover.

“We are asking districts to assess if their investments may have contributed to better outcomes for students,” said Roza, who trains educators to use AROI. “If the new tutoring seems to be working, then maybe they should get rid of something else that they have had for a long time.”

Boulder Valley Rolls out New Approach

At the relatively affluent Boulder Valley district, the AROI process began about six years ago with a much-needed inventory of its programs. Over many years, the district’s semi-autonomous schools and leaders had frequently added new programs – without ending the older ones – in a process called layering, Sutter says.

As a result, no one in the central office, including the superintendent, knew of all of the ongoing programs and whether they were effective. In this way, Boulder Valley was typical of decentralized districts everywhere.

The central office expected that principals and teachers would question its requests to identify their programs and provide performance information, if it existed. Administrators are often seen as bumbling bureaucrats, Sutter says, so Boulder Valley hired an education consulting firm to play the role of the “bad guy.” The consultants collected information on programs and helped develop methods to estimate their costs, performance based on surveys of principals, and alignment with the district’s broad goals.

The inventory uncovered 251 mostly academic and extracurricular programs, such as elementary Honors Choir and Democracy Day, at its 56 schools. They varied widely in effectiveness. The ongoing cost: $46 million.

The revelation spurred the district to change the way it operated. Just before the pandemic, it installed AROI at the center of its decision-making on big and important programs.

When the pandemic hit, the practice helped Boulder Valley strategize about how to maximize the impact of its share of the massive federal relief program. The district decided to build a program that could live on after the aid was gone.

Federal dollars were narrowly focused on hiring teachers, tutors, and coaches, who provided training in data-driven evaluation, in six low-performing schools. Math and English proficiency test scores rose in about half of the schools between 2022 and 2024, and the district is now tweaking the program elements, based on what has worked best, to further improve results over the next three years.

“We were very strategic with the federal funding to create something sustainable,” Sutter said. “There aren’t any news articles about people being laid off in our district because the funding is going away, unlike in other places.”

A Culture of Accountability

At the large Mansfield Independent School District in Texas, AROI has created a culture of accountability based on regular data-driven evaluations of programs – a rarity in public education.

Every quarter, principals from Mansfield’s 49 schools go before the leadership cabinet in what’s called a “cadence of accountability,” says Michele Trongaard, an associate superintendent of business and finance who is part of the group. They present their progress in reaching a goal as part of the district’s objectives, such as reading proficiency by the beginning of third grade or algebra II mastery by the end of 11th grade. Almost half of the district’s 35,000 students come from low-income families.

The principals must reveal why they did or didn’t make progress and what they might need to improve. This is where AROI comes into play. The cabinet might not approve requests for additional staffing based on whether it’s cost effective.

The funding decisions are not always black and white since some program effects are hard to measure with data, but the process is crucial to improving the academic performance of students, Trongaard says. The district has made steady gains in math and reading test scores since the pandemic but has yet to fully recover the lost learning.

“We hold everybody accountable,” she said. “We can show the community where the money is going. Absolutely, we are getting more bang for the buck.”

For some programs, Mansfield can put an exact dollar figure on student outcomes. As part of its career and technical education, the district invested $17,400 in test preparation materials to assist students in earning certifications in five industries and companies like Microsoft that can help them land jobs after graduation. With 320 students obtaining these five certifications, that came to about $45 per credential, a strong AROI, Trongaard said.

Funding Up, Performance Down

Nationwide, the AROI doesn’t look so strong. The influential National Education Association, a union representing millions of teachers, has long been focused on increasing school funding and, for the most part, has succeeded.

Public school funding has doubled since 2003, significantly outstripping inflation, according to Roza’s Edunomics Lab. The money drove an 11% expansion in instructional employees, giving schools more staff than ever by 2023. Student enrollment, however, grew only 2% during the time span.

The academic benefit to students has been mixed. In the first decade of the No Child Left Behind era of accountability, test scores rose in tandem with more funding, producing “steady” AROI, according to the lab. In the last decade, as test-based accountability was rolled back, student performance in math and reading fell, marking “lousy” AROI.

Mississippi has been a surprising AROI success story, bucking the downward trend. The state was ranked the worst academic performer in the nation a decade ago when Carey Wright took over as state superintendent. She promptly lifted academic standards and then leaned on data to identify the worst-performing districts, providing them instructional reading coaches and measuring the effects on student outcomes. The tight focus on data created accountability and turned Mississippi’s rapid reading improvement into the envy of the nation.

When Wright presented the data to the Board of Education, she spoke in business terms. “I told them, ‘You gave me $15 million for the Literacy-Based Promotion Act. Here’s your return on investment,’” Wright said in an interview with McKinsey & Co.

New York state, on the other hand, is now mandating a massive program estimated to cost up to $1.9 billion annually without any show of its cost-effectiveness. At the urging of the United Federation of Teachers, lawmakers are forcing New York City to significantly shrink its class sizes in all grades. The largest district in the country, primarily filled with black and Latino students, will need to hire almost 18,000 additional teachers to accommodate smaller classes, according to the NYC Independent Budget Office.

The union argues that research shows such a plan is correlated with increases in student achievement. But that’s not a full explanation of the research literature, which doesn’t support the kind of across-the-board K-12 program that NYC is rolling out, according to Matt Chingos, who leads a team of education scholars at the Urban Institute, a research group focused on upward mobility. Instead, research suggests that class-size reduction might be most effective in elementary grades with lower-income students. However, that benefit could be offset by the fact that the new crop of teachers will likely be less experienced and less able to get results.

An AROI approach to class size reduction might have started with test programs in a few elementary schools to measure the costs against student outcomes. Those findings would then be compared with other approaches, such as high-dosage tutoring, to accelerate learning.

Chingos says the biggest flaw in the state mandate is that it sidelined school leaders who know better than lawmakers about the needs of their students. “We don’t have evidence on class-size reduction that would lead us to overrule the judgment of principals about how to improve their schools,” Chingos said. “There’s a lot of different things you can do to tackle that problem.”

Steep Learning Curve

AROI comes with a learning curve for school leaders, which helps explain the slow rollout. Districts don’t typically have the in-house expertise to run formal research studies, but they can tap financial officers like Trongaard at Mansfield to help provide a general estimate of the cost effectiveness of academic programs, researchers say.

Districts also have plenty of data – federal and state test scores; student grading, behavior, and attendance records; and more recently, school expenditure reports and digital educational platforms that track academic growth – to help make plausible AROI estimates.

Marguerite Roza is bullish on AROI based on her experience in training hundreds of school leaders. Her lab provides them with free software tools to analyze AROI data. When they realize that their spending often isn’t correlated with academic achievement – some low-performing schools get a disproportionate share of money while some higher performers get much less – they tend to say, “‘Wow, we need to do something about this now,’” said Roza.

The Alliance for Smarter School Spending, a project of the Government Finance Officers Association, also advocates for the adoption of AROI. Carol MacLeod, who now leads the alliance’s outreach efforts, was among the first school leaders in the country to deploy AROI when she worked in the Lake County district. She saw the benefits after the district developed a focused AROI plan with measurable goals to lift the performance of low-income students through improved teaching and curricula.

Now at the alliance, which supports more than 100 districts implementing AROI, MacLeod sees growing interest in the practice. “I am receiving more requests to speak at conferences about AROI, and more and more people come to listen and avail themselves of the information we make available,” MacLeod said.

SUPPORT TRUTHFUL JOURNALISM. MAKE A DONATION TO THE NONPROFIT WND NEWS CENTER. THANK YOU!