EDUCATION NOTEBOOK: 
The Real Score on School Choice Research

August 4, 2006

When statistical research makes the headlines, it's important to read beyond the politically charged conclusions and take a look at the fine print. Interest groups often seize on a part of a study's findings but leave the larger truth buried.  

That was certainly the case when the Department of Education released a study last month comparing the performance of students in public and private schools. Defenders of the public school establishment celebrated the study's findings-that public school students perform about as well or better than their peers in private schools-as proof that we don't need reforms that give parents the freedom to choose the best school for their children. 

But the teachers unions shouldn't declare victory just yet. A sober review of the study tells a very different story than what's been widely reported.  

The Department of Education study compared the performance of public school and private school students on the 2003 National Assessment of Educational Progress exam. The raw data show that fourth and eighth graders in private schools "scored significantly higher than students in public schools for both reading and mathematics." 

But when researchers adjusted the data to account for students' backgrounds, such as their socioeconomic status, the results changed. Among fourth graders, public school students scored "significantly higher" than private school students on mathematics but "not significantly different" for reading. Among eighth graders, the results were reversed-the average score of private school students for reading was "significantly higher" than the public schools average but "not significantly higher" in mathematics.  

In sports, this would be a tie. But not in the bizarre world of education politics. To many, these results were a victory for the public school monopoly and evidence that school choice is wrongheaded.  

"The results… are nothing more than we expected," crowed Reg Weaver, president of the National Education Association teachers union, to USA Today, "We know what it takes to improve public education, and it's not vouchers."  

Not so fast. A new report released by Harvard University scholars on Tuesday interpreted the same data set but arrived at a very different conclusion. It found "a consistent, statistically significant private school advantage." The Harvard researchers assert that the government didn't properly account for student background characteristics in their analysis. For example, the government used participation in the Title I and federal school lunch programs to identify disadvantaged students.  

Researchers explain that this skews the government's results against private schools. "When you use participation in federal programs as a measure of a student's family background, you under-count the number of disadvantaged students in the private sector," explained Professor Paul Peterson, a co-author of the report. "Public schools are expected to participate in these programs, while private schools are not."  

The Harvard University researchers have an even more devastating criticism. They caution that the government study tells us nothing about the value of education reforms like school choice. That's because a one-year snapshot study can't say anything about causation-that is, how public and private schools are actually performing relative to each another. "Without information on prior student achievement," they write, "one cannot answer questions about schools' efficacy in raising student test scores." The Department of Education researchers raised this same point in a section of their study titled "Cautions in Interpretation." But that didn't make the unions' press releases or the newspaper reports.  

So how can we evaluate school choice reforms? "Education studies that include measurement over time are much more useful for drawing conclusions about school quality" and the impact of specific education reforms, explains Shanea Watkins, a policy analyst in the Heritage Foundation's Center for Data Analysis. 

There is a body of high-quality academic research that looks at school performance over time, and it proves that school choice programs benefit participating children. In all, there have been eight random-assignment studies-considered the "gold standard" in medical research evaluations-that compared the academic achievement of students who received vouchers through a lottery against the performance of students who did not receive vouchers and remained in public school. Each of these studies has found that students using vouchers to attend private school made academic gains compared to their peers in public school. 

Dan Lips is an Education Analyst at the Heritage Foundation, www.Heritage.org.

EDUCATION NOTEBOOK: 
The Real Score on School Choice Research

August 4, 2006

When statistical research makes the headlines, it's important to read beyond the politically charged conclusions and take a look at the fine print. Interest groups often seize on a part of a study's findings but leave the larger truth buried.  

That was certainly the case when the Department of Education released a study last month comparing the performance of students in public and private schools. Defenders of the public school establishment celebrated the study's findings-that public school students perform about as well or better than their peers in private schools-as proof that we don't need reforms that give parents the freedom to choose the best school for their children. 

But the teachers unions shouldn't declare victory just yet. A sober review of the study tells a very different story than what's been widely reported.  

The Department of Education study compared the performance of public school and private school students on the 2003 National Assessment of Educational Progress exam. The raw data show that fourth and eighth graders in private schools "scored significantly higher than students in public schools for both reading and mathematics." 

But when researchers adjusted the data to account for students' backgrounds, such as their socioeconomic status, the results changed. Among fourth graders, public school students scored "significantly higher" than private school students on mathematics but "not significantly different" for reading. Among eighth graders, the results were reversed-the average score of private school students for reading was "significantly higher" than the public schools average but "not significantly higher" in mathematics.  

In sports, this would be a tie. But not in the bizarre world of education politics. To many, these results were a victory for the public school monopoly and evidence that school choice is wrongheaded.  

"The results… are nothing more than we expected," crowed Reg Weaver, president of the National Education Association teachers union, to USA Today, "We know what it takes to improve public education, and it's not vouchers."  

Not so fast. A new report released by Harvard University scholars on Tuesday interpreted the same data set but arrived at a very different conclusion. It found "a consistent, statistically significant private school advantage." The Harvard researchers assert that the government didn't properly account for student background characteristics in their analysis. For example, the government used participation in the Title I and federal school lunch programs to identify disadvantaged students.  

Researchers explain that this skews the government's results against private schools. "When you use participation in federal programs as a measure of a student's family background, you under-count the number of disadvantaged students in the private sector," explained Professor Paul Peterson, a co-author of the report. "Public schools are expected to participate in these programs, while private schools are not."  

The Harvard University researchers have an even more devastating criticism. They caution that the government study tells us nothing about the value of education reforms like school choice. That's because a one-year snapshot study can't say anything about causation-that is, how public and private schools are actually performing relative to each another. "Without information on prior student achievement," they write, "one cannot answer questions about schools' efficacy in raising student test scores." The Department of Education researchers raised this same point in a section of their study titled "Cautions in Interpretation." But that didn't make the unions' press releases or the newspaper reports.  

So how can we evaluate school choice reforms? "Education studies that include measurement over time are much more useful for drawing conclusions about school quality" and the impact of specific education reforms, explains Shanea Watkins, a policy analyst in the Heritage Foundation's Center for Data Analysis. 

There is a body of high-quality academic research that looks at school performance over time, and it proves that school choice programs benefit participating children. In all, there have been eight random-assignment studies-considered the "gold standard" in medical research evaluations-that compared the academic achievement of students who received vouchers through a lottery against the performance of students who did not receive vouchers and remained in public school. Each of these studies has found that students using vouchers to attend private school made academic gains compared to their peers in public school. 

Dan Lips is an Education Analyst at the Heritage Foundation, www.Heritage.org.