Two important reports on charter school performance were released this week. On December 14, Harvard University economist Caroline Hoxby announced results showing that charter school students are more likely to be proficient in math and reading than students in the nearest comparable public school.[1] And on December 15, the U.S. Department of Education's National Center for Education Statistics (NCES) released a study showing that charter school students in the aggregate lag behind traditional public school students in fourth grade reading and math, though these differences become statistically insignificant when the numbers are broken down by race.[2]

Media stories have largely overlooked the Hoxby study. The New York Times, for example, completely ignored her results in its December 16 article "A Second Report Shows Charter School Students Not Performing as Well as Other Students."[3] Instead, the first report to which the Times alludes is a corroborating study by the American Federation of Teachers (AFT) based on the same data as the NCES study.[4]

But focusing solely on the NCES report and ignoring Hoxby's results paints a distorted picture of charter school performance. NCES itself is quick to warn that its pilot study of charter school students has a number of shortcomings. Context is critical in comparing student performance across different types of schools, and charter schools in one state may radically differ from those in another. Thus, state-based and even more localized comparisons of charter schools and public schools, such as undertaken in the Hoxby study, are a better way to analyze charter schools' performance.

What the NCES Study Says

According to the new NCES study based on data from the National Assessment of Educational Progress, students at traditional public schools outperformed charter school students in math and reading. While these differences tend to disappear when the results are broken down by race, they are statistically significant among students eligible for free and reduced price lunches and among students in city centers.

Reasonably, NCES is quick to caution about interpretation of its results. "While charter schools are similar to other public schools in many respects, they differ in several important ways, including the makeup of the student population and their location."[5]

In addition, charter schools are unique because they are chosen, unlike traditional public schools, which are usually assigned. Because of this difference, NCES warns, "when comparing the performance of charter and other public school students, it is important to compare students who share a common characteristic."[6]

Following-up on these caveats, Dr. Peggy Carr, Associate Commissioner for Assessment at NCES, said, "This pilot study cannot take into account the varying purposes for which charter schools have been created and the conditions under which they function in individual states."[7]

Comparing the Two Studies

The NCES study relies on averages of student's academic achievement at a sample of charter schools from across the country. It compares these averages with averages of academic achievement among traditional public school students, also from across the country.

This methodology effectively neglects the major ways in which charter schools and public schools differ. For example, some charter schools cater to low-performing students who have languished in traditional public schools. Additionally, many charter schools locate in low-income and minority communities. As Manhattan Institute Senior Fellow Jay P. Greene notes, "Because so many charter schools are specifically targeted to struggling students, a large percentage of their minority and poor students face obstacles greater than students of similar demographics in regular public schools."[8] Thus, charter schools and traditional public schools often serve very different student bodies, and comparing them using blanket averages is problematic at best.

In contrast, the Hoxby study looks at virtually all students who attend elementary charter schools and compares them to students who attend the nearest equivalent public school, yielding "matched pairs" of student bodies. In this way, Hoxby effectively addresses many of the problems associated with the NCES's methodology.

And in contrast with the NCES study's results, the Hoxby study shows that charter school students are more likely to be proficient in math and reading than students in the nearest comparable public school. Overall, charter students' reading proficiency rates are 5.2 percent higher than those of their public school counterparts, and their math proficiency rates are 3.2 percent higher.

Further, Hoxby's study shows that charter school students' gains in academic achievement, relative to their public-school counterparts, tend to increase as their charter schools mature. In other words, charter schools are pulling away from public schools in terms of performance. For example, children at charter schools that have been operating for 1 to 4 years are 2.5 percent more proficient in reading than students in the nearest comparable public school. That proficiency advantage increases to 5.2 percent for charter schools open 5 to 8 years, and to 10.1 percent for charter schools operating 9 to 11 years. When comparing charters with long-lived traditional public schools, then, it is only reasonable to evaluate charter schools after several years of operation. Many charter schools included in the NCES study, however, had only been open for a year or two.

Many charter schools in minority and high-poverty areas show even greater achievement gains, relative to their public school counterparts, than charter schools in general. For example, students at charter schools in Hispanic areas have a 7.6 percent advantage in reading, and charters in African-American areas have a 4.5 percent advantage, compared to a 4.2 percent advantage for students in charter schools in other areas. Similarly, students at charters in high-poverty areas have a reading proficiency advantage of 6.5 percent, compared with a 2.6 percent advantage for students in other charter schools. This suggests that charter schools have the most promise in the areas that have the greatest need for innovative educational reforms.

Which Study is Superior?

As described above, the NCES study suffers from several deficiencies that limit its usefulness. At the December 15 press conference announcing the NCES results, Darvin Winick from the National Assessment Governing Board argued that the proper tool for analyzing charter school results is state assessment data, rather than national aggregate data of the sort used by NCES, because knowing something about "common characteristics" is key.

Dr. Peggy Carr of NCES went a step further. When asked about how the NCES study compared to the Hoxby study, she was very clear: "Matching [i.e., the Hoxby methodology] is a more superior design."[9]

Therefore, the results in the NCES study, as well as the related American Federation of Teachers study that is based on NCES data[10], should be taken with a grain of salt. Even Bella Rosenberg of the AFT was unable to offer any substantial critique of the Hoxby study when asked. Rather than address its methodology, she weakly responded, "No one has been able to replicate her results."[11]

Conclusion

Both the NCES and Hoxby studies of charter schools provide information about the academic achievement of students at traditional public schools and at charter schools. On balance, however, the Hoxby study is superior because it uses better methodology. Even the NCES's own associate commissioner of assessment agrees that Hoxby's methodology is more appropriate than the NCES's.

Given Hoxby's results, Congress and state legislators should continue to support charter schools. Educating more than 1 million students nationwide, charter schools represent America's greatest experiment in school choice to date.

Jennifer A. Marshall is Director of Domestic Policy Studies, and Kirk A. Johnson, Ph.D., is Senior Policy Analyst in the Center for Data Analysis, at the Heritage Foundation.