In February 2002, researchers at Harvard University, Mathematica Policy Research, Inc. (MPR), and the University of Wisconsin released the results of a three-year study to determine the effects of voucher-like scholarships on low-income student achievement in New York City.1 In just three years, the vouchers offered by the School Choice Scholarships Foundation (SCSF) were found to have had impressive effects, especially for African-American low-income students. Specifically:
- Standardized reading and math test scores for black students who had used the vouchers (worth up to $1,400 each year) to attend private schools for three years were 9.2 percentile points higher than those of comparable black students who did not attend a private school.
- Overall test scores for black voucher recipients who attended a private school for at least one of the three years were, on average, 7.6 percentile points higher than those of black students who had never attended a private school.
- Parental satisfaction with their child's school was higher among parents of students who attended a school of choice. When asked to assign a grade to their children's school, 42 percent of voucher parents gave their school an "A," while only 10 percent of the parents of the control group public school students did likewise.
Thus, voucher-like scholarships have
significant effects on the education of low-income students. Even
black students who had attended private school for only one or two
years did better than their public school peers did on standardized
reading and math tests. This CDA Report will first discuss how the
School Choice Scholarships Foundation program selected students to
receive the vouchers and then will review the results of the study
published by Mathematica.
;)
Background
In 1996, the School Choice Scholarships Foundation, a philanthropic, privately funded voucher program, began collecting money to fund scholarships for students in New York City public schools to attend a private or parochial school of choice. Since 1997, 1,300 vouchers have been awarded to the children of low-income families; these vouchers, which have a maximum value of $1,400 annually, are redeemable for at least three years. To be eligible for the vouchers, the student:
- Must be entering the first through fifth grades;
- Must be attending a New York City public school; and
- Must qualify for the federal school lunch program (be from a low-income family).2
Between February and April 1997 alone, SCSF received some 20,000 applications for the vouchers. The overwhelming response necessitated that the foundation employ objective criteria for awarding the scholarships. Once the criteria had been established, a series of random lotteries was conducted in May 1997 to select the students who would receive the vouchers.
During the application and eligibility
verification process in 1997, students were asked to take a
baseline achievement test--the Iowa Test of Basic Skills (ITBS)--in
reading and math. (Kindergartners applying for first grade
scholarships were exempt.) Their parents filled out a series of
questionnaires about their satisfaction with their children's
current school, their involvement with their children's schooling,
and demographic characteristics. Researchers at Harvard University,
Mathematica Policy Research, and the University of Wisconsin used
this information to assess the effectiveness of vouchers in raising
student achievement.
;)
Most of the students who applied (about 70 percent) scored below the citywide median score on the ITBS; thus, most of the recipients of the vouchers would be students who had scored below average. The SCSF, in consultation with the evaluators for the Mathematica study, decided that 85 percent of all the vouchers awarded would go to below-average students. Other than this restriction, the chance of an individual student's winning a scholarship was the same for all the students who were eligible.
The basic design of the Mathematica study gave the researchers a unique opportunity to assess the effectiveness of a working voucher program.3 The randomized lottery effectively assigned the students into two groups: the "experiment/treatment" group who received the vouchers and the "control group" who did not. This design is very similar to the way researchers in the medical community assess the effectiveness of new procedures or new drugs; it is generally considered to be the "gold standard" of research protocols.4
What the Research Shows
The experiment/treatment group chosen by the lottery was comprised of 1,000 families in New York City, while the control group had 960 families. The random nature of the lottery assured that the families in each group would not be statistically different from the others in terms of family background, income, or student achievement. Since all the families, as part of the application process, had to fill out background survey forms and take the ITBS tests, comprehensive baseline information was readily available to the researchers for purposes of comparison.
In the spring of 2000, both the experiment and control groups were invited back to complete third-year follow-up questionnaires and tests. These tests were conducted much as the baseline survey was conducted.
ITBS scores are reported using National Percentile Rankings (NPR) instead of raw scores. The percentile rankings show how well students perform in a particular subject relative to everyone else in the country who took the test that year. If a student received an NPR of 50 on the ITBS reading exam, for example, he or she had scored higher than 50 percent of all students taking that test.
The Mathematica researchers reported each student's reading and math results separately, as well as a composite test score that averaged the scores on both tests.
The 2000 ITBS test scores showed that the vouchers had had significant effects, which were most pronounced for black students.5 Chart 1 shows the average difference in the composite ITBS test scores. Black students who used the voucher to go to a school of choice for three years had a composite NPR of 26.83 on their follow-up tests, compared with a control group of black students who had an NPR composite score of 17.60--a 9.23 percentile point difference. The scores of all black voucher students who had attended a school of choice for at least one year were significantly higher than those of the control group. These "part-time" voucher students had a composite NPR of 25.37, compared with a composite score of 17.82 for the control group--a difference of 7.55 percentile points.6
The
results are even more pronounced for math achievement, as shown in
Chart 2 and Table 1. Black voucher students who stayed in a school
of choice for three years scored a full 11.80 percentile points
higher on the math test than did the control group, and voucher
students who had attended a private or parochial school for at
least one year scored 9.65 percentile points higher in math.7
;)
The differences are somewhat lower, but still statistically significant, for reading achievement. Three-year voucher students scored 6.66 percentile points higher than did the control group, and students who had used the voucher for one or two years scored 5.45 percentile points higher.8
An Overview of Other Choice Activity
The findings of this Harvard/Mathematica study on the School Choice Scholarship Foundation's working private voucher program in New York City add to a growing body of research on the effects of school choice on student achievement. Since the SCSF was founded some six years ago, similar funds have offered voucher-like scholarships to students in Washington, D.C., and Dayton, Ohio. The initial studies of these programs show promising results as well,9 as does the research on the students using a publicly funded voucher program in Cleveland, Ohio.10
As the effects of school choice--ranging from greater satisfaction to higher achievement--are better documented and gain public recognition, the demand for choice programs grows. As of June 2001, charter school laws had been enacted by 37 states and the District of Columbia; 10 states had initiated publicly sponsored private school choice programs, from vouchers to tax credits; and more than 50,000 students had been able to attend a public, private, or parochial school of choice due to the efforts of the almost 100 privately funded scholarship programs across the country.11
In February 2002, hundreds of choice supporters, primarily parents and children, lined up in front of the U.S. Supreme Court to show their support for the publicly funded Cleveland Scholarship and Tutoring Program. Inside, the justices heard oral arguments regarding Zelman v. Simmons-Harris, a federal appeals court ruling that struck down the Cleveland voucher program as unconstitutional. The Supreme Court's ruling on the Cleveland program's constitutionality is not expected until late June. If the program is upheld, choice programs are likely to grow in number as viable options for low-income parents who want their children to excel in school.
In 2001, President Bush signed a tax relief bill that expands opportunities for parents to build tax-free savings for their children's education expenses in education savings accounts (ESAs). This year, he has proposed a refundable tax credit that would enable children trapped in failing schools to attend schools of choice. He also has proposed a $50 million Choice Demonstration Fund that would support research projects to develop, implement, and evaluate innovative school choice programs for low-income students that would enable them to attend private, magnet, charter, and other public schools.
In addition, the No Child Left Behind Act, which was signed into law on January 8, 2002,12 mandates public intra-district school choice for children in persistently failing schools. It also gives poor children the right to choose a supplementary service provider, either public or private, and guarantees that parents are given information on the effectiveness of their schools.13 This information will feed the demand for better schools, but all parents will need to be empowered to act upon the information they receive to choose schools of excellence for their children.
Conclusion
The positive findings of the recently released study by researchers at Harvard University, Mathematica Policy Research, and the University of Wisconsin on how vouchers have affected achievement among poor students in New York City should come as no surprise. It is an example of a growing body of research that reveals both the achievement gains voucher-style approaches offer and the greater satisfaction among parents who use those vouchers to send their children to a school of choice.
In just three years, the effects of the School Choice Scholarships Foundation vouchers were sizeable. African-American students who primarily were poor and scored below average on the Iowa Test of Basic Skills and who used the SCSF vouchers to attend a private or parochial school improved their test scores in math by almost 10 percentile points in just one year. That type of improvement is significant, both for the students and their parents, and for the nation.
Kirk A. Johnson, Ph.D., is a Senior Policy Analyst in the Center for Data Analysis, and Krista Kafer is Senior Policy Analyst for Education, at The Heritage Foundation.
1. Daniel P. Mayer, Paul E. Peterson, David E. Myers, Christina Clark Tuttle, and William G. Howell, "School Choice in New York City After Three Years: An Evaluation of the School Choice Scholarships Program Final Report," Mathematica Policy Research, Inc., Report No. 8404-045, February 19, 2002.
2. The income eligibility levels for the U.S. Department of Agriculture's free lunch program and reduced price lunch program in 2001-2002 for a family of four are, respectively, $22,945 (130 percent of the federal poverty line) and $32,653 (185 percent of the poverty line). See U.S. Department of Agriculture, Food and Nutrition Service, "School Programs: Income Eligibility Guidelines," at http://www.fns.usda.gov/cnd/Lunch/Governance/Notices/01-02iegs.htm.
3. These researchers have analyzed similar experiments elsewhere, including Washington, D.C., and Dayton, Ohio. Their findings will be offered in William G. Howell and Paul E. Peterson, with Patrick J. Wolf and David E. Campbell, The Education Gap: Vouchers and Urban Schools (Washington, D.C.: Brookings Institution Press, forthcoming 2002).
4. For a fuller description of experimental research in the social sciences, see Catherine Hakim, Research Design: Strategies and Choices in the Design of Social Research, Vol. 13, Contemporary Social Research Series (New York: Routledge, 1986), pp. 101-116.
5. Latino students receiving the vouchers exhibited statistically similar scores after three years. Other racial or ethnic groups in the experiment did not have enough students participating to conduct a statistically sound comparison.
9. See, for example, William G. Howell, Paul E. Peterson, Patrick J. Wolf, and David E. Campbell, "School Vouchers and Academic Performance: Results from Three Randomized Field Trials," Journal of Policy Analysis and Management, Vol. 21, No. 2 (2002), pp. 207-233. See also Howell and Peterson, The Education Gap.
10. Jay P. Greene, William G. Howell, and Paul E. Peterson, "Lessons from the Cleveland Scholarship Program," in Paul E. Peterson and Bryan C. Hassel, eds., Learning from School Choice (Washington, D.C.: Brookings Institution Press, 1998).
11. See Center for Education Reform, "Charter School Highlights and Statistics," at http://www.edreform.com.
13. For more information, see Krista Kafer, "A Small But Costly Step Toward Reform: The Conference Education Bill," Heritage Foundation Web Memo, December 13, 2001, at http://www.heritage.org/Research/Education/WM66.cfm.
