Since 1965, the federal Head Start program has served more than 15 million children at a total cost of over $30 billion. The program's general purpose is to provide comprehensive health, social, educational, and mental health services to disadvantaged students.2 Yet, remarkably, according to the U.S. General Accounting Office (GAO), the early childhood development program has continued to operate without any valid, useful study of how well it works.
This year, the Head Start program goes before Congress for reauthorization. The Clinton Administration has asked Congress to increase funding for the program, from $4.4 billion in fiscal year (FY) 1998 to $4.7 billion in FY 1999. With these funds, the government anticipates serving an additional 30,000 to 36,000 children, raising the total annual number of children served to approximately 860,000. Yet, in its five-year strategic plan submitted to Congress on September 30, 1997, the Department of Health and Human Services (HHS), Head Start's parent agency, is unable--after more than 30 years--to describe precisely what the program is supposed to accomplish.3 Worse, the agency's FY 1999 annual performance plan, prepared in response to Congress's demand that agencies link measures of performance directly with their annual requests for funding, makes no mention of what American parents and taxpayers reasonably might expect in return for the $4.7 billion the Administration is asking them to give.4
As Congress considers the FY 1999 budget for the Head Start program, the evaluation of the program must be made a top priority, and funding must be dedicated to carrying out such an evaluation. Although Congress has authorized a study of the Head Start program, it has neglected to appropriate funds for such a study.5 HHS has demonstrated that left alone, it does not consider a true evaluation a priority. Congress must take explicit steps to establish program accountability by making an evaluation a top priority in FY 1999.
The GAO Find No Useful Study Available
A 1997 GAO report assesses the studies of Head Start undertaken to date and concludes that research on Head Start provides little information on the impact of the program.6 The GAO also reviewed HHS plans for future Head Start studies and concluded that this research still would not furnish the information needed to assess Head Start's effectiveness. In June 1998 testimony before the House Committee on Education and the Workforce Subcommittee on Early Childhood, Youth and Families, Carlotta C. Joyner, director of education and employment issues at the GAO, reaffirmed the findings of the 1997 report. Dismissing HHS claims that it will assess performance of Head Start through the Families and Child Experiences Survey,7 Joyner concluded,
We are not convinced that [HHS] initiatives will provide definitive information on impact, that is, on whether children and their families would have achieved these gains without participating in Head Start.8
Among the main conclusions of the 1997 GAO report are:
(1) although an extensive body of literature exists on Head Start, only a small part of this literature is program impact research; (2) this body of research is inadequate for use in drawing conclusions about the impact of the national program in any area in which Head Start provides services such as school readiness or health-related services; (3) not only is the total number of studies small, but most of the studies focus on cognitive outcomes, leaving such areas as nutrition and health-related outcomes almost completely unevaluated; (4) individually, the studies suffer to some extent from methodological design weaknesses, such as non-comparability of comparison groups, which call into question the usefulness of their individual findings; (5) in addition, no single study used a nationally representative sample so that findings could be generalized to the national program.9
Most of the approximately 600 articles and manuscripts about Head Start that we identified could not be used to answer questions about impact for various reasons. Much of this literature consisted of program descriptions, anecdotal reports, and position papers.10
- The vast majority of research, including that cited as proof of the program's effectiveness by Head Start, was methodologically suspect or antiquated.11 The GAO report states,
The most reliable way to determine program impact is to compare a group of Head Start participants with an equivalent group of non-participants.... Only one of the studies we reviewed used random assignment to form the Head Start and non-Head Start comparison groups.12
The research HHS cites to show Head Start's effectiveness relied on studies from 1976 or earlier. But the GAO points out this has served more as an excuse for not evaluating the ever-changing nature of the program. Specifically, the GAO notes,
HHS maintains that early research has proven the effectiveness of early childhood education, including Head Start, so impact research is not the most effective use of limited research funds. Findings from early studies, however, do not conclusively establish the impact of the current Head Start program because today's program differs from that of the late 1960s and early 1970s. Although program changes might be assumed to increase positive impact, this assumption is largely unsubstantiated.... Later studies offered to support Head Start's impact do not provide enough evidence to conclude that current Head Start is effective.13
- Future HHS research of Head Start will not provide enough information to assess its effectiveness. The GAO concludes,
Head Start's planned research will provide little information about the impact of regular Head Start programs because it focuses on descriptive studies; studies of program variations, involving new and innovative service delivery strategies and demonstration projects; and studies of program quality. Although these types of studies are useful in evaluating programs, they do not provide the impact information needed in today's results-oriented environment.14
- Despite methodological difficulties, Head Start research should focus on the effectiveness of the program. The GAO report supports studies that concentrate on the question of whether a child is better off after his participation in Head Start. It states,
It appears that impact studies on Head Start could be done and would provide valuable results-oriented information.... Moreover, comparisons with other service programs, if designed to answer questions about specific program outcomes, would provide useful information about assessing program impacts.15
Evaluating outcome at the national program level is an accepted program evaluation procedure even for programs with a great deal of variability. It is the only way to determine with certainty whether the program is making an overall difference in any particular outcome area.16
Why Program Evaluations are Needed
The GAO argues that national research on the impact of Head Start is a worthy and attainable goal. The 1997 report concludes,
While we acknowledge the difficulties of conducting impact studies of programs such as Head Start, research could be done that would allow the Congress and HHS officials to know with more certainty whether the $4 billion dollar federal investment in Head Start is making a difference. For this reason, we recommend that the Secretary of HHS include in HHS' research plan an assessment of the impact of regular Head Start programs.17
Congress demanded information on the impact of Head Start when it reauthorized the program in 1990. In addition, the 1993 Government Performance and Results Act (the Results Act)--now in its first year of full implementation--requires that agencies clearly outline their mission, goals, objectives, and ways to measure performance. The poor strategic plan and annual performance plan submitted by HHS demonstrate that the problems are much deeper and more fundamental than just the Head Start program.18 HHS is an agency that cannot answer the basic questions "What are we supposed to accomplish?" and "How well are we doing?" The Results Act demands that agencies answer these questions. And Congress must make sure that agencies fulfill this responsibility or hold them accountable through the budget process for failure to perform. Businesses do not reward employees who perform poorly with bonuses or salary increases; Congress should not reward agencies' failures to obey the law and their lack of accountability and performance with larger budgets.
Absent any reliable evaluation of the Head Start program, it is not only difficult to assess the program; it also is difficult to dissect it and decide which elements work better than others. In the concluding paragraph to its 1997 report, the GAO observes,
The ultimate measure of program quality is impact, however. Until sound impact studies are conducted on the current Head Start program, fundamental questions about program quality will remain.19
Not surprisingly, agencies fear program evaluations because such review might prove a program simply does not work and should be ended. Businesses routinely make a habit of evaluating the success of product lines and services; those that are unpopular or unprofitable are replaced. This simple process does not happen within the federal government, however.
The reality of program shortcomings hit HHS close to home recently. A June 1997 report reflecting the culmination of five years of evaluating the Comprehensive Child Development Program (CCDP)--a bundle of targeted social services delivered to at-risk children and their families--shows no correlation between a child's participation in the program and his eventual well-being. The CCDP spends $15,768 per family each year. In the CCDP program evaluation executive summary, the authors write,
Five years after the program began, CCDP had no statistically significant impacts on the economic self-sufficiency of participating mothers, nor on their parenting skills.... CCDP had no meaningful impacts on the cognitive or social-emotional development of participating children.... Nor did CCDP have any impacts on children's health or on birth outcomes for children born subsequent to the focus children.... Length of enrollment in CCDP did not make an important difference to outcomes.20
This serious failure, despite such significant spending, casts doubt on the ability of lesser ambitious social service programs to produce any greater benefits.
Ways to Evaluate the Head Start Program
HHS must conduct a retrospective evaluation of the long-term impact of Head Start on the income of its participants. HHS can do this a number of ways, including:
-
Evaluating the differential effects of Head Start on its participants' income as shown by its Survey of Income and Program Participation (SIPP).21
-
Using the National Longitudinal Survey of Youth (NLSY), which since 1988 has gathered data on children who attended Head Start. The study would examine a wide range of outcomes, including cognitive, socio-emotional, behavioral, and academic development, while controlling for such factors as family background, the mother's intelligence quotient (IQ), and the mother's level of education. Members of Congress should note there is a tendency on the part of NLSY parents to overstate the attendance of their children in Head Start and ask that the researchers adjust the data accordingly.22
-
Employing the Survey of Program Dynamics, a new longitudinal survey required by the 1996 welfare reform act, to conduct an additional study of Head Start. Through a subset of the 1992 and 1993 SIPP population, the Survey of Program Dynamics collects intensive data on a national sample of children. As a result, Congress would be able to obtain yearly reports on the effect of Head Start on children through the fifth grade.
-
Mandating that the Survey of Program Dynamics be linked with the NLSY at least once by the use of a common performance test. This would enable greater generalization of NLSY Head Start data.
Conclusion
Before Congress agrees once again to increase the funding for the Head Start program,23 it must make a top priority of determining whether the program actually works. On its own, HHS has little incentive to do this, as more than 30 years and $30 billion have shown. Congress must make sure, as part of the FY 1999 appropriations for the Head Start program, that HHS is required to initiate an evaluation. The GAO has indicated ways in which valid research can be undertaken. Such a study will help answer the question of whether Head Start parents would be served better if they could spend the $4,637-per-child grant at the social service agency of their choice or in some other way that more directly benefits children and families.24 Congress must use such tools as the Results Act to send a clear message to HHS that future funding will be impossible to justify absent a serious effort to determine the answer to a simple question: "Does Head Start work?"
-- Nina H. Shokraii is Education Policy Analyst at The Heritage Foundation.
-- Patrick F. Fagan is William H. G. FitzGerald Senior Fellow in Family and Cultural Issues at The Heritage Foundation.
Endnotes
1. The authors would like to thank Jake Phillips, a 1998 summer intern at The Heritage Foundation from Duke University, for his contributions to this study.
2. See U.S. General Accounting Office, "Head Start: Participant Characteristics, Services, and Funding," GAO/HEHS-98-65, March 31, 1998, for detailed information on the populations that receive Head Start services, what types of services are delivered, and when and where such services are available.
3. The 1993 Government Performance and Results Act requires all agencies to submit to Congress strategic plans that outline their missions, goals, objectives, and performance measures. The Department of Health and Human Services' strategic plan can be found on the Internet at http://aspe.os.dhhs.gov/hhsplan. Of the 24 strategic plans graded by Congress, this plan ranked 13th with a failing grade of 43 out of a possible 100 points.
4. HHS's annual performance plan can be found at http://www.hhs.gov/progorg/fin/99perfpl.html. Of the 24 performance plans graded by Congress, this plan ranked 14th with a failing grade of 36.5 out of a possible 100 points.
5. U.S. General Accounting Office, "Head Start: Research Provides Little Information on Impact of Current Program," GAO/HEHS-97-59, April 15, 1997, p. 4.
7. In its September 30, 1997, strategic plan for achieving strategic objective 2.3, which covers the Head Start program, HHS states it will work to ensure program quality through a strong research and evaluation agenda that includes the "analysis and use of findings from a new national system for measuring program outcomes--the Head Start Family and Child Experiences Survey."
8. Carlotta C. Joyner, "Challenges Faced in Demonstrating Program Results and Responding to Societal Changes," Testimony before the Subcommittee on Early Childhood, Youth and Families, Committee on Education and the Workforce, House of Representatives, 105th Cong., 2nd. Sess, GAO/T-HEHS-98-183, June 9, 1998.
9. U.S. General Accounting Office, "Head Start: Research Provides Little Information on Impact of Current Program," p. 1.
11. The GAO evaluated 600 pieces of research on the Head Start program and found that only 22 matched GAO standards for research quality. The criteria and the 22 studies chosen are listed in the report's Appendices.
12. U.S. General Accounting Office, "Head Start: Research Provides Little Information on Impact of Current Program," p. 9.
17. Ibid., p. 16 (emphasis added).
18. Academic outcomes should be the most important measurement of program success. Criteria used by HHS to evaluate Head Start, however, include learning skills, physical health, social behavior, and emotional health of children participating in Head Start; the number of Head Start programs that coordinate with child care services; the number of states that provide health services linkages with Head Start and child care; and the number of Head Start and child care sites having behavioral health services.
19. U.S. General Accounting Office, "Head Start: Research Provides Little Information on Impact of Current Program," p. 23.
20. Robert G. St. Pierre and Jean I. Layzer, Barbara D. Goodson, and Lawrence S. Bernstein, National Impact Evaluation of the Comprehensive Child Development Program. Prepared for Michael L. Lopez, Administration on Children Youth and Families, Administration for Children and Families, U.S. Department of Health and Human Services. Contract No. 105-90-1900, June 1997. Quote taken from the executive summary, http://www.acf.dhhs.gov/programs/rde/ccdp00.htm.
21. SIPP identified Head Start participants in the 1995 and 1996 surveys and in Wave 9 of its 1993 survey,
22. When adjusting the data, researchers should note where and why such adjustments have been made.
23. Annual appropriations for Head Start nearly quadrupled between 1985 and 1995, from $1,075,059,000 to $3,534,128,000.
24. According to the March 31, 1998, GAO report "Head Start Programs: Participant Characteristics, Services, and Funding," Head Start grants average between $792 and $16,206 per child.
