A bipartisan group of lawmakers, working in cooperation with the White House, has argued that America should make greater use of faith-based organizations in providing services to America's needy. In the 1996 welfare reform legislation, Congress included the "Charitable Choice" provision, allowing faith-based organizations to compete for federal social services contracts. President George W. Bush and many Members of Congress now desire to expand the ability of faith-based organizations to compete for federal social service contracts. However, charitable choice continues to spark criticism from both conservatives and liberals. Some are concerned that federal funds will undermine religious liberties of such organizations, thus weakening a fundamental pillar of American life. Others are simply hostile to religion and faith-based values.

In May 2003, researchers from Indiana University-Purdue University Indianapolis released Charitable Choice: First Results from Three States. One part of the study reported on the relative effectiveness of job-training programs administered by faith-based and secular organizations and found no significant difference between religious and secular organizations in helping clients find a job or in their starting wages. The only identified difference was that clients assisted by faith-based organizations worked fewer hours on average and were less likely to receive health insurance. Skeptics are using these findings to attack the President's agenda.

However, Members of Congress should be wary of basing policy on this study, given the actual conclusions and the methodology used in the research. As the authors note, "These are preliminary outcomes. They are suggestive, but by no means conclusive" (p. 4). The authors also warn that "broad conclusions about Charitable Choice as a public policy must await many other such inquiries--in other states, in other program areas" (p. 94). In short, Charitable Choice cannot be viewed as a reliable verdict on faith-based programs. This is true for three reasons.

Overly Narrow Focus
When evaluating the differences in outcomes between faith-based organizations and secular contractors, the authors looked only at job-training programs, which is problematic for two reasons. First, faith-based organizations may have a comparative advantage over their secular counterparts in other social services, particularly where outcomes are evaluated. The authors admit this weakness, noting that "this study says nothing about comparative efficacy of other types of social service provision" and that studies of other types of programs could yield different results (p. 93).

Second, federal job-training programs generally have a poor track record. Recent research reveals how little job-training programs, such as the popular Job Corps program, actually raise the incomes of participants. (See David Muhlhausen, "Congress Spends Billions on Ineffective Job-Training Programs" Heritage Foundation Backgrounder No. 1597, October 1, 2002.) The study's finding of no significant difference between federal and faith-based programs may simply indicate the difficulty of job training rather than a program's relative strengths or weaknesses.

Selection Bias
Social scientists increasingly are using experimental designs to evaluate programs in a variety of areas. Such studies are conducted in a fashion similar to medical research, such as drug trials, in which program participants are randomly assigned to competing treatment groups. However, in Charitable Choice, researchers used data from various agencies on clients who had already been assigned to and enrolled in particular job-training programs, rather than randomly assigned as in an experimental design.

This introduced the potential for "selection bias," in which people favor one provider over another, thereby distorting the results. The authors attempted to mitigate this problem in their statistical model, but acknowledge that an experimental design would be superior.

Furthermore, the findings are premature because they are based on only two years of data from a three-year project. The researchers admit, "it is very possible that these initial conclusions will be modified as both states and providers become more familiar with the contours of this law and the complexities of government contracting with faith-based organizations" (p. 94).

Small Sample Size
An additional problem stems from the small number of faith-based organizations analyzed in the study. The authors themselves admit that "providers in only one state were studied" (p. iv) and that the report's findings are based on a sample of less than 3,000 people who completed job-training programs in only two counties in Indiana. Of those, comparatively few were trained by faith-based organizations.

The study also recognizes that "relatively few faith-based providers have become government contractors. Many religious organizations continue to be wary of partnering with government or continue to have difficulty entering the system" (p. iv). Results would likely have been quite different if a larger number of faith-based providers had been examined or included in the study.

Conclusion
The President's faith-based agenda has prompted a useful debate over the importance of faith commitments within social programs, as well as a discussion about the role that they should play in providing services to America's needy. Much more research is needed, however, to understand fully the contribution and effectiveness of faith-based organizations in the social service arena. Charitable Choice attempts to address many of these important questions, but the problems with the study should give policymakers pause before changing policy based on its research findings.

The authors, however, should be commended for voicing the appropriate caveats and disclaimers while warning against drawing broad conclusions based on the results. In particular, they describe the findings as "preliminary outcomes" that are "suggestive, but by no means conclusive." Acknowledging that their study may "raise more questions than it will answer," the authors emphasize the need for further research (p. 4).

Congress would be wise to continue and even expand the opportunities for faith-based organizations to participate in the social service arena. This would also provide an opportunity to conduct the further research needed to fully evaluate the capability of faith-based organizations to serve America's needy.

Kirk A. Johnson, Ph.D., is Harry and Jeanette Weinberg Fellow in Statistical Welfare Research in the Center for Data Analysis, and Melissa G. Pardue is Harry and Jeanette Weinberg Fellow in Social Welfare Policy in the Domestic Policy Studies Department, at The Heritage Foundation.