The recent discovery of serious flaws in a U.S. Department of Agriculture (USDA) land use survey, which compiled data by sampling national land use trends, illustrates why extreme care must be exercised by the federal government in processing, compiling, and reporting the data it derives from the decennial census now underway. Because the 2000 census utilizes both enumeration and sampling techniques, it is essential that both the process and the results obtained are screened to ensure that the type of errors plaguing the USDA's National Resources Inventory (NRI) are not replicated in the 2000 census. As the immediate response to the NRI's release shows, the skewed results of error-plagued surveys can induce government to react with costly programs and regulations that address problems which may not, in fact, exist.

PROBLEMS FROM A FLAWED SURVEY

On March 27, 2000, the U.S. Department of Agriculture warned visitors to its Web site that a "problem" had been discovered in a land use report released by Vice President Al Gore just three months earlier. A revised version of the survey, it advised, would be available in June 2000. Although the March advisory stated that the "revised statistical processing is not expected to significantly change any previously announced findings,"[1] by late April, USDA officials were admitting publicly that they "do not know how significantly" the findings would change.[2]

The flawed land use report, the 1997 National Resources Inventory,[3] was removed from the USDA Web site shortly thereafter,[4] but not before dozens of elected officials in the states and Washington, D.C., had rushed forward with a variety of costly land preservation schemes based on the survey's erroneous data. Vice President Gore, claiming that there was a land use crisis, took the unusual step of releasing the report a day before its formal release by the USDA and Agriculture Secretary Dan Glickman.

At the press conference, the Vice President remarked: "These new figures confirm what communities across America already know--too much of our precious open space is being gobbled up by sprawl." He went on to advocate spending more federal money to preserve farmland and to support the land purchase provisions in his "Livable Communities" plan, which heretofore Congress had largely ignored.

The well-publicized release of these flawed data may have had its intended effect: Congress has scheduled a May 2000 vote on H.R. 701, the Conservation and Reinvestment Act (CARA), which would authorize the spending of $3 billion per year to buy up environmentally sensitive land and to fund other conservation programs.

Potential victims of this needless spending will include more than just the federal taxpayer. The NRI report claimed that the amount of land lost to development in Pennsylvania was extraordinarily high, and it ranked that state second in the nation for land lost to development. This ranking may have led the Pennsylvania legislature to enact, and Governor Tom Ridge to sign into law, a $650 million, five-year land preservation program within two weeks of the survey's release. Similar concerns were raised in other states where the NRI survey also reported above-average land losses.[5]

The flawed NRI survey claimed that Texas led the nation in land used for development, with a reported loss of 2.1 million acres of farmland. In Georgia, where the metropolitan area of Atlanta is subject to the draconian growth-control demands of the Environmental Protection Agency (EPA), the NRI claimed that more than 1 million acres of previous agricultural land and open space had been developed, while Virginia was reported to have lost nearly half a million acres of rural land to development--doubling its rate for the previous five years.

Meanwhile, the report's results assured Oregon's state officials that their highly restrictive anti-suburban land regulation policies for the Portland area had greatly reduced their development rate. (Oddly, however, the NRI survey also reported that rapidly suburbanizing Colorado and Arizona had developed land at a lower rate per new resident than Oregon had, despite its rigid growth-control boundaries).

CHECKING NRI DATA AGAINST OTHER FEDERAL DATA

Had officials at the USDA taken any time to review the NRI findings in advance of the report's release, they most likely would have found that some, if not all, of the data were of questionable accuracy.

Indeed, the authors of this Backgrounder, who at the time were completing an urban sprawl and smart growth report on Pennsylvania's development trends,[6] had contacted the USDA on December 17, 1999, just 11 days after the report's release, to seek an explanation for some of the obvious inconsistencies between the just-released NRI report and a USDA report of similar scope, the Census of Agriculture, which had been released nine months earlier.[7] Specifically, the authors questioned the NRI's finding that the amount of land lost to development between 1992 and 1997 in Pennsylvania (1.1 million acres, of which 901,200 was farmland) was second only to that in Texas, which reportedly lost 2.1 million farm acres.

Such a finding for Pennsylvania made little sense and was markedly inconsistent with the state's development trends. The U.S. Bureau of the Census reported that between 1990 and 1998, Pennsylvania had one of the slowest rates of population growth of any state: Its population grew by only 1 percent over the period, compared with 8.7 percent for the country as a whole. Only three other states--Connecticut, North Dakota, and Rhode Island--had slower population growth rates.[8]

Because housing production and other forms of development closely track demographic trends, it seemed highly unlikely that one of the slowest growing states could have one of the fastest rates of land converted to real estate development. As the Commonwealth Foundation, a Pennsylvania-based think tank, noted, the NRI survey could be correct only if all of the homes constructed in Pennsylvania during that period had been built on 5.3-acre lots.[9]

Apparent discrepancies of this magnitude between related federal data series should have given pause to some of the statisticians and officials at USDA and encouraged further review of the NRI numbers to ensure accuracy before the report was released. Unfortunately, this seems not to have occurred; nor, apparently, was the USDA concerned with disturbing inconsistencies between the findings of its NRI and those of its Census of Agriculture released in March 1999, which covered some of the same land use patterns and was conducted over the same period of time.[10]

Importantly, because the Census of Agriculture is based on an enumeration of farmland (in which every farm and its acreage is counted) and the NRI findings are drawn from a national sample, the Census data would have provided USDA officials with an excellent benchmark to confirm the accuracy of the NRI results before their release. Indeed, had they done so, they would have discovered that the results of the two surveys were vastly different on a state-by-state basis. (See the Appendix for a state-by-state comparison of the USDA Census of Agriculture and NRI findings.)

Pennsylvania and Texas offer the most obvious inconsistencies between the two reports. In the case of Pennsylvania, the NRI survey claimed that the state had lost 901,200 acres of farmland between 1992 and 1997, while the Census reported that only 21,600 acres of farmland had been lost--a figure much more consistent with Pennsylvania's half-century history of very low population growth.

For Texas, the differences are more extreme. Whereas the NRI survey found that Texas had lost 2,105,400 acres of farmland between 1992 and 1997, the Census of Agriculture reported that Texas had gained 421,600 acres of farmland during the same period. Assuming that this is correct, the USDA Census of Agriculture would rank Texas fifth among the states that gained farmland, just behind Wyoming, Nebraska, Oklahoma, and Utah. Because the USDA announced that the NRI results were subject to possible error but made no mention of any similar problems associated with the Census of Agriculture, it must be assumed that Texas most likely did experience a gain in farmland, not the massive loss claimed by the NRI survey.

Surely, someone at USDA should have noticed the extreme differences in results for Texas, with one USDA survey indicating it a top gainer of farmland and the other reporting it as the nation's biggest loser. Moreover, the conflicting activities (farmland lost to development and land becoming farmland) allegedly occurred over the same period of time. Obviously, something is profoundly amiss in data collection at the USDA, as its subsequent withdrawal of the NRI survey from its Web site indicates.

Other notable discrepancies between the NRI and Census involve Georgia. While the NRI reported a loss of 720,000 agricultural acres, the Census of Agriculture found that Georgia added nearly 650,000 acres of farmland. A comparison of the data with population growth would have shown that the number of new acres the NRI alleged had been developed would require the dedication of nearly four acres for each new household in Georgia--an extremely large lot size even for sprawling Atlanta.

Virginia's NRI-reported loss of 300,000 farmland acres is well in excess of the 70,000 the Census of Agriculture reports. The anti-suburban, growth-control model for Oregon did not fare as well in the Census: 32 states had better performances in terms of farmland preservation. Another questionable implication of the NRI findings is that California's 1.3 million new residents required 700,000 acres of new development--less than the 1.1 million acres consumed by Pennsylvania's 30,000 new residents.

CONTROLLING THE RISKS IN SAMPLING

Mistakes, of course, can be made when dealing with large volumes of data, but errors of this magnitude are hard to explain, given the many opportunities to cross-check the results with other federal data and surveys. Apparently, such essential cross-checking was not done by the USDA for the National Resources Inventory of land use, and the consequence of this failing is that states such as Pennsylvania have already committed to vast expenditures of money to solve a problem that may not exist.

Although sampling can be a powerful and cost-effective tool for deriving accurate information about large populations and collections of data, that same power can also serve to magnify and exaggerate relatively small errors in the sampling process and/or in the procedures used to process and compile sampled information. Because of the possibility that small errors can lead to big mistakes, it is essential that samples be properly designed, professionally managed according to generally accepted scientific principles, relentlessly cross-checked against alternative sources of data, and subjected to simple common sense and good judgment.

The NRI debacle not only illustrates the pitfalls of what can happen if careless errors creep into the sampling process, but also illustrates the risks the federal government confronts in ensuring the absolute accuracy of the 2000 decennial census, for which sampling is an important component in identifying the many descriptive characteristics of the U.S. population. It also illustrates the importance of conducting a full companion enumeration to benchmark the accuracy of the sample. Had it not been for USDA's companion census of land use conducted by enumeration, these sampling flaws in the NRI might never have been discovered. For this reason, a detailed, independent investigation should be conducted to determine the nature of the errors and ensure that such errors are not repeated in the decennial census now being conducted by the Census Bureau.

WHAT CONGRESS SHOULD DO

To ensure that the flaws of the sampling process used in the 1997 National Resources Inventory survey are not replicated and the causes of the errors are fully understood, Congress should:

1. Hold oversight hearings with the USDA's professional staff and leadership to determine how the mistakes were made, why they went undetected prior to the NRI's release, and why they remained unacknowledged even after outside information had been received within two weeks that pointed out a discrepancy. At the same time, Congress should postpone any further consideration of H.R. 701, the Conservation and Reinvestment Act, until such hearings can be held and the USDA provides a revised and accurate inventory of land use later this summer.

2. Hold oversight hearings with U.S. Bureau of the Census professional staff and leadership to determine the extent to which similar errors may occur in the decennial census and what measures are being taken to prevent them.

3. Request that the U.S. General Accounting Office (GAO) conduct an independent review of the process by which the USDA conducted the initial NRI survey and other federal data collections. Congress should also request the GAO to suggest appropriate safeguards to ensure that similar problems do not emerge in the compilation of the 2000 decennial census.

4. Give serious consideration to restructuring all federal data collection, compilation, and reporting activities within a single independent and apolitical federal agency. This has been proposed by Senator Daniel Patrick Moynihan (D-NY) in S. 205, which would create an independent commission to study such restructuring, and by Representative Edward Royce (R-CA) in H.R. 2452, which would create a federal statistical service as part of a comprehensive restructuring of the U.S. Department of Commerce. During the 104th Congress, Representative Steve Horn (R-CA) introduced H.R. 2521 to consolidate all federal statistical functions in a single, independent government agency. If data collection and reporting were removed from political influence, higher standards of professionalism could prevail and public confidence in the integrity of the reports and results would increase.

Given the many suspicions and concerns now surrounding the 2000 decennial census and its declining rate of citizen compliance, it is essential that every effort be made to maintain high confidence in the integrity of the nation's basic statistics and data. Had an independent institution been created earlier, much of the acrimonious debate and diminished public confidence in the current census could have been avoided.

-- Wendell Cox, Principal of the Wendell Cox Consultancy in St. Louis, Missouri, is a former Visiting Fellow at The Heritage Foundation. Dr. Ronald D. Utt is Senior Research Fellow in the Thomas A. Roe Institute for Economic Policy Studies at The Heritage Foundation.