The Pell Institute publishes research and analyses that address equal educational opportunity, particularly the outcomes for low-income, first-generation, and disabled students. Additional publications include occasional papers, policy briefs, and an electronic newsletter.

Upward Bound — Compilation of Links Providing Documentation with Regard to the Flawed Mathematica Reports from the National Evaluation of Upward Bound

The material below provides links to key material that document in detail the sampling and non-sampling errors resulting in false conclusions published in the Mathematica Policy Research (Mathematica) reports from the National Evaluation of Upward Bound published by the U.S. Department of Education (ED) in 2004 and 2009. The material also provides documentation of the statistically significant positive impacts when these errors are addressed using evidence and standards based statistical procedures. Also included is documentation of the irregular procedure that resulted in publication of the final Mathematica report in January of 2009 — the last week of the Bush Administration — over the objections of the ED Technical Monitors for the Mathematica contracts and also over the objection of the Office of Postsecondary Education (OPE). Despite the growing documentation of the errors in the Mathematica conclusions these error filled reports remain on the U.S. Department of Education website in 2015 and continue to be cited by researchers and used to make policy decisions concerning Upward Bound and college access programs. By compiling and posting these documents COE and Pell are calling attention to this unresolved and important issue. The issue is important not just for the Upward Bound stakeholders, but perhaps even more important for the field of government sponsored evaluation research and for the research community that advocates for the use of "evidence based practice". Having reports with untrue and false conclusions put forth as examples of rigorous research that is to be trusted with the highest confidence "without reservations" is an insult to the intelligence of both the practitioner and the research community.

Links and Abstracts for the Documentation

  • 2009 Report from the U.S. Department of Education Contracting Officer's Technical Representative (COTR) — Addressing Study Error in the Random Assignment National Evaluation of Upward Bound — Do the Conclusions Change?

    In 2009, Margaret Cahalan, authored a report that detailed the findings from the Quality Assurance (QA) review conducted by the U.S. Department of Education (ED) Policy and Planning Studies Services (PPSS) staff in the course of performing their jobs as technical monitors for the study. Dr. Cahalan was the head of a team that was responsible for technical oversight for the final of three Mathematica contracts for the 12 year longitudinal study known as the Mathematica National Evaluation of Upward Bound. After concerns about the Mathematica study were raised, the PPSS Technical Monitors began a QA review of all the study files from the start of the study in 1992. The QA review included UB project sample selection, student random assignment file, baseline survey data, and 5 follow up surveys. The QA review found serious sampling and non-sampling errors in the Mathematica design, analyses, and reports. These errors which introduced a serious lack of balance between the treatment and control group with a bias in favor of the control group were serious enough to impact the conclusions of the study. Moreover, the technical monitoring team found that when the errors were addressed using standards based methods there were positive and significant results for the Upward Bound program.

  • 2012 COE Request for Correction of Mathematica Upward Bound Report

    In 2012, the Council for Opportunity in Education (COE) and their affiliated regional Educational Opportunity Organizations (AEEE, Association for Equality and Excellence in Education, Inc.; ASPIRE, Inc.; CATP, Caribbean Association of TRIO Programs; MAEOPP, Mid-America Association of Educational Opportunity Program Personnel; MEAEOPP, Mid-Eastern Association of Educational Opportunity Program Personnel; NASP, Northwest Association of Special Programs; NEOA, New England Educational Opportunity Association; SAEOPP, Southeastern Association of Educational Opportunity Program Personnel; SWASAP, Southeastern Association of Educational Opportunity Program Personnel and WESTOP, Western Association of Educational Opportunity Personnel) — submitted a formal request for correction to the U.S. Department of Education. The 2012 Request for Correction asked that the Mathematica Upward Bound Evaluation reports either be corrected or removed from the Department's website. The request again documented the errors in the conclusions of the Mathematica reports and included full documentation showing the statistically significant and substantively meaningful positive results when the study design and analyses procedures were corrected using standards based statistical methods. The document also gives a history of the irregular review process for the Mathematica final reports that were published in the last week of the Bush administration over the objections of the studies Technical Monitors, Margaret Cahalan and David Goodwin, and over the written disapproval to publish of the Office of Postsecondary Education. The only response ever received to the request was a formal letter in March of 2014, from the office of Planning Evaluation and Policy Development (OPEPD), then Assistant Secretary, Carmel Martin, stating that the Department was not willing to re-open any review of the Mathematica reports.

  • 2012 Statement of Concern by Leading Researchers with Regard to Mathematica Upward Bound Reports

    This statement accompanied the COE Request for Correction and was signed by, among others, the Presidents of the American Evaluation Association (AEA) and the American Education Research Association (AERA) at the time. Each of the signers of the Statement of Concern had reviewed the COE Request for Correction (above) prior to signing the Statement of Concern. No response was received from ED concerning this statement of concern.

  • 2014 Request to Rescind the What Works Clearinghouse Rating

    In 2014, David Goodwin and Margaret Cahalan, the original and final U.S. Department of Education Technical Monitors of the Mathematica contracts submitted a formal Request to Rescind the What Works Clearinghouse (WWC) Rating of "Meets evidence standards without reservations" given to Mathematica Upward Bound reports in the 2009. The following documents were submitted by Dr. Goodwin and Dr. Cahalan to the WWC documenting the errors in the Mathematica reports and including the analysis done by the Technical Monitors correcting for these errors that showed significant and substantive positive findings. The request to reconsider the rating was denied by the WWC.

  • 2014 Setting the Record Straight — Strong Positive Impacts Found from the National Evaluation of Upward Bound, Re-Analysis Documents Significant Positive Impacts Masked by Errors in Flawed Contractor Reports

    In 2014, Dr. Cahalan and Dr. Goodwin prepared an easily understood summary publication that again documents the errors in the Mathematica reports and the results from the quality assurance work done by PPSS Technical Monitoring staff.

  • Other Documents of Interest Replicating the Findings of the Technical Monitors