Harvard Educational Review
  1. Randomized Experiments for Planning and Evaluation

    A Practical Guide

    By Robert R. Boruch

    Thousand Oaks, CA: Sage, 1997. 263 pp. $21.95 (paper)

    How can education policymakers evaluate whether a new program has its intended consequences for its target population? In Randomized Experiments for Planning and Evaluation, Boruch uses scores of examples from education and other social science fields to show how carefully designed field trials can answer this question. Through these examples, he describes in detail the necessary steps in the design of a good field trial, from the initial framing of the questions to the final interpreting and reporting of the results.

    Boruch's excellent chapter on the ethics of field trials ("Ethics, Law, and Randomized Experiments") provides much to consider, especially for those who have an aversion to "experimenting with people." For example, in the medical field, an analysis of carefully designed field trials revealed that about 20 percent of new therapies improve patient outcomes, 20 percent are worse than the standard treatment, and the remaining 60 percent perform about the same as the standard treatment (p. 69, quoting Gilbert, McPeck, & Mosteller, 1977). Yet the creators of these new therapies were certain that they would improve patient outcomes. A similar study in the social sciences found that 35 percent of the new programs succeed relative to the control groups, 20 percent fail, and the balance shows no difference (p. 69, quoting Gordon & Morse, 1975). Concluding this chapter on the ethics of field trials, Boruch states:

    A related stream of relevant empirical work over the last 15 years suggests that nothing improves the chances of apparently successful innovation as much as lack of experimental control. Marked enthusiasm for an innovation is negligible in reports on controlled trials. Declarations that a program is successful are about four times more likely in research based on poor or questionable evaluation designs as in that based on adequate ones. . . . Badly designed research can yield misleading results and is ethically unacceptable on that account. (p. 69)

    Educators may reasonably ask themselves whether the evaluations of the innovations they are asked to implement or the curricular materials they are told to use have been well designed, as well as whether the results have been interpreted without bias. These questions are especially important to ask of commercially available innovations, programs, or curricular materials and textbooks, where their vendors, who have an obvious self-interest, try to demonstrate that their product really works to improve schooling for our students. And yet, how many school or district decisionmakers are inclined to ask this question and have the knowledge to evaluate the answer? For this reason, Boruch's practical guide can be a valuable addition to the library of every department chair, principal, and superintendent.

    This clearly written text with its extensive use of examples of successfully implemented evaluations in the social sciences promises to become the successor to the existing venerable texts on the design of field trials and evaluations.

    B.N.
  2. Share

    Abstracts

    Reporting Ethnography to Informants
    Reba N. Page, Yvette J. Samson, Michele D. Crockett
    On the Theoretical Trappings of the Thesis of Anti-Theory; or, Why the Idea of Theory May Not, After All, Be All That Bad
    A Response to Gary Thomas
    Kanavillil Rajagopalan
    From the Treaty of Guadalupe Hidalgo to Hopwood
    The Educational Plight and Struggle of Mexican Americans in the Southwest
    Guadalupe San Miguel, Jr., Richard R. Valencia
    Voices Inside Schools - Teacher as Rain Dancer
    Simon Hole
    Book Review - Will Teach for Food edited by Cary Nelson
    Robert P. Engvall

    Book Notes

    The Role of State Departments of Education in Complex School Reform
    By Susan Follett Lusi

    Improving America's Schools
    Edited by Eric A. Hanushek and Dale W. Jorgenson

    Orly's Draw-a-Story

    Locked in the Cabinet
    By Robert B. Reich

    Journeys of Women in Science and Engineering
    By Susan A. Ambrose, Kristin L. Dunkle, Barbara B. Lazarus, Indira Nair, and Deborah A. Harkus

    The Curriculum Studies Reader
    Edited by David J. Flinders and Stephen J. Thornton

    First Person, First Peoples
    By Andrew Garrod and Colleen Larimore

    Historical Dictionary of Women's Education in the United States
    Edited by Linda Eisenmann

    I Don't Want to Talk About It
    By Terrence Real

    Randomized Experiments for Planning and Evaluation
    By Robert R. Boruch