Voices in Education

Bias in the SAT?
Seven years ago the Harvard Educational Review published an article that inspired great controversy, fiery rebuttals, and highly technical debates. What was the big deal? And why does it matter today?

In 2003, HER published Roy Freedle’s “Correcting the SAT’s Ethnic and Social-Class Bias: A Method for Reestimating SAT Scores.” HER is a generalist journal, speaking to issues of consequence in education across many fields and disciplines. Roy Freedle, former senior research psychologist at the Educational Testing Service, investigated the fairness of the SAT using a special psychometric technique and demonstrated the presence of a relationship between the difficulty of a test item and differential performance on this item across different racial and ethnic groups. While we knew that the technical nature of this topic might prove to be a barrier, we were convinced that the article spoke to wider concerns: that standardized testing has a long history of racial and ethnic bias in the United States, that there needs to be a theory to explain this purported bias, and that that there are ways to reduce this bias. Acknowledging, identifying, understanding, and reducing bias become critical for any society that uses test scores as a measure of learning, achievement, and success.

HER’s editors were not surprised when Freedle’s findings and methods were questioned; but they were surprised by the boisterousness of the response. Even after publishing a dissenting piece by Neil Dorans, from the Educational Testing Service, in 2004 titled “Freedle’s Table 2: Fact or Fiction,” the controversy continued. The journal itself was criticized for publishing such contentious scholarship.

Seven years later, a manuscript arrived in our mailbox. This new article, by Maria Veronica Santelices and Mark Wilson, used fresh data to reexamine evidence of the relationship posited by Roy Freedle between ethnic and racial bias and the difficulty of SAT questions. Moreover, they directly addressed the heaviest criticisms of Freedle’s study. In their 2010 article “Unfair Treatment: The Case of Freedle, the SAT, and the Standardization Approach to Differential Item Functioning,” Santelices and Wilson confirmed that there was a correlation between item difficulty and performance on that item across racial and ethnic groups. HER’s editorial board considered the history of this debate, the potential for another round of criticism and controversy, and the importance of publishing a piece that would urge readers to consider the consequences of a systematic relationship between ethnic and racial bias and question difficulty on a test so widely accepted and used. Once again, with the publication of this article, HER generated controversy.

This Fall issue of HER contains responses to Santelices and Wilson’s article. We asked Neil Dorans to craft a response. We asked Roy Freedle to comment on this study. And we solicited a final word from Santelices and Wilson. As expected, there is little agreement.

Why is this topic so contentious? We believe that aside from the technical nuances of the method, there are important questions at stake. How can we build a theoretical understanding of what causes bias on test items and work to correct it? How can the field of measurement support these efforts and why should they? How do researchers test their assumptions and the relationship of these assumptions to their findings? What roles should educational researchers play in striving to achieve fairness in standardized testing that has historically worked to the disadvantage of certain groups? And what can parents and students—the consumers of the SAT—take from this conversation to gain access into higher education?

The SAT is an iconic test, accepted by many without question as a benchmark for the college-bound. Perhaps the research on this test will promote widespread critical reflection about the content and role of testing in our society. We invite you to read these articles and responses and to enter the conversation. Santelices and Wilson, Freedle, and Dorans provide us with an opportunity to enter a serious and important debate.

The debate that we hope to inspire here will undoubtedly have political dimensions, and we recognize the role of politics in social science research. We also believe, however, that the goal of research on issues of critical concern to education is to transcend ideology. Empiricism, reason, and openness to continued investigation and questioning should inform the decisions we make as a society.

We look forward to hearing your comments.

About the Author: