The following article originally appeared in The Harvard Education Letter (volume 25, number 6). Copyright 2009 President and Fellows of Harvard College. All rights reserved.

In recent years, a raft of research has called attention to the importance of effective teaching in influencing student achievement. Yet federal and state accountability policies continue to focus primarily at the school level: using schools as the unit of performance, identifying “failing schools,” and more recently targeting “turnaround schools” for special intervention. One of the best-kept secrets in educational research, it seems, is the fact that differences in the quality of instruction from classroom to classroom within schools are greater than differences in instructional quality between schools. This finding has been documented in a variety of studies, most of which used indirect measures to evaluate instruction (such as relying on teachers’ perceptions or looking at curriculum materials to determine how much time they spent on particular topics). Despite the limitations of these measures, these studies have suggested that there is considerable variation in practice even among teachers in the same building.

Over the past five years, however, researchers led by Brian Rowan, the Burke A. Hinsdale Collegiate Professor in Education at the University of Michigan, have asked teachers in 112 schools to keep detailed logs of their actual practice. The newly released results of the Study of Instructional Improvement (SII) document dramatic differences in the kinds of skills and content taught from classroom to classroom. For instance, the study showed that a fifth-grade teacher might teach reading comprehension anywhere from 52 days a year to as many as 140 days a year. Similarly, first-grade teachers spent as little as 15 percent to as much as 80 percent of their time on word analysis. Thus, the study found, students in some classrooms may spend the majority of their classroom time on relatively low-level content and skills, while their peers in the class next door are spending much more time on higher-level content.

Even more striking, the majority of schools in the study had implemented comprehensive school-reform models, some of which were fairly prescriptive in the materials they used and the instructional designs they required teachers to follow. Despite these efforts to guide and improve instructional practice, the researchers found, ­dramatic ­variations persist within schools.

“In general, [the SII] reinforces the notion that what matters most is the teacher you get in school, not the school you go to,” says one of the study’s authors, Richard J. Correnti, an assistant professor and research scientist at the Learning Research and Development Center at the University of Pittsburgh.

Topics, Materials, and Teaching Methods

The SII is one of the most extensive examinations yet conducted into classroom practice. It examined 86 schools using one of three comprehensive school-reform models—Accelerated Schools, America’s Choice, and Success for All—along with 26 comparison schools. The researchers followed two cohorts of students as they progressed through the primary grades or the upper elementary grades and asked teachers to compile daily logs indicating the amount of time spent on reading and language arts, the degree of emphasis on a particular language arts topic, and the content and methods used in instruction.

The data, released this year, confirmed that the overwhelming majority of the variation occurred within schools, rather than between schools, and that the variations were substantial. Teachers varied widely on the topics they chose to spend time on, such as word analysis, comprehension, or writing; on the materials they used; and on the types of instructional practices they employed. For example, some teachers were more likely than others to use texts with controlled language, while others were more likely to use chapter books. When teaching word analysis, for instance, some teachers used letters, sounds, or pictures, while others used literature-based texts. And when teaching writing, some teachers tended to lead the class, while others spent more time commenting on student writing.

Rowan cautions that some of the variability may reflect measurement error. But he notes that the amount of “true” variation remains large. Moreover, the study found little evidence that the variability reflected teachers’ responses to student needs. Students’ socioeconomic background or prior achievement levels did not predict the likelihood that teachers would emphasize a particular topic.

“The first response is, ‘That’s good: teachers should be varying their instructional activity based on the kids,’” Rowan says. “[But] there isn’t a lot of that going on.”

A Lack of Clear Standards

Other researchers not involved with the study say the results confirm their own observational research. Katherine K. Merseth, director of the teacher education program at the Harvard Graduate School of Education, recently conducted a study of successful charter schools in Massachusetts that included detailed observations of 90 classrooms. She found that the level of cognitive demand varied widely from classroom to classroom. “In every other aspect, these schools were astonishingly coherent,” she says. “But they were loose inside the ‘black box’ of instruction.”

Merseth says that much of the reason for these wide differences in teaching practice is the lack of clear standards for instruction. Without guidelines for effective practice, she notes, teachers have wide latitude to try what they believe is effective. “We can’t agree on what good teaching looks like,” she says.

State accountability systems do not eliminate variations in practice because they generally set relatively low targets for instruction, Merseth adds. As a result, students in schools like the charters she studied can perform well on state tests, whether their teachers focused on the relatively low-level skills measured on the tests or on higher-level skills.

Even the curriculum and instructional guidance spelled out in comprehensive school-reform models do not reduce variance, as the SII shows. Instead, Rowan says, they end up setting a floor to ensure that all teachers teach at least some important language arts topics. “Success for All gets rid of malpractice, in which teachers aren’t teaching at all,” he says.

Robert C. Pianta, dean of the Curry School of Education at the University of Virginia, who has studied teachers’ practice and was not involved in the SII, notes that the prescriptions in comprehensive models can be interpreted in many different ways by schools and by individual teachers. “There’s always some slippage at every level,” he says. “Districts, schools, and teachers are allowed to adapt the programs or models.”

The result of this variation in practice is a wide variation in students’ opportunity to learn important content, says Robert E. Floden, University Distinguished Professor at the Michigan State University College of Education. This is particularly true in mathematics, he says. Many elementary teachers have little background in mathematics or dislike the subject, he notes, and may simply elect not to teach the kinds of higher-level concepts that would improve students’ mathematical understanding. “If teachers don’t spend time on those, where do kids get it?” Floden says. Yet while avoiding higher-level topics may limit students’ opportunity to learn, another analysis of the SII found that the inclusion of such topics does not always translate into higher student performance (see sidebar "Teacher Practice and Student Performance").

Teacher Practice and Student Performance

Does more attention to higher-level content and instruction mean higher levels of student performance? Not necessarily.

A separate analysis of data from the Study of Instructional Improvement, by Jenny DeMonte of the University of Michigan, compared teachers’ instructional practices with their students’ scores on standardized tests. She arrayed the practices into three levels. The highest level included practices such as examining literary techniques and sharing writing with others. Medium-level practices included techniques such as explaining how to find answers, and low-level practices included asking questions that have answers in the text and summarizing story details. DeMonte then grouped teachers into categories of practice—high writing/high reading, medium writing/medium reading, medium writing/low reading, and low writing/low reading—and tracked student achievement growth for each group.

Perhaps surprisingly, the growth pattern for the students in the high writing/high reading classrooms was about the same as that for students in the low writing/low reading classrooms. Those in the medium writing/medium reading classrooms showed the greatest rate of growth in achievement.

A further analysis suggests a possible reason for those results. DeMonte found that the teachers in the high writing/high reading group taught an average of 13.3 instructional items per lesson, which totaled about 108 minutes. Teachers in the medium writing/medium reading group taught an average of 6.6 items in 102 minutes. Thus, the teachers using the medium-level practices spent twice as much time on each practice as those using the more demanding practices did.

Those teachers might have employed more challenging instruction, but their instruction might have been a “mile wide and an inch deep,” resulting in smaller student growth than for teachers who used less challenging practices but spent more time on them.

Down to the Core

Many researchers are troubled by the lack of clear, agreed-on standards for practice. They argue that schools and districts need to do more to guide and support instruction so that all students have access to high-quality teaching. But some teachers caution that the cure might be worse than the disease. Requiring teachers to follow the same practice will not help improve learning if the practice is ineffective, says Mark Simon, a former teachers’ union president who is now the co-coordinator of the Tom Mooney Institute for Teacher & Union Leadership.

“If eliminating variation in teaching practice means all kids learn to write using a five-paragraph format, you might get a higher ‘lowest common denominator,’ but that’s not leading to good writing instruction,” he says. Moreover, he points out, “good teaching is not done one way.” Rather than imposing prescribed practices, he says, schools should support teachers so that all are capable of making informed judgments about best practices.

While few educators argue for eliminating variability in teacher practice altogether, many suggest that the profession needs to do more to make sure all teachers teach at least the core curriculum that all students should learn.

“I’m not saying we mandate scripted lessons,” says Pianta. “But we ought to have strong, assertive statements about what is the core instructional program, and we ought to monitor it.” In his research, Pianta has identified a set of instructional practices that are associated with improved student achievement, and he works with teachers in grades preK–5 to measure their practice against those guidelines (see "Neither Art nor Accident," HEL, January/February 2008). This approach still “[leaves] room for teachers to take advantage of a tornado warning that day and teach kids about tornadoes,” Pianta says.

Such an effort should include strong curriculum materials, along with professional development to help teachers make use of the materials, Floden says. “If some of the variability is due to teachers’ not feeling [that they have]mastery of the content, materials that do some of the work can be a big help,” he says.

Schools should also do more to enable teachers to observe one another’s classrooms to learn and practice effective instruction, adds Richard F. Elmore, the Gregory R. Anrig Professor of Educational Leadership at the Harvard Graduate School of Education. “There is a deep-seated culture of isolation in the classroom,” he says.

Ultimately, the goal ought to be to raise the level of instruction across the board, suggests Elmore. “The kind of variability that’s good is variability from a high bar,” he says. “The problem is, the base is too low.”