Most followers of education reform can remember their typically quiet summers being interrupted last August by a front-page story in the New York Times. The story suggested that charter schools, heralded by many as havens of free choice and innovative practice, might actually be doing worse than traditionally organized public schools. Specifically, the Times and other media outlets around the country were reporting on a study by the American Federation of Teachers (AFT) that compared the performance of students in charter schools with those in regular public schools on the National Assessment of Educational Progress (NAEP) reading and mathematics tests. The AFT's analysis found that fourth-grade students in charter schools performed about half a grade behind those in regular public schools, and that these differences held true even for low-income students and students in central cities.

The study provoked a rash of point/counterpoint exchanges among charter school advocates and opponents. Amy Stuart Wells, a professor of sociology and education at Teachers College, Columbia University, told the Harvard Education Letter that the report supported her long-held contention that "there is no evidence that market forces will improve education." Wells added, "It's time for us to say we have a naked emperor and move on."

But Caroline Hoxby, a professor of economics at Harvard University, was one of many researchers who had a different take on the AFT's results and the extensive media coverage they received. "Why is the worst study I have ever seen on charter schools on the front page?" Hoxby asked. She criticized the AFT study primarily on methodological grounds, saying that the proportion of charter students in the NAEP sample is too small to yield valid conclusions about performance and that the study failed to compare schools with similar student characteristics.

Beyond the AFT's recent analysis of NAEP scores, a small body of other research has analyzed whether students in charter schools perform better or worse on average than those in traditional public schools. Adding confusion to what is already an inconclusive picture, some studies show a slight advantage for regular public schools and others show that charter schools perform better. Hoxby's own study, released shortly after the AFT report, found that students in charter schools are more likely to be proficient in both reading and math than their counterparts in public schools with similar racial demographics.

Yet Gary J. Miron, principal research associate for the Evaluation Center at Western Michigan University, says that looking at the average performance of charter schools and their students tells only a small part of the story. "[On average,] charter schools are doing similar to, or slightly worse than, regular public schools," notes Miron, who has studied the work of charters in 12 states. "But there are large differences within and across states." And it is in looking at these differences from state to state and from school to school, Miron says, where educators and policymakers can learn the most important lessons from the charter school experiment.

Keeping Troubled Schools Open

Charter schools have spread rapidly and are now among the most widely adopted types of school reform in the U.S. The first charter school was started in 1991 in Minnesota; there are now 2,996 charter schools in 38 states and the District of Columbia, enrolling nearly 800,000 students. The original idea behind charter schools was to open up new options for parents and students and to allow educators, private organizations, and others the flexibility to operate innovative programs by freeing the schools from many district and state regulations. At the same time, the schools would be held accountable for results: since parents had to choose to enroll children in these schools, the charter schools would have to succeed in order to attract and retain students. They would also have to meet performance targets, or they could lose their charter or be shut down.

Yet despite the strong rhetoric about accountability, one factor that may be keeping the average level of achievement in charters down is that many weak schools are allowed to continue operating, while some states have placed caps that limit the growth of successful charters, says Robert Cane, director of Friends of Choice in Urban Schools (FOCUS), a group that supports charter schools in Washington, D.C. "Not enough schools have closed down, and not enough have been chartered in the first place," says Cane.

Of the roughly 3,400 charter schools that have opened in the U.S. since 1991, only about 9 percent have closed, according to the Center for Education Reform, a Washington, D.C.-based organization that advocates for charters and other forms of school choice.

One reason low-performing charters have remained open is that many states lack good information on student performance and have a difficult time demonstrating that a school is doing poorly. "It's hard to prove a case [for closing a school] when it comes to student performance," says Miron. "You may go into a school and see an absolute catastrophe, and see that the results are poor, yet you can't capture those data over time." As a result, he notes, most of the charters that have closed have done so because of financial mismanagement rather than poor academic performance—although many mismanaged schools are also low performing.

Another reason low-performing charters continue to exist is that parents like the schools for reasons that may have little do with academic performance. In many cases, notes Cane of FOCUS, parents will support a charter school because they consider it safer than the regular public school their children left, even if the charter is not helping their children learn at higher levels.

But Miron points out that some states, notably Connecticut, have been aggressive in closing charter schools that aren't showing results, and those states are seeing relatively high performance in the charters that remain. In Connecticut, a third of the 19 charters that started since the state's charter school policy took effect in 1997 have shut down, and Miron's study found that student performance on state tests improved more rapidly in the charters that remained than in regular public schools. Miron says that Connecticut has been able to move quickly to shut down low-performing schools because charter schools have bipartisan support and there is consensus on the criteria for success. In other states, where charter schools have been more contentious, advocates tend to "circle the wagons" and defend schools that are not doing well.

Why Some Charters Struggle with Achievement

Some preliminary research also suggests other possible reasons why charter schools as a whole may perform at lower levels than regular public schools. Barnett Berry, president of the Southeast Center for Teaching Quality, says lower teacher qualifications may help to explain the low achievement numbers we're seeing from many charter schools. Only 38.5 percent of mathematics teachers in charter schools have a college major or minor in mathematics, compared with 51 percent of public school teachers, according to the federal Schools and Staffing Survey. And charter school teachers are more than twice as likely as their regular public school counterparts to have five or fewer years of experience. Berry is proposing a study to examine the link between teacher qualifications and performance in charter schools. "I have a hunch [teacher quality] may explain some of the student achievement gaps that are clearly at play in the charter school world," he says.

Another factor that may contribute to lower achievement in some charter schools is, perhaps ironically, parental choice. Although choice is one of the factors that make charter schools appealing to many parents, too much student movement may inadvertently depress overall performance, suggests Helen F. Ladd, associate director of the Terry Sanford Institute of Public Policy at Duke University. A study she and Robert Bifulco conducted of charter schools in North Carolina found that students in charter schools performed at lower levels than they would have in regular public schools (based on their performance when they attended regular public schools), and they gained in performance at slower rates. In addition, Ladd and Bifulco found that charter school students were twice as likely as those in regular public schools to move from school to school.

"More choice is good, but it also leads to instability in the education process," Ladd says. "It's a problem for the student moving, and also for other students in the classroom."

On this last point, however, Miron suggests that student mobility may be a result of poor performance, not a cause of it. "Poorly performing schools have high rates of attrition," he says. "Is it because the schools are poor, or are the schools poor because they have high rates of ­attrition?"

In addition, advocates of charter schools point out that many charters appear low performing because they take in students who are far behind their grade level academically. While tests like the NAEP show that students in these schools are performing at low levels, the school may be helping them in ways that do not show up in test scores. "We are not making excuses for them, but it is important to realize that we have several schools that are dealing with kids in the juvenile justice system who are recovering dropouts," says Cane. "What is success for those schools? Suppose a [former] dropout gets to go to school four days a week. Is that success or failure? We haven't decided how to evaluate that yet."

But Bella Rosenberg, an assistant to the president of the AFT and a coauthor of that union's study of charter school performance, says regular public schools also deal with students who are far behind grade level. She suggests that educating students who come to school with severe disadvantages is a complex problem and that deregulation alone is not the answer. "The whole argument that, if we just deregulate, achievement will soar is at best theoretical, and more likely ideological," she says. "Education just doesn't work that way—and I'm not saying this as a fan of overregulation."

Learning from Charters That Succeed

While the overall performance of charter schools may be similar to or perhaps even below that of other public schools, many charters are achieving at high levels. What factors account for their success and what can we learn from them?

The evidence from national- and state-level studies of charter schools, along with testimony from school leaders and advocates, is limited, but it suggests that the most effective charter schools share some important characteristics. These studies confirm the importance of factors already identified as critical for the success of all schools (see sidebar "Four Traits of Successful Charter Schools"). Four Traits of Successful Charter Schools

Research on charter schools has pointed to a number of success factors that also apply in other educational settings. These include:

1) Frequent and thoughtful student assessment
2) Use of data (from assessments and other
sources) to plan programs and inform change
3) Effective and stable leadership
4) Staff who share a sense of the school’s mission

For one thing, charter school leaders note that effective charters use data wisely to plan their programs and make adjustments when the data suggest that changes are needed. Many charters assess students regularly and use data on student performance to change instructional practices during the school year. The schools operated by Edison Schools, Inc., for example, administer web-based benchmark assessments throughout the year to provide teachers with early indications of students' strengths and weaknesses.

"The good [charter schools] are very nimble," says Cane. "They keep track of students and make changes as needed."

The quality and stability of school leadership also contributes to charter school success, as it does for regular public schools. A 2002 study by Tom Loveless, director of the Brown Center on Education Policy at the Brookings Institution, found that charter schools operated by educational management organizations, often for-profit firms, showed achievement gains that were significantly larger than those for other charter schools and regular public schools. Regardless of whether or not for-profit firms and public education are a good match, Loveless says his study provides further evidence that strong and skillful leadership is crucial to a school's success—a finding that confirms a growing body of evidence also emerging from traditionally run public schools.

"The findings raise doubts about a strain of thought in the charter school movement: that anyone can successfully start and operate a school if he or she possesses abundant energy and a love of children," Loveless writes. "In the extreme, this form of romanticized amateurism dismisses the importance of educational expertise. That appears unwise."

Effective charter schools with strong leaders also attract capable staff members who share a sense of the school's mission, another factor that research on schools generally suggests contributes to success. By contrast, national- and state-level studies show that charter schools that do not do well have high turnover of leadership and staff. "If you see schools with high turnover of board members and directors, you see schools that are not going to do well," says Miron.

While factors like leadership and the effective use of data may suggest some attributes of successful charter schools, many researchers and practitioners caution that we still do not know enough to make definitive judgments about what makes some charter schools succeed and others fail—or what factors may boost or limit their success over time. The movement remains relatively young, and more revealing results on the quality and performance of charter schools may have to wait until it matures.

Robert Rothman is a principal associate at the Annenberg Institute for School Reform at Brown University and the editor of Voices in Urban Education.