Uncontrolled Experiments

AISD Superintendent Meria Carstarphen (Photo by John Anderson)

Data, facts, research.

Data, facts, research.

The mantra of the Austin Independent School District under Superintendent Meria Carstarphen has been simple: Look at the data, examine the facts, study the available research, then let the results drive policy. It is supposedly the data that Carstarphen has collected that has driven her push for two radical new proposals. First, she recommends turning the Eastside Memorial Vertical Team over to IDEA Public Schools – a South Texas-based charter group – beginning in 2012. Second, at some future date, she wants to turn Pearce and Garcia middle schools into single-sex academies.

In making her proposals, Carstarphen has presented research about the supposed successes of IDEA's schools in the Rio Grande Valley and told the board that those results can be replicated in East Austin. At the same time, she has argued that the record of AISD's Ann Richards School for Young Women Leaders shows that single-sex education works. But there are two major, recent reports, each published by respected figures within Austin's public education research community, which raise a serious question: Are those results real and replicable among all students, or are those high-performing campuses only getting their results by recruiting the best and most committed students?

The Big IDEA

Charter schools and single-sex academies have become politically fashionable in Austin and elsewhere. But so has abstinence-only sex education and, as Texas' teen pregnancy rates show, political fashion does not always produce successful policy. So when AISD administrators want to make the case for radical proposals like the IDEA charter, they need accurate research. When it comes to number-crunching about public education, they don't come much more respected than professor Edward J. Fuller. After a decade as one of the University of Texas' foremost education researchers, Fuller is now executive director of Penn State's Center for Education Evaluation and Policy Analysis. When the Texas Business & Educa­tion Coalition was looking for an independent study of charter school data, it turned to Fuller; midway through that research, Texas American Federation of Teach­ers secretary-treasurer and TBEC coordinating committee member Louis Malfaro asked him about data specifically relating to IDEA's track record.

The resulting preliminary report, titled "Is IDEA a Good Idea for Austin ISD?: Analysis of IDEA Charter Schools," reflected three basic findings: One, that IDEA does not enroll "underserved students" by any standard definition of "underserved"; rather, kids who pass the Texas Assessment of Knowledge and Skills (state accountability) tests in math and reading were twice as likely as their failing peers to be enrolled. Two, while IDEA's claim that 100% of its graduates go on to higher education might be accurate, only 65% of students in IDEA's ninth-grade cohort actually make it to graduation. Three, IDEA tends to shed lower-scoring students prior to graduation – that means that a cohort's average scores rise over time in part because the bottom of the class is falling away. Additionally, IDEA's high schools lose – either by dropping out or transferring to regular public schools – as many as or more students than any regular campus in the three South Texas counties IDEA currently serves. While the kids that make it into IDEA and stay in its schools do improve their grades, Fuller said, the charter group is "not enrolling kids who are struggling academically and helping them to achieve. They're more likely to enroll kids who are already doing quite well." Overall, he said, "Whether intentionally or not, they skim at the top and they push out at the end."

Ann Richards School principal Jeanne Goka (Photo by Jana Birchum)

Despite Carstarphen's often reiterated commitment to research, Fuller's warnings did not deter her from adding the IDEA charter to her academic wish list for the 2012-13 school year (see "AISD Administration Proposes Changes for 2012"). Although she concedes that charters can "in no shape, way, or form ever do as well as [AISD] in the aggregate," she still called IDEA's work "extraordinary and in some areas better than AISD." She told the Chronicle that the district currently has lost 4,000 kids to existing charters, and they take with them $26 million in state funds every year. "For the Eastside Memorial area, it looks like it's about $6.6 million. So to have an internal option, where we're not losing the [average daily attendance state funding] and we're not losing the children and the family to an outside school that's not part of the AISD family, I think is a legitimate academic design."

Fuller has a simple response to that analysis. If that is true, then why doesn't IDEA just take over the entire Eastside attendance zone system, rather than just admit a few hundred kids annually? "They won't do that," he said. The Knowledge Is Power Pro­gram chain of charter schools "tried it and failed miserably and actually made a policy decision to never do it again." In its first attempt to run a regular general-admission campus, KIPP took over Cole Middle School in Denver Public Schools in 2004; three years later it pulled out amid poor test scores, saying it could not find a principal. Fuller says history is repeating itself in KIPP's schools in Houston's North Forest ISD. "They're terrible. They're some of the worst-performing schools in the state," he said, and the Texas Education Agency seems to agree with him. In June, Com­mis­sioner of Education Robert Scott ruled that North Forest be dissolved in 2012. As for Car­star­phen's claim that an in-district charter would bring high-achieving students flooding back to the district, Fuller's preliminary data reflects instead that the high-achieving kids are already staying. Rather, Fuller says, it is the kids with falling grades who get pulled out by their parents and moved to another district or a private charter.

Fuller said that neither AISD nor IDEA has contacted him to discuss his findings. Nevertheless, Carstarphen didn't hesitate to attack his research techniques. She told the Chronicle: "All I have to say is like, I think it's really important when you do research ... it's essential that you tell the whole story. Treat it like a [scholarly] lit review: Have a peer review. Show all the years, not just select years. Don't pick years with small cell sizes. This is like basic research methodology that, while some things can be challenged, at the end of the day there's a bottom line about IDEA. It's a high-performing school system."

That's big talk, responded Fuller, for someone with no extant research record. "I searched for journal articles from her – nothing. I searched on the Internet for any article she's ever written – nothing." He points to her state of the district report: "She claims that she's a statistician as well, and she knows research methodology. They calculated the achievement gap wrong, and they implied that student growth can be measured by 'percent passing,' which it cannot. Both of them are fundamental issues that you learn in your first quantitative research class. As superintendent I can guarantee you she read through that report, and she didn't catch it."

This is not just a game of numbers for Fuller. An AISD graduate himself, he was a student at Lanier in the early 1980s, when it was one of AISD's highest performing schools. Two years after he graduated, the district opened the science academy there. The change immediately pulled the highest performing students out of the main student body, Fuller said; overall grades there cratered, and the school collapsed. When you skim the best and most committed students and families, he said, "You're taking parental involvement out of the school, you're taking kids who are more likely to be leaders and good role models out of the school, and what do you have left?" As for Carstarphen's criticism of his work, he said his methodology had been reviewed and approved by colleagues at UT and Rutgers. The sample sizes are small, he argued, because IDEA does not enroll many kids; in 10 years, the program has graduated a little more than 300 students (IDEA's website reports 314). He found it baffling that the district would not want to study those numbers. "If we're going to make a decision about choosing someone to come in and help with schools," he said, "we should have the full set of information about who they're successful with and who they're not successful with."

Single-Sex & Magnetism

If Carstarphen gets her way on IDEA – formally proposed to the board of trustees last Friday for a Dec. 12 vote – then parents in the Eastside Memorial Vertical Team will be contending with the charter system as early as August 2012. However, families and staff at Pearce and Garcia have been given a reprieve for at least a year as Carstarphen says she wants more time to refine her pending proposal to turn the two Eastside middle schools into single-sex academies. The delay also means more time to sell the idea to the community, and so far the district's big public relations push has been based upon selling the success of the Ann Richards School for Young Women Leaders.

No one doubts the basic academic success of the district's only girls' school. The district consistently touts its "value-added" data, indicating how much student scores improve for the girls it serves in grades six through 10. In 2011, it was the third-highest-ranked AISD campus for net growth in reading scores and sixth for math. But in a recent study, UT professor of psychology Rebecca Bigler raised a core question: Can the academy really raise scores for all students, or is it instead acting as a magnet school, enrolling the best girls from across the district and then providing the resources they need to excel?

UT professor of psychology Rebecca Bigler (Photo by John Anderson)

That central question was addressed in Bigler's 2011 study, "The Efficacy of Single-Sex Education: Testing for Selection and Peer Quality Effects." First she compared the scores of Ann Richards' students with the scores of girls who applied but did not get accepted. "When you just look at the end of sixth grade, the girls at Ann Richards were doing better," she said, "so it looks like, on that first pass, that Ann Richards is producing higher grades." However, that raised a corollary question: What were the students' scores before they applied? Bigler looked at the fifth grade scores for the same kids. She said, "When you compare the girls who got in with the ones who didn't, the Ann Richards girls are doing better already at the beginning of fifth grade. So that suggests that there was a selection bias, and that has to be a selection bias on the part of the school, not the girls – because all the girls had applied."

Ann Richards Principal Jeanne Goka is very sensitive to any criticism of her admissions process. The school's website boasts, "You don't have to be a straight-A honor student to apply, but you do have to be a student who enjoys learning, has the desire to succeed, wants to go to college, and cares about the future of our world." Admission is via lottery, with 75% of desks reserved for kids from Title I (at-risk, low-income) campuses. However, not every student who applies is entered into the lottery.

As part of the original grant agreement, applicants have to be C students or better with clean attendance and discipline records. Each potential student has to submit an application form; a handwritten, two-page application letter; last year's TAKS scores; a report card complete with attendance and tardy records; and two teacher recommendations. That's a lot of hoops that students (and their families) must jump through, but Goka insists that doesn't mean hers is a magnet school. Each application component is given up to four points, so "a perfect score is 20," Goka said. "But because I want that diverse learning population ... instead of creaming like the magnet schools do ... we're going to take 17 and 18 and 19 and 20 scores." She says she has placed girls with scores as low as 14 in the lottery, "so we're able to take in girls who didn't do very well in the TAKS scores but had perfect attendance."

However flexible and well-intended, in the end, Ann Richards uses a closed lottery. One senior policymaker, speaking off the record, said that Bigler's analysis has merely confirmed what had long been presumed – that Ann Richards was being run as an extremely effective magnet. That's good as far as it goes – but it immediately raises a serious question about whether the results at Ann Richards, which prescreens its applicants, can be replicated in any proposed East Austin single-sex academies. Unlike Goka, their principals would not be able to pick and choose their students.

The seemingly scholarly dispute has also taken on a personal angle. Goka has accused Bigler of promoting her own agenda and cites her position as executive director of the American Council for CoEdu­ca­tional Schooling, an academic group based at Arizona State University, as a sign of bias. However, Bigler said that it was Goka who had initially approached her about doing research on issues like gender stereotyping, student aspirations, and bullying in her school. The pair met a month ago in the UT provost's office to discuss this latest report on grades, and Bigler said Goka "was upset because there was some sense that she understood that we had entered into some kind of 'win-win' relation, a relationship where we help you and you help us." Bigler firmly rejected that notion, saying, "I never do research under those circumstances."

Not only Goka has pushed back against Bigler's research, but so has the whole AISD press machine. On Oct. 3, UT issued a press release about the publication of a second paper on single-sex education, "The Pseudoscience of Single-Sex Schooling," co-authored by Bigler and published in Science magazine. Three days later, AISD struck back with "The Facts About Single-Sex Schools" and launched a Web page criticizing Bigler's paper. Again, the success of Ann Richards was used to bolster the argument that "a single-gender school [is] a good idea for Austin."

Bigler said she's surprised that the district is so determined to attack her work when the administration could just use countering research. "There are 2,000 studies. Some proportion of them find good outcomes for single-sex schools, some proportion find good outcomes for coed schools, and a whole bunch of them find no difference whatsoever," she said. If the literature comes to any sort of consensus, she said, it is that single-sex schooling "isn't good for most people, but some people benefit. But what's noticeable about that is that there is no test or agreed-upon way to determine who those 'some people' are." So what happens to the kids Goka rejects? Bigler found that successful applicants did better than girls with comparable scores who stayed in their schools. She said, "So now the question is: Is that because it's a single-sex school or because that girl has gone to school with peers who were, overall, higher-scoring?"

Fuller suggests that inflated scores at self-selecting schools may well be due to "really strong peer effects. ... The kids around you affect how well you do." He argued that KIPP, IDEA, and other charter school groups boost their numbers because "they tend to bring in kids that are average to above-average, and they push out the lower-performing ones." Unlike regular public schools, which can only remove kids for severe disciplinary or behavioral issues, many charter schools require kids and their parents to sign achievement and attendance contracts: Fail to comply, and the students are sent back to their original public schools. He asked: "Is that how they're going to run things in Austin? You're just going to kick them out of their school and send them back to another school? I don't think anyone's been clear on that."

If the board of trustees gives Carstarphen what she wants, Fuller added: "She'll do this, throw this on the East Austin community, and then she'll leave town. She won't have to live through it. East Austin will, forever."