https://www.austinchronicle.com/news/2004-02-06/196189/

The Figures Do Lie

'Statesman' finds headline cops-and-race story – except it just ain't so

By Jordan Smith, February 6, 2004, News

On the public relations front, last week was a bad one for the Austin Police Department. In a four-day series, the Austin American-Statesman reported, stridently – as if Editor Rich Oppel had discovered Jimmy Hoffa's body buried beneath the department's Eighth Street headquarters – that, after a "comprehensive" analysis of police records, including a database with five years' worth of APD "use-of-force" reports, their reporters had found that Austin police used force against Austin's Hispanic and African-American residents "at significantly higher rates than they did against whites." According to the daily, Hispanics are 25% more likely to have force used against them, and African-Americans are 100% more likely to have force used against them, than are whites. These are inflammatory, even shocking, statistics. Unfortunately for the city, the APD, and most especially the Statesman, they appear to be dramatically exaggerated – indeed, almost entirely untrue.

The difference in treatment "defies easy explanation," wrote reporters Andy Alford and Erik Rodriguez. That's because the analytical methodology used by the paper defies logical explanation. To arrive at its conclusions, the paper compared two sets of data that bear little relation to one another, while ignoring the more obvious and accurate approach used by the city itself to analyze its own data. The Statesman team also failed to take the simple precaution of analyzing the use-of-force data against itself. And most importantly, the paper failed to recognize that the data in the use-of-force database is so inconsistent that it is effectively worthless for any broad conclusions.

Apples vs. Oranges vs. APD

According to the Statesman's series, beginning Jan. 25 and titled "Unequal Force: A Four-Day Look at Police Violence," reporters analyzed 6,447 Use of Force Report forms – a form officers sometimes fill out when they use any form of force against a citizen, from activating a pressure point to firing a gun – completed by APD officers from Oct. 31, 1998, to May 11, 2003. Those reports related to 4,701 individual subjects and 4,280 individual incidents. (A single incident can involve multiple officers and subjects, with a report being generated for each.) The paper then compared this data against the total number of police contacts for each year. That latter number includes nearly every situation in which a citizen might encounter police – from flagging an officer down for help on the side of the road to being the subject of a felony arrest warrant.

In matching the two bodies of data against one another, the paper purports to have created a "rate of force – a precise measure of how likely a person who encountered police was to be met with force." Based on this model, the paper concluded that police used force 7.4 times per 1,000 contacts with black residents, who make up about 10% of Austin's population; 4.6 times per 1,000 contacts with Hispanic residents, who make up about 29% of the population; and 3.7 times per 1,000 contacts with white residents, who make up about 54% of the population.

These numbers are doubtlessly worrisome – or would be if they were true. But the comparison behind the daily's "rate of force" is fundamentally flawed. Nearly every Austinite has some occasion, often an innocent one, to make contact with the police during the course of a year. Very few Austinites, of any ethnicity, are the targets of police force, and they are not simply a random subset of the entire population of the city. The Statesman's "rate of force" suggests how likely it is that any random Austinite, under any random circumstance involving police contact, will be roughed up by the cops. This calculation might appeal to a paranoid actuary – or would, "all other things being equal," as the actuaries say. But all other things are not equal; the Statesman rate fails to accurately reflect the real circumstances under which a person is likely to be met with police force.

– And Liars Figure

The APD began recording officer use of force in October 1998, as part of the department's bid for national accreditation from the Commission on Accreditation for Law Enforcement Agencies. The reports are designed to be an internal tracking tool, based on an underlying assumption that citizens are met with force almost exclusively while being arrested – an assumption borne out by the database. In November, in response to a request made under the state's open records laws, the Chronicle received a database of APD's use-of-force reports recorded from September 2000 through June 2003, which contained 5,555 total reports, reflecting 3,660 unique events and 4,154 unique individuals. Of the 5,555 reports, 4,911 – or 88% – related to incidents that ended in an arrest.

The high correlation between use of force and arrests makes comparing those two sets of data a better and more compelling indicator to reflect when, how, and against whom police actually use force. In fact, this is the manner in which the APD calculates its own "rate of force." According to department statistics, from Oct. 31, 1998, to May 11, 2003 – the time period covered by the Statesman's analysis – APD officers made 553,194 total arrests and officers filed 6,447 force reports. Broken down by race and then compared to the totals, whites accounted for 36.4% of all arrests and 35.3% of all use-of-force reports, Hispanics accounted for 37.1% of all arrests and 30.3% of all force reports, and blacks accounted for 26.2% of all arrests and 30.6% of all force reports. These reflect statistically significant differences between the races, but hardly "stunning" and "shocking" disparities. In other words, the likelihood that a suspect facing arrest will be met with force is roughly equal across racial lines.

What that correlation also shows, however, is that relatively speaking, blacks are far more likely to be arrested than are whites – a national phenomenon that prompts serious questions about social equity, which the entire community should address. "[Neither] Austin's police nor any other police agency in the United States can solve the root causes of crime in America," Austin Police Association President Mike Sheffield wrote in a press release responding to the Statesman report, "and until political leaders start adequately addressing issues such as access to education, decent jobs, housing, mental health care specifically and health care generally, there will ... continue to be 'two Americas.'"

The Statesman repeatedly lauded its own analysis of departmental data as "unique" and "comprehensive," while noting that APD Chief Stan Knee said, "[H]e would have analyzed use-of-force rates differently," and that outside experts concluded that the paper's methodology was "rare in its approach." Yet not once in four days did the daily explain how its methodology was different, nor did the reporters address the statistical chasm created by ignoring other data and relying only on their chosen method of analysis.

Unfortunately, thus far city leaders have also taken for granted that the Statesman's conclusions are correct without independently reviewing any of the readily available data. On Jan. 28, the final day of the series, the Statesman reported that as a result of its crack investigative journalism, Mayor Will Wynn is set to take action. "I would be surprised if any of these numbers are refuted – it's just too well done," he told the daily. "This is a classic example of why we have a free press in this country. This series is going to serve the community well, as painful as it might be for some folks."

Sixth Street Crime Wave

The Statesman apparently chose not to delve too deeply into the more than six dozen different variables contained for each record within the use-of-force database. In a large graphic map of the city, the Statesman broke out its rate of "force per 1,000" residents for each police sector in the city. Not surprisingly, the resulting correlation reaffirms their hypothesis: According to the Statesman, in every sector of town, blacks are disproportionately likely to have police force used against them.

But where and how do Austin police actually use force? Apparently, the paper did not care to find out, although the question is easily answered by the database. In the three-year section analyzed by the Chronicle, persons against whom force is used are most likely to be white (34.7%), male (84.6%), and under 30 (45.7%); those people encounter police during a dispatched call for service – meaning after someone has called police for assistance (47.8%); and they encounter police in the Central West and Downtown area commands (37.4%). The Downtown Area Command was created, carved out of the Central West Area Command, in 2001. On Tuesday, Jan. 27, the paper ran an itemized list of the 10 officers who have filed the most use-of-force reports. The reporters listed each officer's name, the number of years on the force, and the total amount of training each has received, and acknowledged that most of the use-of-force incidents took place at night and Downtown. The paper did not, however, note that every one of the 10 officers is assigned to the DTAC – which just happens to patrol Sixth Street.*

Painful (Misleading) Numbers

Even had the Statesman used arrest rates as the basis of its use-of-force figures, and even if it had done a more thorough analysis of what the database actually says, its conclusions would still be fatally flawed – because the use-of-force database is itself so unreliable. Under APD policy, officers are required to complete a use-of-force report when the person against whom force is used complains of pain or injury. "For purposes of reporting, the temporary discomfort associated with the initial arrest procedure does not constitute a complaint of injury," the report reads (emphasis in original). "The determination to fill out a Use of Force Report Form will be based on a consistent and repetitive complaint of pain beyond the initial arrest procedure."

Yet the use-of-force database includes numerous reports made by officers even when no complaint of injury was ever made. (The Statesman inconsistently described the policy for filing – as required in case of a complaint of injury, and also as a duty in every instance.) Of the 5,555 reports reviewed by the Chronicle, 2,166 (39%) were filed even though no complaint of injury had been made. (Many others were filed in response to complaints arising from "discomfort" associated with arrest – for which officers are not required to file reports – and 202 left the complaint of injury field blank.) Once those 2,166 reports are removed, there are 3,389 total reports that include 2,567 individuals and 2,310 incidents. Of those, 92% ended in arrest, 45% of the individuals involved were white, and almost half of the incidents occurred in the Downtown Area Command or CWAC. Those numbers suggest that the presence of the unnecessary reports skews the database, though not by much. But what about the reports that are not there? Who are the officers who use force and – either properly, according to APD policy, or not – choose not to file reports? Who were their subjects? In what ways are they different from the officers, and the subjects, whose reports are in the database? We can never know, because the database is neither a complete set nor a valid sample. APD veterans say younger officers, attempting to document everything, are more likely to overreport incidents of force. Conversely, the database may also be missing reports that should've been filed but that more veteran officers didn't feel were required.

The report form itself amplifies the unreliability of the data. For example, there are no definitions provided for what constitutes a "serious" vs. "minor" injury. That determination is left up to the individual officer, and an "elbow abrasion" might be recorded as serious by one officer and as minor by another. The department is certainly aware of the reporting deficiencies: On Oct. 6 the board members overseeing Austin's federal Local Law Enforcement Block Grant (including First Assistant District Attorney Rosemary Lehmberg, District Judge Jon Wisser, and APD Assistant Chief Rick Coy) approved allocating $263,100 of federal funding, to be matched by $29,233 in city money, to improve the department's Early Warning System, of which the use-of-force database is a part. The EWS is designed to allow the APD to track officer behaviors and identify patterns of conduct that could lead to more serious incidents. Coy said at the time that although parts of the EWS model are in place, the department is aware that the system needs to be improved.

So, months before the Statesman's overreaching and melodramatic exposé, the department was already pursuing ways to better police its own regarding use of force. At a time when tensions between the minority community and the APD are already heightened – most recently in connection with the June 14 shooting death of Jessie Lee Owens by Officer Scott Glasgow – the Statesman took the low road, choosing to throw a little gasoline on the fire. In short, while it was a bad week for APD's rank-and-file, it was a dismal week for accurate and responsible journalism.

For more on this story, see Austin at Large: The Real Race Problem.

*Editor's Note: Corrected from original print edition.