APD's Early Warning System for Reckless and Aggressive Officers Is Broken. Is It Worth Fixing?

Relations between the Austin Police Department and the people it serves are strained by APD's long history of using excessive force against Austinites of color, including at least 20 fatal shootings since 1990. To begin to repair the breaches of trust and fulfill the city's commitment to "reimagine public safety," it would be useful to have a system to identify which officers need help learning to use force properly. And there is one. It's called GAP – the Guidance Advisory Program. And it's broken.

A July report from the Office of the City Auditor states, and APD agrees, that GAP doesn't accurately identify officers who may be more reckless or aggressive than their peers. Those that are identified have not been connected with the counseling and retraining necessary to improve their performance, and their supervisors have not followed up. Originally conceived as a data-driven, nondisciplinary, early warning system, GAP is now seen internally as a burden rather than a useful tool, and externally as adding no value to police oversight. The question the city faces now: Fix it, or abandon it as a lost cause?

“Effective” Data Tracking?

GAP is what law enforcement agencies call an "early intervention system," designed to monitor performance data for individual officers and alert the chain of command when the information suggests trouble lies ahead. This could include how often an officer uses force against suspects, or fires a weapon, or is the subject of a citizen complaint, or many other things. When officers are "activated" in the system, supervisors examine the officers' work and decide if they need counseling or training or other support, create action plans for improvement, and make sure those plans are completed by the officers.

Austin's GAP, created in 2006, is quite basic. Systems in other cities track as many as 20 data points. When created, GAP tracked six, but since 2010 it has tracked only three: uses of force, Internal Affairs complaints, and hours of sick leave used. An officer "activates" in GAP if, over the course of a year, he uses force six or more times (nine or more if working Downtown); receives two Level 1 Internal Affairs complaints (or four of the less-serious Level 2 complaints); or takes more than 160 hours of sick leave.

Early intervention systems are promoted as a best practice, including by the U.S. Department of Justice, to help improve the performance of police departments and respond to issues that drive calls for reform. Council Member Alison Alter is one of many leaders at City Hall and APD who want to see GAP or a system like it used to track what we know about our police force. "We need modernized, data-driven systems across the city," Alter said. "Without some form of an effective data-tracking system, each officer-involved incident is evaluated in a vacuum, and any larger issues or problematic patterns are missed."

The key word there is "effective," and it's not a consensus position that early intervention systems like GAP are effective, even among data-driven justice advocates. In Austin, those include Scott Henson and Kathy Mitchell of Just Liberty, who consider the systems to be little more than public ­relations smoke screens that allow police departments to control what outside observers know about their problem officers and thus shield them from consequences.

"None of them work," Henson said of the systems, "and if you want to know which officers need retraining or closer supervision, Farah Muscadin" – director of Austin's Office of Police Oversight – "can give you a list. Go look at the level of detail that they're using in monitoring complaints, and the outcomes of disciplinary processes, and all that. They know better than any police manager or supervisor which officers need more attention."

Doing the GAP Math

The city auditor's report concurs with Henson that GAP, in its current form, does not work. Over the past five years, the system, which is run quarterly, has on average activated 429 times each year; most of these come when officers cross the thresholds for uses of force (or as APD calls them, "Response-to-Resistance incidents"). In fiscal year 2020, the audit team found that GAP missed about one-third of the officers it should have identified – again, mostly because they'd exceeded the use-of-force thresholds.

Over the five years of data reviewed by the audit team, 56% of officers who activated in GAP did so only once – out of a total of about 1,700 officers, plus another 400 or so who've resigned or retired over that time. So some portion of the remaining officers – hopefully the larger portion – had no GAP activations, and the rest had many. Given the flaws in the system, it's "difficult to determine if there are officers who are activating more often than others," the report reads. Nonetheless, looking just at the 2020 numbers, it continues: "Our analysis found that 7% of officers account for 17% of activations with an average of five activations per officer. Also, 11 officers had six or more activations during this time."

This would seem to be the sort of information that GAP is designed to convert into action. But where GAP did flag officers for intervention by the chain of command, little came of it. "We looked at the supervisors' memos reviewing activations from fiscal year 2020 for 5 of these officers," the report continues. "None of the memos identified any issues [or] mentioned previous activations, and often stated there was 'no pattern of behavior.' One officer activated three quarters in a row with a total of 45 Response-to-Resistance incidents. The super­visor noted no issues and 'no patterns or trends' that might cause concern."

The report does not, of course, identify who these officers are. But we already know of officers whose repeated use of excessive force has had public and tragic consequences in the time frame covered by the report. Officer Christopher Taylor shot and killed Mauris DeSilva, a professor experiencing mental illness, in late July 2019. Nine months later, Taylor shot and killed Mike Ramos, an unarmed Black and Latino Austinite, as he slowly drove away from APD officers, presenting no threat. Taylor is currently awaiting trial for murder in the latter shooting. Officer Chance Bretches is currently awaiting trial for aggravated assault, for a violent arrest carried out in March 2019. He is also under investigation for shooting a demonstrator in the head a year later, during the Black Lives Matter protests of May 30-31, 2020.

In addition to all the FY 20 activations, the audit team also looked at a random sample of 60 activation memos from the five-year review period, and found that in 93% of cases supervisors decided there was no issue to address. In the remaining 7%, supervisors chose to speak informally with the officers to resolve the issues. In none did they recommend counseling or retraining even though APD has such services available, such as through its Wellness Div­is­ion, which coordinates both mental and physical health care for officers.

"The system isn't broken because the technology failed," Kathy Mitchell told us. "It is broken because supervisors won't participate and don't even talk to the officers flagged. I expect they don't launch early interventions because they don't agree that interventions are needed. Like so much else about our department, this is a culture problem."

“Essentially Defunct”

The executive team at APD has known about GAP's issues for some time and did not dispute the audit findings. Interim Chief of Police Joseph Chacon told members of the Council Audit and Finance Com­mittee, which Alter chairs, that the computer software designed with help from a third-party vendor back in 2006 to query APD's databases for GAP has stopped working properly. "The person that wrote the script is no longer with the city," Chacon explained, "and because it was written in a programming language that is essentially defunct, no one has been able to resolve it."

Chacon said that once the city's information technology pros in its Com­­mun­ications and Tech­no­lo­gy Man­age­ment Depart­ment could not resurrect the program, APD assigned one officer, Sissy Jones, to track the GAP data manually. Four times a year, Jones compiles use-of-force, IA complaint, and sick leave data for the entire department. No one assists Jones or checks her work, and she has several other duties. Chacon praised Jones' work but said that "because it's a manual process, there's going to be some human error in there."

Neither Jones nor anyone else keeps track of what happens in the rare cases when a supervisor does recommend an action plan to help an officer activated by GAP. The system does require that a completion date for that plan be established, but does not require tracking of whether officers complete their assigned counseling or training, who is responsible for following up, or what consequences await those who don't.

The report contrasts this disorganized state of affairs with Houston's Early Warning System. As described in Houston Police Department's General Orders dating from 2017, the EWS likewise begins with monitoring data on individual officers and identifying cases that need "referral," the equivalent of GAP "activation." Those referrals are investigated by designated caseworkers and then reviewed by a committee of officers, supervisors, and members of the city's police oversight board, all obligated to maintain confidentiality. (There's a separate committee for civilian employees.) When the committee recommends an action plan, which is ultimately approved by the police chief, the officer's progress is tracked through weekly meetings with the chain of command. If no progress is made, the committee recommends to the chief what should happen next, which can include firing the officer.

Return on Investment

The audit team writes that even though GAP isn't a disciplinary process, APD officers don't see it that way and resent that it has no appeal provisions, but mostly think of it as an annoyance. Supervisors feel the system tells them nothing they don't already know, because uses of force and IA complaints are reviewed and addressed in real time. Training on GAP for supervisors and officers alike is inconsistent and infrequent; many APD personnel, as well as the team at the OPO, know little about it. So there's generally not much buy-in.

"I think that much of the initial training and messaging [on GAP] had fallen off," Chacon explained to Council members on July 21. "And that will become a priority for me going forward, to ensure at all levels that the buy-in is there." Chacon is one of three candidates in the running to lead APD; read more. "And I can tell you at the command and executive level the buy-in is there. We understand the need for the early intervention system – we're trying to prevent problems that officers might have, and we're also trying to save officers' careers and keep them out of trouble."

Chacon wants a system that will provide insight not just into individual officers' conduct, but also trends throughout the police force, such as increases in uses of force or of employee downtime. To do this, Chacon would like GAP to monitor far more than the three data points it currently uses. Houston's EWS has nine; Pittsburgh's system, cited by both APD and the U.S. Department of Justice as a model, tracks 18, including citizen complaints, civil claims relating to officers' conduct, disciplinary reports, mandatory counseling, missed court dates, officer-involved accidents, subject resistance reports, warrantless search and seizure, and weapon discharges, among other things.

Since 2006, data analytics in general, as well as within law enforcement specifically, have become much more sophisticated, tracking these more extensive datasets and using machine learning to interpret them rather than just activating when thresholds are met. The current gold standard from this perspective, as noted in the audit report, is the Charlotte-Mecklenburg PD in North Carolina, which claims that its system, implemented in 2017, has correctly identified 10-20% more officers who went on to have adverse incidents, compared to traditional systems. It also claims to have reduced incorrect identifications by 50%.

"Here's what always happens," Scott Hen­son said. "There's always one study where one department comes out and says, 'We solved it, we have some great model that everyone should do.' And then everyone says, 'Oh great, we can all rest easy. This has solved all the problems.' And then slowly but surely people try and replicate it, and it never quite works out.

"I feel like Charlie Brown and the football with these sorts of claims, because these are never things that are promoted by advocates," Henson continues. "These are always things that departments say: 'Here, we can fix it, we'll create this secret, internal procedure that only we control, and only we see any information from, but it's going to solve it! Trust us.'"

Who’s in Charge?

Chacon, who identifies as a data-driven administrator, said he wants a system that will track 30 or more data points. He believes it should be integrated with a comprehensive updating of APD's data management system, which has been in the works for several years though was put on pause at the peak of Council and community scrutiny of APD in 2020. "There are some off-the-shelf solutions that can be tailored for a department; they're quite expensive," Cha­con said. "[But] it's not just about the early intervention system ... The system in which all of our Internal Affairs records reside is also at end-of-life. So what we're trying to do is connect and be very smart as to how we procure all of this, so that in the end we have something that all works together."

Council members, particularly Alter and Kathie Tovo (her predecessor as Audit and Finance chair), would like to see that new program implemented as soon as possible. Until it's ready, Chacon says the department will downscale GAP to lessen the load on the lone officer analyzing the data, hoping to more accurately identify which officers need help and stop identifying those who don't. He told the council members that he'll soon ask the city to pony up significant funds to update APD's technology and have a new system online by FY 23.

Kathy Mitchell was an active organizer and member of the Reimagining Public Safety Task Force, which in April made its recommendations for permanent reductions of the APD budget and restructuring to move key functions outside the department. Those plans were more or less scotched with the passage of House Bill 1900, signed into law by Gov. Greg Abbott in June, which punishes cities that "defund the police." Nevertheless, Mitchell says, Austin should try to stay true to the vision endorsed by the community and use the money it has available to add more medics, mental health officers, and park rangers, rather than updating GAP.

"I think there's no solution that's controlled and managed by APD, in a black box, that is going to solve it, whether it's new tech or added criteria or a different threshold," Mitchell said. "If they're making all of these decisions internally and there's no oversight, no external review, no second set of eyes on it, then no one's going to trust it. And history tells us that we shouldn't expect better outcomes."

She and Henson reiterate that the Office of Police Oversight already knows which officers need counseling or further training. "Farah is the person to make the early warning list," Henson said. "She's the only one who sees all of it. Even within the department, there's nobody who has that global view. And this is what we created that office for, it's why we gave it new power, and the police department and the city manager insisting that these processes should be internal to APD is really just sort of going backwards."

At present, though, OPO's still delicate relationship with APD does not include such a mandate. The office does, however, support the resurrection of GAP. "The Guidance Advisory Program and the Office of Police Oversight are aligned to help improve public safety for both officers and community members," OPO said in a statement to the Chronicle. "APD has an opportunity to take corrective action to transform GAP into a program that will provide real support for their officers and show that they are willing to change for the better."