hellsite.austin.gov

For the thousands staring anxiously at their computer screens for hour after hour, the message came like a slap to the face. "There are currently no appointments available. As of 11:40pm, our scheduling system has run into technical issues resulting in massive delays. After working with our vendor for multiple hours, it appears there is no immediate fix we can make."

With that, thousands waiting through the evening of March 15 to make appointments for the COVID-19 vaccine on Austin Public Health's webpage were kicked out of the sign-up process. Many immediately began typing:

"We sat here for six hours. This is unacceptable."

"I was literally at the screen selecting an appointment time. Does this mean that I lose this spot and start completely over in the queue tomorrow?"

"They can't keep the lights on. They can't keep the water running. They can't even handle a basic scheduler. What a waste of space."

It wasn't the first time APH's COVID pages had crashed, or randomly kicked people out of its database, or confounded their attempts to register in the first place. And while the dysfunction was shocking to some, many of the city's own IT workers were definitely unsurprised. They had been predicting the mess for months before watching it unfold.

Tori Breitling was one. Breitling left the city in January after three years of trying to reinvent, or just improve, its website, which has fallen short of tech-friendly Austinites' expectations since the city first went online in the 1990s. She spent the last weeks in her position watching the city try to make the vaccine portal work, offering her advice in vain to confused bosses. It makes her angry to talk about.

"Just try to sign up," Breitling said. "It's a nightmare. I mean, it's barely usable. If you look at the APH Facebook page, in the comments, you will see people complaining about how horrible it is. You know, we're supposed to be this tech city? And Austin has not hired one person who knows what they're doing and put them in control. It's embarrassing. They all should be really embarrassed and frankly I can't believe that they've managed to get away with it for this long."

The city of Austin's Communications & Public Information Office is, among other things, in charge of the content on the city's website. Another department, Commun­i­ca­tions & Technology Management, is, among other things, in charge of providing technical support for the website. Both CPIO and CTM employ web developers and other in-house technical experts. However, neither department created APH's COVID-19 vaccine portal. Instead, the city contracted with third-party vendors using software from Salesforce, the Fortune 500 tech giant that dominates the customer relationship management market.*

The Chronicle interviewed a dozen former and current tech workers with the city to compile this report, many of whom preferred to speak off the record. They say that the vendors implementing the Salesforce platform did not test the vaccine portal to see how many users it could handle before the site went live. So when thousands tried to sign up for vaccines, it crashed. This failure to "load test" the portal, our sources say, was one of several serious blunders made in its design.* But the workers share the conviction that though a vendor built the vaccine portal, its failure is the responsibility of city leaders. As one said, "If you're going to use a vendor, you still need to tightly control the process."

City leaders have shown no eagerness to embrace responsibility for what happened on March 15, or before, or since. Chris Stewart, the chief information officer of CTM, made it clear after a prior crash in February that though he is the city's top technology manager, he wants no blame for the malfunctions. After KXAN ran a critical news piece on the February crash, Stewart sent memos to the City Manager's Office and to his workers, both obtained by the Chronicle.

Stewart wrote to management: "The portal platform used for Austin Public Health's COVID-19 vaccine and testing registration system is entirely separate to the City's main website, austintexas.gov, and is managed by an external vendor. KXAN's conflation of the two is unfortunate and misleading." To his workers, Stewart wrote: "As many of you are aware, that platform and the City's website are completely independent and have nothing to do with each other." He then reminded his workers not to talk to reporters.

Sarah Rigdon is another tech worker who recently left the city's employ. She points out that Stewart's memos show the city's IT department disavowing responsibility for a city webpage, one crucial to saving lives. "The fact that this detail is something they considered important enough to single out, in a huge list of things they could have addressed, is exactly the problem," Rigdon said. "They are so separate from their users that it doesn't occur to them that it doesn't matter who manages each website. For the user, it's the same thing. And their lack of understanding and lack of interest in learning has endangered public health, and continues to make it harder on residents just trying to get the services they need every day."

The City Knows What's Up

There's a reason the heads of APH looked for a third-party vendor to design their vaccine portal. It's because the city's website, austintexas.gov, is fragile, tangled, and a general mess.

This has long been the view of many Austinites, who like Breitling have been amazed, and not in a good way, that a city known worldwide as a tech capital could do such a poor job establishing a useful online presence. The Chronicle's Richard Whit­tak­er reported in February 2012 on the launch of austintexas.gov – four years in the making, after the city abandoned plans to use an outside vendor and moved the job in-house – to replace the original cityofaustin.org, which was launched in the early days of the World Wide Web and also clunky and buggy. The then-leaders of CTM and CPIO acknowledged that the new site was incomplete, that there were many bugs still to be worked through, that large sections of it had to be pulled back for retooling, and that its search functions were inadequate. But they were optimistic that, moving forward, all these issues would be resolved.

They were not. Nearly five years later, an October 2017 report by the Office of the City Auditor told much the same story. The report compared austintexas.gov to websites in nine other large cities and found that it was poorly organized and difficult to navigate. "This, coupled with the website not meeting accessibility guidelines, means that services are more difficult for people of all abilities to find and use," the report's authors wrote.

The audit zeroed in on the way that austintexas.gov presents information. It explained that visitors to the city's website are looking for services – they want to pay parking tickets, apply for building permits, adopt pets, learn about rebate programs. But when they logged on, they weren't asked what kind of service they were looking for. Instead, they were greeted by a list of departments. Rigdon says this is not the way information on modern websites is arranged, because people don't think like that: "The primary mental model of residents – and we know this from research – is that they want their services. They have a task, they go to the city website, and they just want to complete their task. They don't care what departments are involved, or which managers are in charge."

The audit also found that austintexas.gov didn't comply with Americans With Dis­abil­i­ties Act requirements that public websites be accessible equally to people with visual impairments, because screen readers (used to convert text to speech) couldn't make sense of all of austintexas.gov's information. In addition, the language on some pages is written at a college reading level, instead of at the maximum eighth-grade reading level recommended by federal guidelines.

The audit recommended that the city devise a plan for providing online services (something that our sources say was done, but that the city has not implemented); that it assign responsibility for austintexas.gov to a single manager (something that our sources say was never done); that it designate a manager to help each department identify the online services they offer and assure they are accessible (never done); and develop a plan for ADA accessibility (never done).

When asked for comment on our sources' assessment, the city said that it is making "significant progress" on the audit's recommendations. A spokesperson told the Chronicle that the city continues to develop its online services plan, but CPIO and CTM still share responsibility for austintexas.gov, rather than a single manager. The spokesperson said that work is complete on 66 of the more than 500 services identified for digitization; as for complying with ADA requirements, the city plans to continue improving the website and has set a goal of meeting the requirements by 2022.

These good intentions are what's left of the plan agreed to by the leaders of CPIO, CTM, and the City Manager's Office back in 2017 that they hoped would address austintexas.gov's issues. They agreed to continue using CTM to maintain it, but also created a separate team within the department to simultaneously develop a new city website. This new team was named the Office of Design and Delivery – ODD.

ODD People at Work

Breitling was one of ODD's first employees. She's been working in tech since the late 1990s, an expert at user experience design – that is, understanding and improving how people use a website. Breitling joined 10 or 12 other idealistic web experts in the spring of 2018.

The new hires at ODD had moved over from the private sector and were taking a pay cut. They signed on as temp workers, ineligible for holiday or vacation pay. Health insurance wasn't offered until they'd worked for the city for a year. But the ODD workers were motivated by something besides money or job security. For Breitling, it was Donald Trump's election in 2016. "I was really upset at the part that technology had played – Facebook especially – in that turn of events," she said. "And I wanted my next job to feel like I was doing good in the world."

Andrew Do, who worked as a service designer from 2018 to 2020, also wanted to create positive change. "I'm a big believer that government is one of the few institutions that can help solve our collective problems," Do said. "All of my colleagues, we talked philosophically like, 'What is the role of government, why are we in this game?' All of us wanted to work a job that aligned with our values."

For Sarah Rigdon, hired as a content strategist in 2018, working at ODD was an opportunity to help the city deliver services more efficiently. "We wanted to get a modern digital services program," Rigdon said, "which is different from a bunch of people who work on a website and have skills in web design. It's more like asking the city, 'What services do we offer?' And getting them to think about offering services as effectively as possible."

Central to ODD was the concept of equity. The website it would create would prioritize Austin's most vulnerable citizens. It would work well for those with lower levels of education, for those with limited internet access, for those with limited English proficiency, for the visually impaired.

To make a website that met those equity goals, the ODD workers rewrote existing austintexas.gov pages, stripping away the jargon of city government and emphasizing services over departments. But offering services effectively involves more than making a cleaner and more streamlined front-facing webpage. It requires really understanding how a service is provided and then eliminating any barriers that hinder its provision. Web developers call this "service design," and it begins before writing a single line of code.

"The way service design works is you interview a smaller group of people but you understand their experience deeply," Do said. "With big data, you have your large sample size. With us, you have your small sample size, but we're really trying to understand people's experience deeply, start to finish."

Service design is rarely overlooked in major digital projects in the private sector, but it had never been part of the creation or management of austintexas.gov. So ODD's foregrounding of the concept was an innovation, something that distinguished it from the existing web teams at CTM and CPIO. The Service Design Lab, a group of experts within ODD, played a part in every project the office undertook, internally and for departments and programs throughout city government.

Food Trucks, Parking Fines

One was the creation of a guide for people applying for a food truck license. "The food truck permitting was notoriously bad," Breit­ling said. "If you talked to the people who were trying to get permitted, they basically had their own Google Groups to help each other. They talked about, 'If you go to the city, you're going to get a different story every time, depending on who you talk to.'"

Getting a license to operate a food truck is not simple. There are requirements related to the siting of the truck; to its use of electricity, generators, and propane; to cooking surfaces; to refrigeration; to water sources. The truck must be inspected by Austin Public Health, Austin Water, and the Austin Fire Department. ODD's service designers found that these departments operated in silos, reluctant to engage with one another. So they brought the heads of the departments, their front-line workers, and the people trying to get licenses into the same room and got them to agree on what the actual process should be.

"It was almost like therapy," said Taylor Cook, who managed the Service Design Lab from 2018 to 2019. "Just making sure that everybody, together, on both sides of the service, are seeing each other, are collectively seeing the problems together. And then we do a co-creation process – think about our shared goals, think about the services that we need, and then collectively think about what we could do to improve those processes."

Once that consensus was achieved, ODD organized the information and wrote up a guide in English and Spanish. This mobile food vendor permit guide was one of the early success stories incorporated into the website ODD had begun to build – alpha.austin.gov, which most people just called "Alpha." The guide still exists; a link to it is buried on the austintexas.gov landing page for mobile food vending. Clicking it shows one immediately how the Alpha interface differs from the existing page.

Another of ODD's early projects was the creation of the website for the Office of Police Oversight, created in the fall of 2018 to replace the former Office of the Police Monitor – itself created by the city's contract with the Austin Police Association. City Manager Spencer Cronk installed the police monitor, Farah Muscadin, as director of the new office, independent of the union and APD. (In the May 2021 special election, voters approved making the OPO more independent still, moving it under Council's direction.)

With APD's history of brutalizing Black and brown people on the city's Eastside, Muscadin wanted to change how people could complain about police misconduct, as the police monitor process (again, approved by the union) was difficult and frightening for anyone who'd been subjected to force. Chris Luedtke* researched the process as a member of the Service Design Lab. "We went through the experience," Luedtke said. "And you had to take a bus up to [APD's Northeast substation off] Rundberg, you had to walk between police cars, show your ID, be escorted to a room. There were all these blockers that were physical, that didn't need to be there for someone to make a complaint."

Using Luedtke's research, Rigdon helped Muscadin create a form to allow Austinites, for the first time, to criticize or thank officers anonymously. The process occurs completely online, on Alpha. Disciplinary reports by APD's chief of police and complaints from citizens are available on the site as well and, because the information is on Alpha rather than austintexas.gov, these documents can be searched by keywords and date ranges.

ODD also rewrote the Municipal Court's website. As the place where low-level infractions are handled, such as parking and traffic tickets, Municipal Court is one of the city's busiest customer service departments. Many who need services from the court have lower levels of education and English proficiency. Yet they are forced to grapple with concepts like "affidavit of indigency," "motion to expunge," and "petition for deferred disposition."

From time immemorial, it has fallen to the court clerks to make such things understandable. "The way that Municipal Court traditionally operated was that people would either have to call in, or go in, and they would get a very helpful court clerk that would guide them to austintexas.gov/courtforms, which was like a filing cabinet of different forms," explained Gabriela Perez, a former ODD worker who helped rewrite the Municipal Court pages. "And they would tell you which one you needed and you would fill it out. But if you didn't have that person on the phone, you just had a list of dozens of PDFs [documents to be downloaded and, often, printed out] that didn't necessarily make sense to you."

Those who searched austintexas.gov to find the document they needed might come upon outdated, contradictory PDFs from 2002, 2006, or 2012, because the administrators running the Municipal Court site had never removed the older documents. ODD set about digitizing the documents; then they created software that would help Austinites find the documents on their own.

ODD also simplified the website's language and reorganized its content so that services like "Handle a Ticket" and "Appeal a Municipal Court Case" appear at the top of the page. And as they simplified and streamlined content, they worked on bringing another cutting-edge digital enhancement to the city's residents – a translation tool that would accurately convert English to Spanish and other languages.

Websites in most cities use Google Translate to render English into other languages. But Google Translate is not as accurate as more sophisticated translation software. It might translate "court" as a judicial court or a basketball court. So ODD chose a higher-quality software product that could corrected and trained to translate better over time. In the interim, they used human translators to recreate the Municipal Court content; ODD's version of the site radically improved access for Span­ish speakers and reduced the need for workers to answer questions over the phone.*

Non-Fixing Austintexas.gov

As the ODD workers moved content from austintexas.gov onto Alpha, they marveled at the state of the city's old website. "On some pages, we learned that it had been maybe a decade since anyone had done maintenance – or never," Rigdon said. "It was like walking into a giant mansion and every breath you take is blowing a bunch of cobwebs."

Back in 2012 when it debuted, austintexas.gov was already behind the times. It was built with Drupal, a then-popular open-source platform that theoretically would allow content managers throughout city government to easily and effectively update the site, as the CPIO's then-Director Doug Mat­thews told the Chronicle at the time. This was when the use of mobile devices – the new generation of tablets and smartphones – to access websites was expanding exponentially, and becoming the primary interface for many Austinites. But austintexas.gov's Drupal pages didn't adapt well to mobile, and its multi-lingual functionality was poor in a city where a quarter of the residents speak Spanish as their first language.

Drupal also proved to be more complicated than anticipated for administrators to use, which became a real problem as the city's 51 different departments with webpages decided on their own what to post and what to delete. "You have all these people adding content to the website that basically have no training," Breitling said, "whose primary job is not writing for the web. They're doing their other jobs, so maybe they're contributing to the web once a week, once a month, right?"

Because the content management system is so hard for nonspecialists to use, most departments only add content to austintexas.gov when they must and avoid removing old content entirely. Over the years, the website has accumulated 17,000 pages, when it should have no more than 5,000; 45,000 PDFs, when it should have 3,000; and 27,000 images, when it should have 2,000. As one developer said, "We have much more unwanted content than content we want." (Back in 2012, this was already a problem; Matthews told the Chronicle the prior cityofaustin.org had 10,000 pages, of which only 500 saw much traffic.)

Whatever functionality austintexas.gov had when it debuted in 2012 was reduced by 2018, after several teams of developers did maintenance on the site and it became tangled with programming hacks. Meanwhile, the staffers who'd been in place in the system's early years moved on, leaving CTM and CPIO with no in-house experts. So when city leaders decided to update austintexas.gov from Drupal 7 to Drupal 8 in 2018 – as ODD continued to develop Alpha – they decided not to use CTM to do the work, and instead hired an outside company, a third-party vendor named Zilleem.

Those the Chronicle spoke with emphasized that their bosses at CTM and CPIO were warned not to use the company. Nonetheless, the city paid Zilleem to freshen up austintexas.gov over a three-month span; it was to debut in August 2019. That launch was delayed several times, but the updated austintexas.gov finally appeared in the spring of 2020. However, when the site went live, employees at CTM and CPIO discovered literally thousands of bugs.

Our sources say that Zilleem didn't follow best practices in updating austintexas.gov. "Zilleem simply took the existing Drupal 7 site and shoved it poorly into Drupal 8, which broke a lot of basic functionality," one worker said. "To explain how bad this is, a good process has planning, building, testing, launching. Our process was to break a very complex system and then try to fix it little bug by little bug. That process began in 2019 and continues today, as basic elements of the site are still not working."

These bugs are observed easily enough. A search for "City Council minutes" returns a list of pages, none of which link to the current minutes of the City Council. (A Google search, on the other hand, will turn up austintexas.gov's Council Meeting Information Center as the first link.) A search for "Parks and Recreation" may return the department's homepage or it may return links to duplicate pages with no actual information on them. When users click on pictures, they open in a new window, uncompressed and far too large. If you're visually impaired, you won't even know the image is there because screen readers can't interpret it.

Those the Chronicle spoke with said it could take the city's tech workers years to fix all the many bugs littered throughout austintexas.gov. At the rate they're going, they say, they probably never will.

Who Shut Down Alpha?

The successful Municipal Court rollout on Alpha coincided with a shake-up in the city bureaucracy. Nuria Rivera-Vandermyde had been hired in 2019 as deputy city manager, with direct oversight of city administrative departments such as CPIO, CTM, and ODD. Within months, she learned that Austin was, as the administrators later characterized it, "funding two websites." Pressure built on CPIO, CTM, and ODD through the spring and summer of 2020 to begin transferring content from austintexas.gov to Alpha and finish the project. Then, in late August, ODD's main advocate in city management – the head of CTM, Stephen Elkins – suddenly retired. Many believe he was pushed out by Rivera-Vandermyde. Within a week, the new head of the department, Chris Stewart, alerted workers that management would soon choose one website to keep. The other would be abandoned.

Stewart, Rivera-Vandermyde, and Jessica King, Mat­thews' successor at CPIO, began, ostensibly, to collect the information necessary to make that choice. Yasmin Wagner, a CPIO manager and crucial link between the tech workers and the higher­-ups (and familiar to many as an Austin ISD trustee), was a big part of that process, which can be viewed as it happened in many emails obtained by the Chronicle through public information requests. In these, management portrays itself as doing its due diligence. The workers, aware of the shifting ground beneath their feet, earnestly make the case for Alpha.

The arguments they used are documented in a complaint filed with the City Auditor's Office by former ODD employee Joanna Dwyer. The complaint links to a spreadsheet comparing austintexas.gov and Alpha, sent to King, Stewart, Wagner, and Rivera-Vandermyde, and displaying 21 categories such as "accessibility," "document management," and "reading level." In 19 of the 21 categories, the spreadsheet shows that austintexas.gov "doesn't meet expectations," while Alpha's performance is described as "good." Dwyer's complaint also links to a 100-page document that ODD released in August 2020 promising that they could migrate austintexas.gov to Alpha in six months. With these documents and in communications with their superiors, the workers sought to drive home the point that austintexas.gov was old, broken, and expensive to maintain, while Alpha was smart, equitable, and ready to deploy.

King, Stewart, Rivera-Vandermyde, and ODD manager Marni Wilhite met on Oct. 6, 2020, to decide which website to go with. Within an hour of the meeting's conclusion, they sent an email to the staff announcing that they had chosen austintexas.gov. Rigdon, her co-workers, and Wilhite processed the news in a video call where, Rigdon told us, Wilhite reported that Wagner – invited by King to the meeting at the last minute – spoke against Alpha, calling it a risky choice. "That kind of framing doesn't make any sense unless her goal was to scare Nuria and Chris into agreeing with the choice she wanted, which conveniently keeps control of the website under her purview," Rigdon said.

The city says it has no record of who said what at the Oct. 6 meeting, but that the choice was made by Stewart and King – and that it stands by it. "The 2020 decision to continue with Drupal can be described as committing to the product that was both furthest in development and that presented the best long-term opportunity for success," a city spokesperson said. "The work completed on Alpha over the years was not sufficient to prove success or to instill confidence in the product."

Three weeks later, on Oct. 27, the leaders and the workers met to discuss the decision. According to Dwyer's complaint, Tim Ziegler, a content strategist with CPIO who had advocated for Alpha, reiterated the warnings about austintexas.gov. "The truth is that our Drupal implementation is actually very much still broken," Ziegler told the bosses. Stewart replied that he didn't agree. King said that austintexas.gov was being fixed by yet another third-party vendor. Ziegler explained that vendor was just trying to keep the site afloat, not fix it. He said: "This is my area of expertise and I'd be happy to do a deep dive with anyone who wants to at any time." According to our sources, the bosses never requested more information from Ziegler.

Again and again, our sources emphasized that the leaders weren't experienced enough, or interested enough, to process the information they had been shown. They point out that Stewart had been the head of CTM for only a month when he agreed to abandon Alpha; that King has no background in website development; and that Rivera-Vandermyde, who recently left Austin to become the city manager of Boul­der, Colo., is similarly inexperienced.

After the Oct. 27 meeting, Dwyer submitted her complaint to the Auditor's Office. Emails reviewed by the Chronicle show that they punted it to the City Manager's Office, a seeming conflict of interest that will allow Rivera-Vandermyde's former colleagues to decide if her choice of austintexas.gov over Alpha was appropriate.

Better Than the Other

The regret in Rigdon's voice is unmistakable. "All of our design approaches worked for people who have capped data plans. They worked for people with low internet access. This was true before COVID, but now with COVID it's way more clear – you have to access city services online. If you're poor and you're relying on a government program to log on to WIC [food assistance], for example, you don't have time to go into an office anyway. This is a huge equity issue."

ODD has been shut down since February. All of its employees quoted in this story have now left city employment. Rigdon's last day was May 28. The website that she tried to build is still up, but workers at CTM have begun moving it into austintexas.gov.

The leaders don't seem to realize, our sources say, that inserting Alpha into austintexas.gov will destroy many of its innovations. At the Office of Police Oversight, which for now is still on Alpha, users will no longer be able to search police officer discipline and complaint documents by keywords and date ranges. On the Municipal Court page – already transferred over from Alpha – they won't see document titles or descriptions in Spanish. Those with vision impairment won't hear everything on the city's webpages with their screen readers.

When KXAN ran its piece on the APH web crashes in February, it detailed the creation and dissolution of ODD, relying on a familiar framing – that government bureaucracy had wasted millions of dollars. The station tacitly accepted the city's explanation that it was ODD that had done the wasting, and treated that mismanagement as something that was in the past. The ODD workers say that yes, the project cost somewhere around $4 million over its four-year lifespan, and no, there wasn't enough gained for the money spent. They insist, though, that the project was much closer to completion than city leaders realized. They reiterate that if those leaders had simply told the city's departments to work with ODD, Alpha would have been complete in six months.

They also emphasize that the cost of keeping austintexas.gov afloat over the last four years has not been cheap either. The Zilleem contract wound up costing $540,000. The city spent $124,000 more when another third-party vendor, the well-regarded Favish, was brought in to clean up austintexas.gov. Add to that the salaries that the city has been paying two Drupal developers and a contract manager to work alongside Zilleem and Favish for the last few years, and the fact that workers in CPIO and CTM are still fixing the thousands of bugs that went unfixed or were created anew by the 2020 relaunch, and the cost of continuing with austintexas.gov rises into the millions.

That meter is still running, because austintexas.gov is nowhere near being fixed. "Workers in CTM and CPIO spend their days trying to help city departments get their content onto the site, or make it work, or fixing bugs," one worker said. "They're not developing multi-language capabilities or changing the way the pages are presented to help users find forms or services. They're just putting out fires."

Our sources say the city still has no realistic plan to fix austintexas.gov. "There's no acknowledgment that the current website is broken a lot of the time, because they take that as a personal affront," Rigdon said. "There's really no collective acceptance of reality. And we were trying, our team, everyday, to get to that shared reality, but it's just chaos.

"So it's a really sad story. Because you had this team of people that were there for the right reasons, willing to do the work, actually building something. Which is hard to do. And we were 90% of the way there in terms of functionality. It could have happened. And it was obvious – one was better than the other."

---

* Editor's Note: This story has been updated since publication to clarify that the APH Salesforce install was implemented by third-party vendors and not Salesforce itself; to more accurately reflect the progress made by the Alpha team on the site's translation tool; and to correct the spelling of Chris Luedtke's name.