Humane Algorithms: Why Deborah Raji is taking on Amazon

Deborah Raji is at Toronto Pearson Airport in Canada when it happens again. She has her ticket, her luggage is checked in, and everything is ready to go. She’s standing in front of a scanner that is supposed to match her face with the photo on her passport. She turns and rotates her head, then adjusts her distance from the scanner, but without success. “Perhaps the light shone unevenly on my face,” she says, rolling her eyes. After a few minutes, she calls for staff assistance. The machine has not recognized her.

Facial recognition software is often presented by tech companies as “miracle technology.” Sophisticated programs are supposed to help recognize scammers, terrorists, and possible suspects. That these technologies work surprisingly well on paper and exceptionally poorly for people who don’t fit the “standard white type,” which Silicon Valley researchers have in mind, is known due to the work of three women. And Deborah Raji is one of them. “Critics accuse me of not knowing enough about how machine learning works in practice,” she says, brushing a braid away from her face. “Yet it was precisely this work that radicalized me.”

Deborah Inioluwa Raji was born in Nigeria, raised in Canada, and radicalized in the US. Anyone who wants to meet her has to be patient and persistent. She doesn’t respond to my first enquiry. She takes her time in responding to my second enquiry, which initially meets with refusal. The reason for a change of mind lies in my identification as a doctoral student with an interest in similar topics. The poor response isn’t so much an attitude as a consequence of her mission to hold tech companies and programmers to account.

So far, it hasn’t worked out too badly. While her peers dream of their first internship, the 24-year-old has already founded an educational initiative for low-income groups, managed to control some algorithms produced by Big Tech companies, and forced one of the biggest tech companies to withdraw its software. Forbes ranks her among the 30 most influential tech personalities under 30, and the Massachusetts Institute of Technology’s MIT Technology Review has named her as one of the most important visionaries under 35. Raji now sits in a student café not far from the university campus in Berkeley, California, and says she increasingly wonders if she is actually part of the problem she is trying to solve.

“A lot of the systems we unleash onto the population don’t work properly,” Raji says. By “we,” she means computer scientists and programmers who like herself develop algorithms and technologies whose exact inner workings they don’t really understand, but which can have very real implications for those affected. “Just because we work with programs doesn’t absolve us of responsibility,” Raji says. If a doctor makes a mistake, it can mean the difference between life and death. It’s the same with technology, she says. “Depending on the range of an algorithm, a 0.1 percent error rate can cost hundreds of thousands of people their lives” if it means they don’t get the right health care, are denied social services, or suspected of being terrorists. All these things have happened before. That’s why Raji has now become a kind of “digital watchdog.” She calls it algorithmic auditing.

Perhaps it’s because, as the third of five children, she always had to assert herself, says Raji, “but I identify injustice pretty quickly.” In any case, she’s fast at most things, not only in thinking and typing but also in speaking. When Raji talks herself into a rage, entire syllables sometimes get lost. Then she grins sheepishly, takes a big sip of her mocha latte, and looks like any other student in the floodlit café, with the exception that Raji actually does the things that in a typical academic career one would only expect to do in a few years’ time: international lectures, podcast appearances, research articles virtually every two weeks. She publishes so much that Timnit Gebru, now one of the most well-known AI ethics researchers globally, commented on the announcement of a new paper by asking her where she finds the time to write the papers, as she already has trouble reading all her contributions.

Facial recognition technology works, but for white men

Her radicalization began in the summer of 2016 when she took an internship at the startup Clarifai, which uses machine learning to analyze images and videos. Raji is considered to be a creative student who enjoys participating in hackathons and writing her own programs in her spare time. At Clarifai, she’s supposed to work on an algorithm that identifies inappropriate content. She’s fascinated by the dynamic work environment and the fearless attitudes of her colleagues until she realizes that her algorithm is over-flagging content that derives from people with darker skin tones.

The data used to train the algorithm is to blame. As it is difficult to get hold of enough training material, programmers make use of classic image archives as well as images used in pornography. While the images from the former group are almost entirely of people with light skin tones, the images from the latter group display a high degree of ethnic diversity. The algorithm therefore automatically associates darker skin tones with inappropriate content. When asked about this, her manager shrugs, it’s hard enough to get enough training data. You can’t worry about ethnic diversity as well.

The problem with the algorithm continues to bother her, so Raji seeks out some other like-minded people. She finds Joy Buolamwini, a master’s student at the MIT Media Lab, who has been working on an ethnically balanced database since she was misclassified as a man by facial recognition software. Together with Timnit Gebru, a doctoral student at Stanford, the women want to test how well current facial recognition technologies work in a society made up of people with different skin tones and genders — in other words, in real life. They are testing three companies that are already actively marketing such technologies: IBM, Microsoft, and the Chinese company Megvii. The results of the “Gender Shades” study are clear: The technology works almost perfectly for white men, but up to 30 percent worse for women with darker skin tones. Various media outlets pick up the story, and the companies promise to improve their technology.

“In our case, the flaws in the system show up”

That three black women have taken notice of the technology’s flaws is no accident, Raji says. “We are part of the group that is not represented in the data. In our case, the flaws in the system show up.” A year later, Raji and Buolamwini conduct a second round of testing. They add the technologies of two new companies as control variables: Amazon and Kairos. This time, Raji leads the investigation. They find that while the companies from the first round have improved greatly, Amazon and Kairos perform similarly poorly to those in the previous year. Shortly after publication, The New York Times picks up the story because the study has some serious implications: Amazon is in the process of selling the flawed technology to U.S. Immigration and Customs Enforcement.

And that’s when things start to get ugly. First, Amazon sends in its own researchers, who claim that the results contradict internal studies. The company does not respond to the request to make the results public. Instead, Buolamwini and Raji are pressured with high-level private emails, with Buolamwini’s supervisor at the Massachusetts Institute of Technology, Ethan Zuckerman, also being put under pressure.

At the same time, the researchers hear that the company is apparently frantically trying to improve the technology internally, while outwardly denying any mistakes. When top researchers in the field of Artificial Intelligence support the study in an open letter and IBM announces that it will take its software off the market, Amazon also backs down. The software is suspended, then taken off the market until further notice. In June 2021, Democratic Senator Edward Markey introduced a legislative proposal that would largely ban government use of facial recognition technologies.

Raji says the case with Amazon showed her “that the discussion about the pros and cons of algorithmic systems is superfluous so long as it is unclear whether they work at all.” What is clear is that faulty technologies disproportionately affect minorities who are already under increased pressure. The danger is that new systems repeat old problems and even amplify them due to their greater reach within a digital context. That’s also why Raji’s work only focuses on algorithms that have already been released onto the population. It’s important to talk about ideal standards, she says, but first companies and programmers must learn to take responsibility for the products they develop and release onto the market. “No one is talking about ideal outcomes. It’s about minimal protection.”

No technology without ethics

In arguing that ethical responsibility and product development are inseparable, Raji is upending established disciplinary boundaries. For decades, researchers have been debating whether engineers can and should take ethical responsibility for their work. The debate falls into two camps. The first assumes a disciplinary separation of powers — ethical and philosophical issues are best left to those who know best. Supporters of the second camp believe that this separation is a dangerous illusion, and that any technical system implicitly adopts the values and ideas of those who developed it. Until now, the divide has run along disciplinary lines, with engineers and programmers on one side and sociologists and philosophers of science on the other. Raji is part of a small but growing group of researchers who have technical backgrounds but who advocate for ethical responsibility. Or maybe it is precisely because of her background that she has become an advocate.

“Deborah is a strong programmer who is not afraid to ask uncomfortable questions. That’s what sets her apart from many of her contemporaries,” says Janice Wait of the Mozilla Foundation, a foundation that funds research into Artificial Intelligence, transparency, and algorithmic bias. Wait knows Raji from various projects they’ve worked on together. What distinguishes Raji, she says, is her strong social engagement both in and aside from research. Raji takes a stand on current issues on Twitter, tirelessly and precisely explaining why algorithmic discrimination is not just a matter of data but of public policy.

The fact that technicians often opt out of the responsibility debate is not due to ill will, Raji says. “It's because there’s a lack of understanding about the concrete impacts that faulty systems can have.” Contrary to the simple and elegant logic of a mathematical problem, ethical issues are often multi-layered, contradictory, and complex, making any task many times more complicated. In addition, it often remains unclear what ideals such as “fairness,” “equality,” and “autonomy” mean in the digital context. For this reason, Raji has decided to return to university to pursue a PhD. She wants to study how machine learning works in theory and what this means for real-world problems. Her goal is to find out how algorithms can best be evaluated, identifying errors that have not yet been discovered.

But it doesn’t always go down well. With her focus on ethical responsibility, she often feels out of place, says Raji, glancing down at her pink leggings. When she notices that she is the only woman in her class, for example, or when she overhears her fellow students talking admiringly about Elon Musk, or when she is asked by a professor to give an introduction to the field of algorithmic bias because hardly anyone in the class has ever heard of it. Or when the first thing someone suggests when working on a common research paper together is to delete the paragraphs about responsibility.

“It’s often forgotten how small the proportion of critical voices is in our field,” Raji says. The fact that critical questions about Artificial Intelligence are often no more than a footnote, even in academia, is partly due to the fact that the field depends to a large extent on industry partners who support research with generous grants, both in Europe and in the US. In the meantime, Raji views her work experience with some of the large technology companies like Google as an advantage. “It helps to understand how these companies work.” While ethics do play a role, “it’s only as long as they don’t get in the way of the business model.”

She no longer believes in self-regulation of the industry. She sees the status quo improving only through improved legal standards, regulation, and the emancipation of those who today suffer most from the negative effects of algorithms. That’s why she founded Project Include at the University of Toronto, an initiative that teaches children and young people from low-income backgrounds technical skills such as programming. It aims to help them solve problems that have a concrete impact on their community through technical skills. The goal is to lay the groundwork for a more balanced and equitable future.

“We don’t just make observations. We aim to make change.”

What makes her hopeful is that there is now a growing network of activists and initiatives specializing in algorithmic discrimination, Raji says. She mentions the Algorithmic Justice League, an organization founded by Joy Buolamwini against algorithmic discrimination, the British law firm Foxglove and non-profit media such as The Markup. What sets them apart, apart from their method, is their mission, Raji says. “We don’t just make observations. We aim to make change."

The fact that US policymakers have also recognized algorithmic discrimination as a problem and have filled key technology posts with critical voices — such as Lina Khan, Alvaro Bedoya, and Tim Wu — is something she observes with mixed feelings. This represents a great opportunity, she says, “but also a great danger.” Ultimately, her concerns are incompatible with those of the Big Tech companies. “What’s good for people isn’t necessarily good for business,” she says. The more attention a topic receives, the more resources are poured into it, the greater the danger of dilution. The worst thing that could happen to her mission is a lot of media attention, flowery speeches, lavish budgets, “and a movement that no longer makes any difference to those affected.”

Cover Image: University of Toronto

This article was originally published in German in the online magazine Republik (translated by Kristy Henderson).