Absent this question, A/B testing reproduces structural inequality through several mechanism

A is the “best” option for user experience? In this case, what is best for the majority of current site visitors (set the default to White) produces an unequal experience, with the ever-so-slightly more time-consuming experience (additional clicks) reserved for PoC, who may also experience a microaggression in the process. Although our hypothetical “default to white” race/ethnicity dropdown is rarely implemented because of widespread sensitivity to such a blunt reminder of ongoing racial disparity, the same underlying principle is constantly used to develop and refine UX, UI, and other elements of sociotechnical systems.

their gender, race, age, class, and so on are not specified. If we shift to an intersectional framework, one of the implications is that we must restructure testing at all stages, from early prototypes through quality control in mass production, around what Algorithmic Justice League founder Joy Buolamwini has described as intersectional benchmarks

wever, in practice this approach also leads to the reproduction and reification of existing social categories through algorithmic surveillance, tracking users across sites, gathering and selling their data, and the development of filter bubbles (only showing users content that we believe they are comfortable with).

Buolamwini is best known for demonstrating that facial analysis software performs worst on women with darker skin tones, but also advocates for greatly increased regulation and oversight of facial analysis tools, against their use by military or law enforcement, and fights to limit their use against marginalized people across areas as diverse as hiring, housing, and health care.9

n manufacturing and in science, so in design: an intersectional critique of the ways that current design practices systematically reproduce the matrix of domination ultimately requires not only more diverse design teams, community accountability, and control, as we will explore in chapter 2, but also a retooling of the methods that shape so many design domains under the current universalist paradigm.

onsider an algorithm for university admissions. An (individualized) algorithmic fairness approach attempts to ensure that any two individuals with the same profile, but who differ only by, say, gender, receive the same recommendation (admit/waitlist/decline). Auditing an admissions algorithm under the assumptions of algorithmic fairness can be conducted through paired-test audits: submit a group of paired, identical applications, but change only the gender of one of the applicants in each pair and observe whether the system produces the same recommendation for each. If the algorithm recommends admission for more men than women (at a statistically significant level) in otherwise identical paired applications, we can say that it is biased against women. It needs to be retrained and reaudited to ensure that this bias is eliminated. This is the approach proposed by most of the researchers and practitioners working on algorithmic bias today

Now, imagine auditing the same admissions algorithm, but under the assumptions of algorithmic justice. This approach is concerned not only with individualized symmetrical treatment, but also with the individual and group-level effects of historical and ongoing oppression and injustice within the matrix of domination, as well as how to ultimately produce a more just distribution of benefits, opportunities, and harms across all groups of people. In our example, this means that the algorithm designers must discuss, debate, and decide upon what they believe to be a just distribution of outcomes.

algorithmic

The point is not that it's wrong to privilege some users over others; the point is that these decisions need to be made explicit