ellenson says that there are much simpler ways of discovering a person’s sexual prefer-ence, such as looking at their social-media accounts. “The idea that a government would need a DNA test to figure out if someone is gay is ridiculous,” he say

ad Institute in Cambridge, Massachusetts, thinks the app was misleading — even danger-ous. “

We are still using these algorithms called humans that are really biased,” says Ghani. “We’ve tested them and known that they’re horrible, but we still use them to make really important decisions every day.”these unregulated tools can harm individu-als and society, causing anxiety, unneces-sary medical expenses, stigmatization and worse. “It’s the Wild West of genetics,” says Erin Demo, a genetic counsellor at Sibley Heart Center Cardiology in Atlanta, Georgia. “This is just going to get harder and harder.”Bellenson posted his app on GenePlaza, an online marketplace for DNA-interpretation tools, in early October. For US$5.50, a person could upload their genetic data — as supplied by consumer DNA sequencing companies such as 23andMe of Mountain View, Califor-nia — and the app would place them along a Nature | Vol 574 | 31 October 2019 | 609©2019SpringerNatureLimited.Allrightsreserved.©2019SpringerNatureLimited.Allrightsreserved.

Developers should routinely run tests such as those performed by Obermeyer’s group before they deploy an algorithm that affects human lives, says Rayid Ghani, a computer scientist at Carnegie Mellon University in Pittsburgh, Pennsylvania.

Developers should routinely run tests such as those performed by Obermeyer’s group before they deploy an algorithm that affects human lives, sa

46.5% if the algorithm was unbiased

had to be sicker than white people before being referred for additional help. Only 17.7% of patients that the algorithm assigned to receive extra care were black. The researchers calculate that the proportion would have been 46.5% if the algorithm was unbiased.

The scientists speculate that this reduced access to care is due to the effects of systemic racism, ranging from distrust of the health-care system to direct racial discrimination by health-care providers

The researchers found that the algorithm assigned risk scores to patients on the basis of total health-care costs accrued in one year. They say that this assumption might have seemed reasonable because higher health-care costs are generally associated with greater health needs. The average black person in the data set that the scientists used had similar health-care costs to the average white person

tumbled across the problem while examin-ing the impact of programmes that provide additional resources and closer medical super-vision for people with multiple, sometimes overlapping, health problems.

But smaller studies and anecdotal reports have documented unfair and biased decision-making by algorithms used in everything from criminal justice to education and health care

. Hospitals and insurers use the algorithm and others like it to help to manage care for about 200 million people in the United States each year

The study, published in Science on 24 Octo-ber, concluded that the algorithm was less likely to refer black people than white people who were equally sick to programmes that aim to improve care for patients with complex medical need

n algorithm widely used in US hospi-tals to allocate health care to patients has been systematically discriminat-ing against black people, a sweeping analysis has foun

ed care, a study found

And here’s one more thing about algorithms: they can leap from onefield to the next, and they often do. Research in epidemiology can holdinsights for box office predictions; spam filters are being retooled toidentify the AIDS virus. This is true of WMDs as well. So if mathematicalmodels in prisons appear to succeed at their job—which really boils downto efficient management of people—they could spread into the rest of theeconomy along with the other WMDs, leaving us as collateral damage

It’s that so manysuffer. These models, powered by algorithms, slam doors in the face ofmillions of people, often for the flimsiest of reasons, and offer no appeal.They’re unfair.

The scoring algorithm is hidden. Acouple of the other WMDs might not seem to satisfy the prerequisite forscale. They’re not huge, at least not yet.

So to sum up, these are the three elements of a WMD: Opacity, Scale,and Damage. All of them will be present, to one degree or another, in theexamples we’ll be coverin

. Thepenal system is teeming with data, especially since convicts enjoy evenfewer privacy rights than the rest of us. What’s more, the system is somiserable, overcrowded, inefficient, expensive, and inhumane that it’scrying out for improvements. Who wouldn’t want a cheap solution likethis

Here, the LSI–R again easily qualifies as a WMD. The people putting ittogether in the 1990s no doubt saw it as a tool to bring evenhandednessand efficiency to the criminal justice system.

like Google, Amazon, andFacebook, these precisely tailored algorithms alone are worth hundredsof billions of dollars.

nd yet many companies go out of their way to hide the results of theirmodels or even their existence. One common justification is that thealgorithm constitutes a “secret sauce” crucial to their business.

Andthey keep quiet about the purpose of the LSI–R questionnaire. Otherwise,they know, many prisoners will attempt to game it, providing answers tomake them look like model citizens the day they leave the joint.

e hypothetical family meal model. If mykids were to question the assumptions that underlie it, whether economicor dietary, I’d be all too happy to provide them.

They are transparent andcontinuously updated, with both the assumptions and the conclusionsclear for all to see.

But in others, includingIdaho and Colorado, judges use the scores to guide their sentencing

Thequestionnaire does avoid asking about race, which is illegal. But with thewealth of detail each prisoner provides, that single

o if early“involvement” with the police signals recidivism, poor people and racialminorities look far riskier.

recidivism models. These help judges assess thedanger posed by each convict. And by many measures they’re animprovement. They keep sentences more consistent and less likely to beswayed by the moods and bi ases of judges.

That pattern isn’t unique to Texas.According to the American Civil Liberties Union, sentences imposed onblack men in the federal system are nearly 20 percent longer than thosefor whites convicted of similar crimes. And though they make up only 13percent of the population, blacks fill up 40 percent of America’s prisoncells.So you might think that computerize

a press release, he declared: “It isinappropriate to allow race to be considered as a factor in our criminaljustice system....The people of Texas want and deserve a system thataffords the same fairness to everyone.

is powered by haphazard data gathering and spuriouscorrelations, reinforced by institutional inequities, and polluted byconfirmation bias.

Whether it comes fromexperience or hearsay, the data indicates that certain types of people havebehaved badly. That generates a binary prediction that all people of thatrace will behave that same way.

This is not to say that good models cannot be primitive. Some veryeffective ones hinge on a single variable

. Ittakes some time to gather new data about the child and adjust theirmodels

This is true of internal models as well. You can often see troubles whengrandparents visit a grandchild they haven’t seen for a whil

Preferences would count for little or nothing. By contrast, if mykids were creating the model, success might feature ice cream at everymeal.

evaluates teacherslargely on the basis of students’ test scores, while ignoring how much theteachers engage the students, work on specific skills, deal with classroommanagement, or help students with personal and family problems. It’soverly simple, sacrificing accuracy and insight for efficiency

we make choices about what’s importantenough to include,

No model can include all of the real world’scomplexity or the nuance of human communication. Inevitably, someimportant information gets left out. I

I would also include parameters, or constraints. Imight limit the fruits and vegetables to what’s in season and dole out acertain amount of Pop-Tarts, but only enough to forestall an openrebellion. I also would add a number of rules. This one likes meat, thisone likes bread and pasta, this one drinks lots of milk and insists onspreading Nutella on everything in sight

The better solution would be to train the model over time, enteringdata every day on what I’d bought and cooked and noting the responsesof each family member

n for me? Or whatif my friend who has kids wants to know my methods? That’s when I’dstart to formalize my model, making it much more systematic and, insome sense, mathematical. And if I were feeling ambitious, I might put itinto a computer program.Ideally, the program would include all of the available food options,their nutritional value and cost, and a complete database of my family’s

hesecorrelations are discriminatory, and some of them are illegal. B

Moreover, their data is highly relevant to the outcomes they aretrying to predict. This may sound obvious, but as we’ll see throughoutthis book, the folks building WMDs routinely lack data for the behaviorsthey’re most interested in

“Moneyball

Shifting defenses is only one piece of a much larger question: Whatsteps can baseball teams take to maximize the probability that they’llwin? In their hunt for answers, baseball statisticians have scrutinizedevery variable they can quantify and attached it to a value.

In other words, he was thinking like a data scientist. He had analyzedcrude data, most of it observational: Ted Williams usually hit the ball toright field. Then he adjusted

But a mathteacher named Sarah Bax continued to push the district administrator, aformer colleague named Jason Kamras, for details. After a back-and-forth that extended for months, Kamras told her to wait for an upcomingtechnical report. Bax responded: “How do you justify evaluating peopleby a measure for which you are unable to provide explanation?

This underscores another common feature of WMDs. They tend topunish the poor. This is, in part, because they are engineered to evaluatelarge numbers of people

ttempting to calculate the impact that one person may have onanother over the course of a school year is much more complex. “Thereare so many factors that go into learning and teaching that it would bevery difficult to measure them all,”

The going theory was that the students weren’t learning enoughbecause their teachers weren’t doing a good job. So in 2009, Rheeimplemented a plan to weed out the low-performing teachers. This is thetrend in troubled school districts around the country, and from a systemsengineering perspective the thinking makes perfect sense: Evaluate theteachers. Get rid of the worst ones, and place the best ones where theycan do the most good. In the language of data scientists, this “optimizes”the school system, presumably ensuring better results for the kids. Exceptfor “bad” teachers, who could argue with that

And they tended to punish the poor and the oppressedin our society, while making the rich richer

And increasingly they focused not on the movements of globalfinancial markets but on human beings, on us. Mathematicians andstatisticians were studying our desires, movements, and spending power.They were predicting our trustworthiness and calculating our potential asstudents, workers, lovers, criminals

The housing crisis, the collapse of major financial institutions,the rise of unemployment—all had been aided and abetted bymathematicians wielding magic formulas. What’s more, thanks to theextraordinary powers that I loved so much, math was able to combinewith technology to multiply the chaos and misfortune, adding efficiencyand scale to systems that I now recognized as flawed

My thesis was on algebraic number theory, a field withroots in all that factoring I did as a child.

Math provided a neat refuge from the messiness of the real world.