Cathy O’Neil’s “Weapons of Math Destruction” and the Collateral Damage of Big Data

It’s not often you can genuinely say that everyone is talking about a book. However, Weapons of Math Destruction, by Cathy O’Neil, has quickly become one of those books that anyone working in data science simply must read. Certainly, the critics are reading it: in 2016, The Guardian, Fortune, and Wired, among others, all named it the Book of the Year.

It’s not hard to see why: Big Data is everywhere, and Ms. O’Neil describes, in terms non-programmers can understand, both how its models work, and the consequences of using them. She begins with IMPACT, the ill-fated teacher evaluation model at the center of the 2012 Baltimore City schools cheating scandal, where educators systematically changed students’ answers on the statewide accountability test to improve their own end-of-year reviews.

From there, she lays bare the collective insanity of U.S. News and World Report’s college rankings; the predatory marketing made possible by individually-targeted advertisements; and the self-fulfilling prophecies created by models that evaluate the fitness of job applicants, recent offenders, and minorities. Along the way, O’Neil gives us an inside view of the risk models that helped bring about the 2008 subprime mortgage crisis: models that not only failed miserably, but enabled the greedy and unscrupulous to knowingly profit from the damage they themselves had inflicted.

Her point is not that all models are bad: rather, she identifies these “Weapons of Math Destruction” (WMDs) as helping to create the outcomes they claim to predict. A model to predict recidivism, she notes, can seem very successful because many of the convicts it identifies as high-risk do in fact re-offend. However, simply labeling someone as high-risk makes him more likely to re-offend, by making legal employment harder to find.

In addition, she argues that these models discriminate against minorities: for example, by counting the number of “contacts with police” prior to arrest, which for minorities often includes borderline-unconstitutional “stop and frisk” encounters. This “feedback loop” not only perpetuates existing inequalities, it gives them credibility: WMDs may not be sufficiently advanced to be magic, but to the vast majority of people who aren’t data scientists, they might as well be. Add to this that successful models are quickly adopted and re-purposed, and you have a pretty fair case for grouping WMDs with their more conventional namesakes.

Most readers will find Weapons of Math Destruction a compelling glimpse into the Big Data economy. Like Michael Lewis’s Moneyball, it showcases the inner workings of an economy shrouded in trade secrets and jargon. As a data scientist, I have first-hand knowledge of just how the sausage is made: what I gained from O’Neil’s book is a framework for understanding how these models, once loosed into the wild, create vast ecosystems ruled by their own unassailable logic.

The snake oil vendor, O’Neil writes, has been around for ages; and, as any economist will tell you, so have perverse incentives and attempts to game the system. What’s new in today’s Big Data economy is the scope of these things: how a single model from someone’s graduate thesis can, in just a few years’ time, completely transform our fundamental understandings and behavior. By itself, that level of change is deeply unsettling; add to it the ruined lives and shattered dreams of people fed into the algorithm, and you have an unmistakable new evil that must be confronted. Read the book.

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *