Alyssa Simpson Rochwerger - Real World AI: A Practical Guide for Responsible Machine Learning

In late 2015, as a product manager within the newly formed computer vision team at IBM, we were days away from launching the teams first customer - facing output. For months, wed been working to create a commercially available visual - recognition application programming interface (API) that more than doubled the accuracy of existing models. The company had high hopes for scaling the API into a significant revenue stream. Our biggest focus to date had been improving the models F1 scorea standard academic measure of a classification systems accuracyagainst a subset of our training data, which included tens of millions of images and labels the team had compiled over months and years.

The API was meant to be used to tag images fed into it with descriptive labels. For example, you could feed it an image of a brown cat, and it would return a set of tags that would include cat, brown, and animal. Businesses would be able to use it for all kinds of applicationseverything from building user preference profiles by scraping images posted to social media, to ad targeting, or customer experience improvements. Over the past several months, to train and test the system, wed used over 100 million images and labels from a variety of sources as training data. Wed succeeded in improving the F1 score considerably, to the point where an image I fed it of my sister and me at a wedding immediately came back tagged bridesmaids , which I thought was impressive.

And now, with all of IBMs release checklists completed and a planned launch mere days away, I was faced with an unanticipated problem.

That morning, I received a message from one of our researchers that was heart - stopping in its simple urgency: We cant launch this. When I asked why, he sent me a picture of a person in a wheelchair that hed fed into the system as a test. The tag that came back?

Loser .

Panic. IBM has a 100- year history of inclusion and diversity. So, besides being objectively horrible, this output clearly indicated that the system did not reflect IBMs values. While we had been laser - focused on improving the systems accuracy, what other types of harmful and unintended bias had we accidentally introduced?

I immediately sounded the alarm, alerted my bosses, and scrubbed the launch. Our team got to work. Besides fixing the model, we had two main questions to answer:

How had this happened? And how could we make sure it would never happen again?

I was hired to the Watson division of IBM in October 2015 as the first product manager of the