Diakopoulos - Automating the news: how algorithms are rewriting the media

Every fiscal quarter automated writing algorithms dutifully churn out thousands of corporate earnings articles for the Associated Press (AP), a more than 170-year-old newswire service. Drawing on little more than structured data, the stories are short, under 200 words, but disseminated very quickly to the AP wire, where they can then be published by any of the more than 1,700 news organizations that constitute the cooperative. By 2018 the AP was producing more than 3,700 stories this way during every earnings season, covering most US traded stocks down to a market capitalization of $75 million. Thats more than ten times the number of stories they wrote without automation, enabling a far greater breadth of coverage. The stories wont be earning Pulitzer prizes any time soon, but they do convey the basics of corporate earnings in a straightforward and easily consumable form, and they do it at scale.

This is the era of news algorithms. Automation and algorithms have reached a point in their maturity where they can do real newsworkcontributing to the journalistic endeavor in a variety of ways. But the story is as much about the people designing and working with automation as it is about the computational algorithms themselves. The technology doesnt supplant human practices so much as it changes the nature of the work. Algorithms are not going to replace journalists wholesale. Instead, the era of news algorithms is about designing efficient and effective human-computer systems.

News organizations such as the Associated Press know thisits part of the strategy. The difference is that rather than having to rush to write that first 200-word story on what earnings were, they [reporters] can actually take some time to digest an earnings release and focus on if theres news, explained Lisa Gibbs, a business editor who helped with the initial roll-out of the automated earnings stories. The automation frees up valuable time for staff to focus on thematic stories that use earnings as a news hook for deeper analysisthe content human journalists are generally more excited to work on anyway. They can really focus on adding value and explaining whats going on in a particular industry or with a particular company, Justin Myers, the news automation editor at AP told me. The organization is quick to argue that no jobs have been lost to automation, and that the technology has in fact offloaded an equivalent of about three full-time jobs worth of effort from business reporters. They are now freed up to pursue other work, including more creative and ambitious stories about corporate trends and business in general.

The business reporters at AP sometimes blend their own efforts with that of the machine, treating the automated earnings reports as a starting point. Its a way to get something out on the wire quickly and cheaply and gives them cover to circle back and write-through an article later on after additional reporting. Maybe they add a quotation from a corporate executive to enrich the story with context and perspective. Other situations call for editorial override. If an experienced reporter thinks that the earnings consensus from the data that feeds the automation is not well calibrated to other sources, he or she might manually write-in additional context and interpretation from an alternative data source. In these cases the signpost at the end of the story updates to reflect the human-machine collaboration: Elements of this story were generated by Automated Insights using data from Zacks Investment Research.

In some edge cases the automation still isnt up to the task, and experienced editors have to step in to get the job done. When it was first automating earnings reports, the AP would not automate bank earnings because banks were still reporting settlements and unique circumstances related to the 2008 financial crisis. There was just no way we were going to be able to get an accurate, sensible [automatically generated] earnings story as long as that was happening, Gibbs told me. Everybody else was merrily enjoying the fruits of automated earnings. But my banking reporter was still coming in at 6:30 in the morning to write on Bank of America earnings. Clearly there are limits to what algorithms and automation can do for news production; human journalists will be needed more than ever.

Of course, automated writing technology is just one piece of this new era. Algorithms and automation are suffusing the entire news production chain, whether enhancing investigative journalism with machine-learning and data-mining methods, creating new interactive media such as newsbots that converse with audiences, or optimizing content for various media platforms using data-driven headline testing. Theres almost no facet of the news production pipeline, from information gathering to sense-making, storytelling, and distribution that is not increasingly touched by algorithms.

Ebullient mysticism swirls around all of the possibilities algorithms create. Automatically written texts ready to publish without a second glance do have an almost magic air about them. And as I write this, artificial intelligence is at a pinnacle of hype. But my hope is that this book will help to inure you to that seduction, to keep your feet firmly planted on the ground. While full automation sounds tantalizing (sure, what person wouldnt want to let the computers do the hard work, while they themselves take a long lunch), the reality is that the era of news algorithms is more aptly characterized as a human-computer symbiosis. Algorithms and automation will continue to be incorporated into news production in important ways, but in most cases they will act as a complement to human effort, rather than a substitute. Whether to enhance scale, speed, efficiency, and breadth or to create new possibilities through content adaptation, optimization, and personalization, there are exciting and seemingly magical things that algorithms make possible. But behind the curtain are designers, editors, reporters, data scientists, and engineers all contributing in direct or indirect ways.