Using Machine Learning to Spotlight Quality Journalism on the Web

• How does Deepnews.ai define quality?
We define quality narrowly; in its simplest terms, we look for value-added journalism. This means coverage built on a genuine journalistic approach: depth of reporting, expertise, investigation, angle, ethical processes, and resources deployed by the newsroom.

 

• How does Deepnews.ai work?
Under the hood of the Deepnews system is a convolutional neural network. This type of deep learning model is usually employed for image recognition. (Read this good explanation by a UCLA student, or, if you are math freak, watch this series of lectures from Stanford). The Deepnews.ai model is made with 360 filters and 22~25 million parameters or weights (depending on the version) which is roughly speaking, the grid created by the model to look at a story. Our Deepnews Scoring Model detects and classifies the features, creates links between them, organizes a library of weights for each of them and infers probabilities on their interactions, mutual influence, and meanings. To put these interactions in concrete terms, our model has the equivalent of about 70,000 pages of text. We built and tested 55 versions of the model for more than 1300 hours on Google Cloud, during which hundreds of thousands of articles have been crunched.

 

• How accurate is the Deepnews Scoring Model?
The model is right about 80~85 percent of the time. Which is not bad for a deep learning system. A sophisticated analysis of news articles is a challenging task for artificial intelligence. While perfectly clean datasets of images, for instance, will yield nearly 100 percent accuracy, deep-learning has a difficult time dealing with fuzzy and subjective material such as news.

Sometimes, our scoring model goes awry, granting a mundane story an outstanding score, or trashing a Pulitzer-worthy piece. Countering these aberrations is obviously difficult. But it gets better after each iteration.

 

• Does Deepnews.ai totally exclude human judgment?
No. We frequently calibrate the model against human testers by submitting different batches of articles to English-speaking journalism students. To warrant some statistical reliability, three of them score one article; simultaneously, the model does the same. So far, the deviation between humans and machine ranges from 0.5 to 0.8 points, depending on the type of article.

 

• Does Deepnews.ai address issues such as accuracy and fake news?
Indirectly, yes. From what we have observed, fake news has a different structure from legitimate journalism. We are certain that it can be measured; for instance, we found some distinctive patterns in the sentence structure from different news sources. Also, by relying on the reputation of publishers and authors, and by taking into account their performance over time, Deepnews.ai should be able to downgrade suspicious content items.