SmartNewsの推薦システム — ニュースを最適化する機械学習

SmartNews was founded in 2012 and is currently a news distribution app used in both the Japanese and US markets. With tens of thousands of articles flowing in from around the world per day, the recommendation system's role is to select roughly twenty optimal articles for each individual user. This article organizes the overall picture, covering click prediction and diversity balancing, breaking news detection, partnerships with newspaper companies, and more.

The base layer of the recommendation system is a binary classification model that predicts "will this user read this article or not." Many features are fed in — the user's past browsing history, dwell time, time of day, device, article category, the source media outlet — and gradient boosting or deep learning models are used.

Sorting only by click prediction score causes a phenomenon where articles in the same category (for example sports or entertainment) dominate the top spots. To deliver a balanced mix from broad categories like politics, economy, international, science, and entertainment, SmartNews adds diversity constraints at the re-ranking stage. Rules like "no more than three of the same category in a row" or "any one media outlet capped at a certain ratio" are applied to suppress bias.

When articles covering the same event are posted in parallel by each newspaper company, the same news ends up lined up multiple times on the user's screen. SmartNews uses body text embedding vectors and headline similarity to aggregate similar articles into a single cluster, displaying only one representative article. The remaining articles are shown collapsed as "other reports."

News with high urgency — earthquakes, political announcements, major incidents — must bypass the normal recommendation flow and be distributed to all users immediately. SmartNews monitors phenomena where articles on the same topic surge in a short time, and the appearance frequency of specific keywords, to detect breaking-news characteristics. When detected, the news is forcibly exposed via push notifications or dedicated slots.

If recommended articles contain discriminatory expressions or spread misinformation contrary to fact, the trust of the distribution operator is fundamentally shaken. SmartNews combines automatic judgment by machine learning with human review to exclude articles suspected of hate speech or misinformation from distribution. Collaboration with third-party fact-checking organizations is also carried out.

Each article has a thumbs-down button placed to indicate "not interested," and articles with characteristics similar to ones that were pressed become less likely to appear for that user going forward. Combined with methods that treat unclicked articles as implicit negative examples, the user's preferences are continuously updated.

Before deploying a new model or diversity-constraint adjustment to production, the difference from the existing model is always measured via A/B test. Evaluation metrics emphasize multiple facets — not only click rate, but dwell time, return rate to the next session, diversity metrics, and thumbs-down rate. If clicks increased but the return rate dropped, true satisfaction is judged to be declining.

SmartNews concludes formal distribution contracts with each newspaper and publishing company, and has built a mechanism that pays royalties according to the number of articles read and ad revenue. This both supports publisher-side revenue and serves as a foundation for continuously sourcing high-quality articles. It is positioned as an attempt to create a new revenue source for publishers that does not rely on ads or subscriptions.

To instantly return roughly twenty articles per individual user from tens of thousands, two-stage recommendation is the basic approach. The first stage narrows candidates down to hundreds based on freshness, popularity, and user history; the second stage applies a heavy click-prediction model and diversity re-ranking. Applying the model to all items would explode response time, so devices for producing accuracy at realistic computational cost are applied.

SmartNews's recommendation is not limited to the simple optimization of "showing articles that are likely to be clicked." It is a composite system that simultaneously considers diversity, breaking-news characteristics, misinformation exclusion, and publisher partnerships. Only when both machine learning and journalism perspectives come together is a news experience that supports long-term user satisfaction realized.