In Chapter 3, we provided a theoretical overview of the explore-exploit problem and its importance in scoring items for recommender systems, in particular, the connection to the classical multiarmed bandit (MAB) problem. We discussed the Bayesian and the minimax approaches to the MAB problem, including some popular heuristics that are used in practice. However, several additional nuances that arise in recommender systems violate assumptions made in the MAB problem.These include dynamic item pool, nonstationary CTR, and delay in feedback. This chapter develops new solutions that work well in practice.

For many recommender systems, it is appropriate to score items based on some positive action rate, such as click probabilities (CTR). Such an approach maximizes the total number of actions on recommended items. A simple approach that is often used in practice is to recommend the top-k items with the highest CTR. We shall refer to this as the most-popular recommendation approach, where popularity is measured through item-specific CTR. Although conceptually simple, the most-popular approach is technically nontrivial because the CTR of items have to be estimated. It also serves as a good baseline for applications where nonpersonalized recommendation is an acceptable solution. Hence we begin in this chapter by developing explore-exploit solutions for the most-popular recommendation problem.

In Section 6.1, we introduce an example application and show the characteristics of most-popular recommendation in this real-life application. We then mathematically define the explore-exploit problem for most-popular recommendation in Section 6.2 and develop a Bayesian solution from first principles in Section 6.3. A number of popular non-Bayesian solutions are reviewed in Section 6.4. Through a series of extensive experiments in Section 6.5, we show that, when the system can be properly modeled using a Bayesian framework, the Bayesian solution performs significantly better than other solutions. Finally, in Section 6.6, we discuss how to address the data sparsity challenge when the set of candidate items is large.

Example Application: Yahoo! Today Module

In Section 5.1.1, we introduced feature modules that are commonly shown on the home pages of web portals. The Today Module on the Yahoo! home page (see Figure 5.1 for a snapshot) is a typical example. The goal for this module is to recommend items (mostly news stories of different types) to maximize user engagement on the home page, usually measured by the total number of clicks.