Skip to main content Accessibility help
  • Print publication year: 2013
  • Online publication date: May 2013

11 - Decentralized reinforcement learning techniques for interference management in heterogeneous networks


Game theory (GT) is a mathematical tool that analyzes interactions among decision makers. Game theory is seen as a natural paradigm to study and analyze wireless networks where players compete for the same resources. The importance of studying the coexistence between macrocells and femtocells from a game theoretical perspective is multi-fold. First, as illustrated in Figure 11.1, by modeling the dynamic spectrum sharing among network players (macrocell base stations (MBSs), femtocell base stations (FBSs), mobile user equipment (MUE), and home user equipment (HUE)) as games, the behaviors and actions of players can be analyzed in a formalized structure, by which the theoretical achievements in GT can be fully utilized. Second, GT equips us with different optimality criteria for various spectrum sharing problems, which are of key importance when it comes to analyzing the equilibrium of the game. Third, the application of GT enables us to derive efficient distributed algorithms for self-organized networks relying only on partial information. In order to achieve this, the theory of strategic reinforcement learning is of utmost importance by allowing players to choose their optimal strategies and gradually learn from their environment through trial and error procedures. A comprehensive source of game theoretic approaches and their application to wireless communications can be found in [1].

[1] S., Lasaulce and H., Tembine, Game Theory and Learning for Wireless Networks: Fundamentals and Applications, 1st edn. Academic Press, 2011.
[2] D., Niyato and E., Hossain, “Dynamics of network selection in heterogeneous wireless networks: an evolutionary game approach,” IEEE Trans. Veh. Technol., vol. 58, no. 4, pp. 2008–17, May 2009.
[3] H., Li, “Multi-agent Q-learning of channel selection in multi-user cognitive radio systems: a two by two case,” in IEEE Int. Conf. on Systems, Man and Cybernetics (SMC), Oct. 2009, pp. 1893–8.
[4] M., Bennis and D., Niyato, “A Q-learning based approach to interference avoidance in self-organized femtocell networks,” in Proc. IEEE Global Telecommun. Conf. (GLOBECOM) Workshops, Dec. 2010, pp. 706–10.
[5] M., Galindo-Serrano, L., Giupponi, and M., Majoral, “On implementation requirements and performances of Q-learning for self-organized femtocells,” in Proc. IEEE Global Telecommun. Conf. (GLOBECOM) Workshops, Dec. 2011, pp. 706–10.
[6] S., Guruacharya, D., Niyato, E., Hossain, and D. I., Kim, “Hierarchical competition in femtocellbased cellular networks,” in Proc. IEEE Global Telecommun. Conf. (GLOBECOM), Dec. 2010, pp. 1–5.
[7] A., Feki and V., Capdevielle, “Autonomous resource allocation for dense LTE networks: a multi armed bandit formulation,” in Proc. IEEE Int. Symp. Personal, Indoor, Mobile Radio Commun. (PIMRC), Dec. 2010, pp. 706–10.
[8] M., Bennis and S. M., Perlaza, “Decentralized cross-tier interference mitigation in cognitive femtocell networks,” in Proc. IEEE Int. Conf. on Commun. (ICC), June 2011, pp. 1–5.
[9] F., Pantisano, M., Bennis, W., Saad, and M., Debbah, “Spectrum leasing as an incentive towards uplink macrocell and femtocell cooperation,” IEEE J. Sel. Areas Commun. (JSAC), vol. 30, no. 3, pp. 617–30, Apr. 2012.
[10] S., Bennis, M., Gurucharya and D., Niyato, “Distributed learning strategies for interference mitigation in femtocell networks,” in Proc. IEEE Global Telecommun. Conf. (GLOBECOM), June 2011, pp. 1–5.
[11] J., Shamma and G., Arslan, “Dynamic fictitious play, dynamic gradient play, and distributed convergence to Nash equilibria,” IEEE Trans. Autom. Control, vol. 50, no. 3, pp. 312–27, Mar. 2005.
[12] J. F., Nash, “Equilibrium points in n-person games,” P. Nat. Acad. Sci. USA, vol. 36, no. 1, pp. 48–9, 1950.
[13] M., Bennis, S., Perlaza, and M., Debbah, “Learning coarse-correlated equilibria in two-tier networks,” in Proc. IEEE Int. Conf. on Commun. (ICC), June 2012, pp. 1–5.
[14] J. W., Weibull, Evolutionary Game Theory. The MIT Press, 1997.
[15] T., Alpcan and T., Basar, Network Security: A Decision and Game Theoretic Approach, 1st edn. Cambridge: Cambridge University Press, 2011.
[16] S., Hart and A., Mas-Colell, “A simple adaptive procedure leading to correlated equilibrium,” Econometrica, vol. 68, no. 5, pp. 1127–50, Sep. 2000.
[17] A., Nedic and A., Ozdaglar, “Distributed subgradient methods for multi-agent optimization,” IEEE Trans. Autom. Control, vol. 54, no. 1, pp. 48–61, Jan. 2009.
[18] “3rd Generation Partnership Project; Technical Specification Group Radio Access Networks; 3G Home NodeB Study Item Technical Report (Release 8),” 3GPP, 3GPP TR 25.820, Mar. 2008.