Dynamic programming is an approach to model and solve multi-period decision problems. The fundamental principle of dynamic programming is the Bellman equation, a certain kind of optimality condition. As we detail in this chapter, the central idea of the Bellman equation is to break down a multi-stage problem into multiple two-stage problems. Under suitable conditions, the Bellman equation yields a recursion that helps in characterizing the solution and in computing it. Before embarking on a formal description, we illustrate the dynamic programming approach via some examples.

Some Examples

Example 13.1 (Matches puzzle) Suppose there are 30 matches on a table and I play the following game with a clever opponent: I begin by picking up 1, 2, or 3 matches. Then my opponent must pick 1, 2, or 3 matches. We continue alternating until the last match is picked up. The player who picks up the last match loses. How can I (the first player) be sure of winning?

Solution. If I can ensure that it will be my opponent's turn when 1 match remains, I certainly win. Let us work backwards one step: If I can ensure that it will be my opponent's turn when 5 matches remain, I will also win. The reason for this is that no matter what he does when there are 5 matches left, I can make sure that when he has his next turn, only 1 match will remain. Hence it is clear that I win if I can force my opponent to play when 5 matches remain. We can continue working backwards and conclude that I will ensure victory if I can force my opponent to play when 5, 9, 13, 17, 21, 25, or 29 matches remain. Since the game starts with 30 matches on the table, I can ensure victory by picking 1 match at the beginning, bringing the number down to 29.

Example 13.2(Knapsack problem) Given a set of items, each with a certain weight and value, select the collection of items with total maximum value such that their total weight does not exceed some fixed weight limit W.