This masterclass surveys the theory and numerical methods for solving dynamic programming (DP) problems. The DP framework has been extensively used in economic modeling because it is sufficiently rich to model almost any problem involving sequential decision making over time and under uncertainty. It will first introduce participants to theoretical concepts, and then focus on empirical applications covering both discrete and continuous decision problems as well as the estimation of dynamic games.
Theory
Derivation of Bellman’s Principle of Optimality for stochastic decision processes involving maximization of expected discounted payoffs over finite and infinite horizons. Relationship between Bellman’s equation and contraction fixed points in infinite horizon, stationary Markovian decision problems. Generalizations to recursive utility theory including non-time-separable and non-expected utility preferences.
Computation
Numerical Methods for Solving Finite and Continuous State Dynamic Programming Problems. The Curse of Dimensionality and two ways of breaking it: a) randomization and b) exploiting special structure.
Empirical Applications: Discrete Decision Processes
Discrete Decision Processes are Problems where the choice variable is restricted to a finite set of alternatives. I describe parametric econometric methods for inferring the unknown parameters of these processes, and survey some of the numerous applications of these models in many different parts of economics.
Empirical Applications: Continuous Decision Processes
Continuous Decision Processes are Problems where the choice variable can take on a continuum of possible values. I describe parametric econometric methods for inferring the unknown parameters of these processes based on the “Euler Equation” and via parametric maximum likelihood and simulated method of moments approaches.
Estimation of Dynamic Games
I extend the single agent decision framework to multi-agent dynamic games. This is much harder and is at the current frontier of research in this area. We will discuss several recent papers that make substantial headway on these topics.