Causal Reinforcement Learning

Tutorial Overview

Causal inference provides a set of tools and principles that allows one to combine data and structural invariances about the environment to reason about questions of counterfactual nature — i.e., what would have happened had reality been different, even when no data about this imagined reality is available. Reinforcement Learning is concerned with efficiently finding a policy that optimizes a specific function (e.g., reward, regret) in interactive and uncertain environments. These two disciplines have evolved independently and with virtually no interaction between them. In reality, however, they operate over different aspects of the same building block, i.e., counterfactual relations, which makes them umbilically tied.

In this tutorial, we introduce a unified treatment based on this observation, putting these two disciplines under the same conceptual and theoretical umbrella. We show that a number of natural and pervasive classes of learning problems emerge when this connection is fully established, which cannot be seen individually from either discipline alone. In particular, we'll discuss generalized policy learning (a combination of online, off-policy, and do-calculus learning), when and where to intervene, counterfactual decision-making (and free-will, autonomy, human-AI collaboration), policy generalizability, and causal imitation learning, among others. This new understanding leads to a broader view of what counterfactual learning is, and suggests the great potential for the study of causality and reinforcement learning side by side. We call this new line of investigation "Causal Reinforcement Learning" (CRL, for short).

Target Audience

This tutorial is targeted at researchers working on the foundations of decision-making, learning, and intelligence as well as in applied areas, including robotics and healthcare. After the tutorial, attendees will be familiar with the basic concepts, principles, and algorithms to solve modern problems involving causal reinforcement learning. In particular, they will develop a basic understanding of the differences between more traditional, a-causal RL tools and the new generation of causal reinforcement learning machinery. The prerequites of this tutorial are an undergraduate level understanding of reinforcement learning and graphical models.

Goals and Tasks in CRL

The goals of the tutorial are (1) to introduce the modern theory of causal inference, (2) to connect reinforcement learning and causal inference (CI), introducing causal reinforcement learning, and (3) show a collection of pervasive, practical problems that can only be solved once the connection between RL and CI is established. The implications in decision-making and explainability are direct, so we feel that the tutorial should be compelling for a large part of the community.

There are at least six prominent CRL tasks that have been catalogued:

References

We compiled a list of papers and books relevant to the tutorial. You can click on the references to read the papers.
We followed the CRL classification developed in the tutorial, where [c] indicates a classic paper (off- or on-line).
(This is an initial, tentative list, and if you want to see your paper included on it, please, send us the link as well as a short paragraph explaining how it relates to a specific CRL task. )

Books and Surveys

• [F 1935] Fisher, R. A. The Design of Experiments. Oliver and Boyd, 1935.
• [SB 1998] R. Sutton, A. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
• [P 2000] Pearl, J. Causality: Models, Reasoning, and Inference. Cambridge Press, 2000.
• [PM 2018] Pearl, J., D. Mackenzie. The book of why: The new science of causal and effect. Basic Books, 2018.
• [LS 2020] Lattimore, T., Szepesvári, C. Bandit algorithms. Cambridge University Press, 2020.
• [BCII 2020] Bareinboim, E., Correa, J., Ibeling, D., Icard, T. On Pearl’s Hierarchy and the Foundations of Causal Inference. In "Probabilistic and Causal Inference: The Works of Judea Pearl" (ACM Books). 2020, forthcoming.
• [BLZ 2020] Bareinboim, E., Lee, S., Zhang, J. An Introduction to Causal Reinforcement Learning. Columbia CausalAI Laboratory, Technical Report (R-65). 2020. forthcoming.

Papers

• C [WD 1992] Watkins, C., Dayan, P. Q-Learning. Machine Learning volume 8. 1992.
• C [BP 1994] Balke, A., Pearl, J. Counterfactual Probabilities: Computational Methods, Bounds, and Applications. In Proceedings of the Conference on Uncertainty in Artificial Intelligence 1994.
• C [ACF 2002] Auer, P., Cesa-Bianchi, N., Fischer, P. Finite-time Analysis of the Multiarmed Bandit Problem. Machine Learning volume 47. 2002.
• C [M 2002] Murphy, S. Optimal dynamic treatment regimes. Journal of the Royal Statistical Society (Series B), v.65(2), 2003
• C [AN 2004] Abbeel, P., and Ng, A. Y. Apprenticeship learning via inverse reinforcement learning. In: Proceedings of the twenty-first international conference on Machine learning. 2004.
• C [JOA 2010] Jaksch, T., Ortner, R., Auer, P. Near-optimal Regret Bounds for Reinforcement Learning. Journal of Machine Learning Research 11. 2010.
• C [DLL 2011] Dudik, M., Langford, J., Li, L. Doubly robust policy evaluation and learning. In Proceedings of 28th International Conference on Machine Learning. 2011.
• 4 [BP 2014] Bareinboim, E., Pearl, J. Transportability from Multiple Environments with Limited Experiments: Completeness Results. In Proceedings of the 27th Annual Conference on Neural Information Processing Systems, 2014.
• 3 [BFP 2015] Bareinboim, E., Forney, A., Pearl, J. Bandits with Unobserved Confounders: A Causal Approach. In Proceedings of the 28th Annual Conference on Neural Information Processing Systems, 2015.
• 4 [BP 2016] Bareinboim, E., Pearl, J. Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences, v. 113 (27), pp. 7345-7352, 2016.
• C [JL 2016] Jiang, N., Li, L. Doubly robust off-policy value evaluation for reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning. 2016.
• 3 [ZB 2016] Zhang, J., Bareinboim, E. Markov Decision Processes with Unobserved Confounders: A Causal Approach. CausalAI Lab, Technical Report (R-23), 2016.
• 3 [FPB 2017] Forney, A., Pearl, J., Bareinboim, E. Counterfactual Data-Fusion for Online Reinforcement Learners. In Proceedings of the 34th International Conference on Machine Learning, 2017.
• 5 [KSB 2017] M. Kocaoglu, K. Shanmugam, Bareinboim, E. Experimental Design for Learning Causal Graphs with Latent Variables. In Proceedings of the 31st Annual Conference on Neural Information Processing Systems, 2017.
• 1 [ZB 2017] Zhang, J., Bareinboim, E. Transfer Learning in Multi-Armed Bandits: A Causal Approach. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, 2017.
• 5 [GSKB 2018] Ghassami, A., Salehkaleybar, S., Kiyavash, N., Bareinboim, E. Budgeted Experiment Design for Causal Structure Learning. In Proceedings of the 35th International Conference on Machine Learning. 2018.
• C [KZ 2018] Kallus, N., Zhou, A. Confounding-robust policy improvement. In Advances in Neural Information Processing Systems 2018.
• 2 [LB 2018] Lee, S., Bareinboim, E. Structural Causal Bandits: Where to Intervene?. In Proceedings of the 32nd Annual Conference on Neural Information Processing Systems, 2018.
• 3 [FB 2019] Forney, A., Bareinboim, E. Counterfactual Randomization: Rescuing Experimental Studies from Obscured Confounding. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, 2019.
• 5 [KJSB 2019] Kocaoglu, M., Jaber, A., Shanmugam, K., Bareinboim, E. Characterization and Learning of Causal Graphs with Latent Variables from Soft Interventions. In Proceedings of the 33rd Annual Conference on Neural Information Processing Systems. 2019.
• 2 [LB 2019] Lee, S., Bareinboim, E. Structural Causal Bandits with Non-manipulable Variables. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, 2019.
• 4 [LCB 2019] Lee, S., Correa, J., Bareinboim, E. General Identifiability with Arbitrary Surrogate Experiments. In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence, 2019.
• 1 [ZB 2019] Zhang, J., Bareinboim, E. Near-Optimal Reinforcement Learning in Dynamic Treatment Regimes. In Advances in Neural Information Processing Systems 2019.
• 4 [CB 2020] Correa, J., Bareinboim, E. Transportability of Soft Effects: Completeness Results. In Advances in Neural Information Processing Systems 2020.
• 5 [JKSB 2020] Jaber, A., Kocaoglu, M., Shanmugam, K., Bareinboim, E. Causal Discovery from Soft Interventions with Unknown Targets: Characterization & Learning. In Advances in Neural Information Processing Systems 2020.
• 2 [LB 2020] Lee, S., Bareinboim, E. Characterizing Optimal Mixed Policies: Where to Intervene, What to Observe. In Advances in Neural Information Processing Systems 2020.
• 1 [ZB 2020a] Zhang, J., Bareinboim, E. Designing Optimal Dynamic Treatment Regimes: A Causal Reinforcement Learning Approach. In Proceedings of the 37th International Conference on Machine Learning. 2020.
• 1 [NKYB 2020] Namkoong, H., Keramati, R.,Yadlowsky, S., Brunskill, E. Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding. arXiv:2003.05623. 2020.
• 6 [ZKB 2020] Zhang, J., Kumor, D., Bareinboim, E. Causal Imitation Learning with Unobserved Confounders. In Advances in Neural Information Processing Systems 2020.
• 1 [ZB 2021] Zhang, J., Bareinboim, E. Bounding Causal Effects on Continuous Outcomes. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, 2021.
• 6 [KZB 2021] Kumor, D., Zhang, J, Bareinboim, E. Sequential Causal Imitation Learning with Unobserved Confounders. In Advances in Neural Information Processing Systems 2021.
• 2 [ZB 2022a] Zhang, J., Bareinboim, E. Online Reinforcement Learning for Mixed Policy Scopes. In Proceedings of the 36th Annual Conference on Neural Information Processing Systems, 2022.
• 3 [ZB 2022b] Zhang, J., Bareinboim, E. Can Humans Be Out of the Loop?. In Proceedings of the 1st Conference on Causal Learning and Reasoning, 2022.
• 4 [CLB 2022] Correa, J., Lee, S., Bareinboim, E. Counterfactual Transportability: A Formal Approach. In Proceedings of the 39th International Conference on Machine Learning, 2022.
• 6 [RZDB 2022] Ruan, K., Zhang, J., Di, S., Bareinboim, E. Causal Imitation Learning via Inverse Reinforcement Learning. Technical Report (R-89), CausalAI Lab, Columbia University 2022.