Research

This direction constitutes the major focus in my lab. In the off-policy learning setting, data from a behaviour policy is used to learn about a different target policy. For example, a water treatment agent could execute a trusted policy (behavior) for controlling mixing speeds, and learn an improved policy (target) in the background. Once it is confident about the new policy, it can deploy it, or report expected outcomes under this new policy to the human operator.

This research question is critical because off-policy learning is critical for sample efficiency: reducing the amount of interaction needed during learning. For real-world deployment, each sample is precious. Logged, older data is off-policy, and we want to be able to re-use that data. On-policy learning would constrain the agent to only use the most recent data. Though natural, off-policy learning results in more difficult learning problems, due to distribution shifts. It is well-known that, due to this issue, there are convergence issues with some of the foundational algorithms in reinforcement learning, including temporal difference learning methods and actor-critic methods. Despite these known issues, much of the community continues to use these algorithms, potentially because they are simple and well-understood and also potentially because so much research is conducted in simulation where soundness and reliability are less of an issue than in the real world.

We have made several theoretical contributions resolving these issues, and to improving these foundational algorithms that are widely used. There are two most important contributions. One introduces a unified objective that simplifies sound algorithm development for foundational TD learning methods (Patterson et al, Generalized Projected Bellman Error, 2022), that also provides a comprehensive study to understand how to make it more feasible to switch from the well-understood foundational algorithms to these newer, sound ones. The second solves a long-standing open problem about the existence of a Policy Gradient Theorem for the off-policy setting (Imani et al. 2018), with a follow-up journal paper providing further insights and a sound, practical actor-critic algorithm (Graves et al., 2023). These papers actually fix an issue in my own paper, called Off-policy Actor Critic published in 2012. This paper introduced the first off-policy actor-critic algorithm, and it has become the foundation of many other algorithms since.

In addition to these papers that provide the most important insights for algorithm development, my lab has published more than 30 papers on off-policy reinforcement learning that would be too long of a list above.

Another of the primary research questions in my lab is how to learn representations from a stream of correlated data. We have made significant progress, with papers on sparse representations [C19,C33,C39,C51,C104] and on recurrent neural networks [C42,C84, J50, J71]. A key insight in my group is that sparse activations within the network can significantly improve learning stability [C32], with one of our works showing that such sparsity naturally arises when using an objective that optimizes for online adaptation [C27]. We have also developed a simple activation to facilitate learning such sparse activations [C16], and in recent empirical studies [J3,] have found it can significantly improve performance as compared to the standard ReLU activation.

For recurrence, we are exploring architectures that make learning more feasible in RL. In particular, we are looking at diagonal or block-diagonal archiectures for which RTRL is efficient, facilitating online learning, such as in our work on Recurrent Trace Units. We have also considered extensions on TD networks called General Value Function Networks [J50] and a completely different way to learn RNNs framing the problem as fixed points [C42].

Learning and using models is critical towards developing sample efficient agents that learn in deployment. As a simple analogy, consider a vacuum cleaning robot that learns to vacuum the living room and then is moved to a new room. If the policy is a memorized procedure to vacuum the living room, then the agent will not be able to leverage that knowledge in the new room. But a learned dynamics model of the environment might actually generalize between the two rooms. This dynamics model outputs predictions of expected sensory information, after executing an action. The agent can use this model to compute its policy, and adapt its behavior more quickly when seeing novel situations.

This intuition, unfortunately, has not translated into reality, particularly due to the difficulties with learning these dynamics models. We have shown in several works that small inaccuracies in these learned dynamics models can result in bad policies [C26,C36,C44,J79]. It has motivated the introduction of an alternative approach in my lab, that we call goal-space planning [J86]. This approach learns only local models in an abstract space. Importantly, the model itself is composed only of learned policies and learned value functions, for which we have good (off-policy) algorithms. We have shown that, with this form of model, we can get efficient planning and are much more robust to model inaccuracies.

Designing agents that keep learning and adapting throughout deployment, rather than freezing after training. We study plasticity loss, catastrophic forgetting, and learning under non-stationarity. Arguably, a lot of our work can be considered continual learning, but here we list some that are more specifically framed to be about continual learning.

A key goal in the group is to develop better actor-critic algorithms. A key component of these algorithms is to learn the critic (value function), and so better value estimation algorithms are critical for actor-critic. For example, we have incorporated gradient methods into value estimation for PPO ([94], work on gradient eligibility traces listed below).

For actor-critic, though, it is also key to understand how to update the parameterized policy (actor) and interactions with the critic (value estimates). We have a line of work understanding a variety of different updates for the actor [61,63,92], as well as better understanding how to parameterize the actor [73,76,99]. We are pursuing a variety of different avenues to better understand why actor-critic methods can be finicky, with the goal to get stable learning performance in continuous control.

I have recently made an intentional shift to using reinforcement learning in process control, including on real physical systems. We have published one key work on our project using reinforcement learning for a real water treatment problem [74], where we set up a physical pilot system in collaboration with a town in Alberta and with an engineering firm. This work led to a larger vision that reinforcement learning could change how we do process control and help facilities reduce costs and improve sustainability. I have started a venture-backed company, called RLCore, that is bringing this technology to water and wastewater treatment plants, and ultimately broadly to industrial automation. It has been deployed to 5 facilities (some for more than 16 months), saving between 10-25% in chemical costs and improving process reliability, with more deployments in the pipeline.

A key differentiation in our work is focusing on the use of reinforcement learning without access to a simulator, which remains a big open question. Most advanced process control strategies either require strict assumptions or access to a model (simulator) of the environment. Our focus on learning in deployment is largely unique, but I think is critical for wide-spread deployment of reinforcement learning in process control. Note that the most common route has been to painstakingly create simulators, train policies in those simulators, and then transfer to the real world, resulting in the well-known sim2real gap; I was invited to participate in a debate and follow-up position paper on sim2real in robotics [54].

Much of our work on off-policy learning and continual learning is towards practical deployment. A couple of other key topics focused on deployment in the real world is offline-to-online RL (aka hybrid RL) and handling hyperparameters. In offline-to-online RL, the goal is to leverage historical data and then continue updating learning online in deployment [64,65,68,76]. Though learning online is arguably the most important component, starting from a better solution when deploying in the realworld is also key. Another often overlooked issue is the reliance of RL algorithms on hyperparameters, and the barrier this causes to use in applications. In simulation, we can test many hyperparameters; this is not feasible on a real system (as each sample is precious and we cannot run terrible agents). We have developed ideas around how to specify hyperparameters without access to the real system or a simulator [60,97,105].