Open Research Lines

Transfer and Meta RL​

The core idea behind transfer and meta-learning techniques is that experience gathered in learning a given task might help in learning or improving performance in related tasks. Sharing knowledge by successfuly transfering information can lead to drastical improvements over a broad spectrum of real-world problems.

Our Contribution

Our group contributed to transfer and meta-learning problems in several directions. Those include offline transfer of samples and value functions to overcome poor performance at the beginning of the learning process, learning shared representations in deep-reinforcement learning domains, and meta-learning hyper-parameters to facilitate the learning process in similar problems.

Our Contribution

In the field of Inverse RL, our group has achieved relevant results by inventing some techiniques (based on finding a linear basis for the reward function) that allow to solve the IRL problem overcoming the well-known issue of needing to solve a dircet RL problem multiple times (see papers [1] and [2]). Instead, in the field of Imitation learning, we have gave some theoretical contributions connected to continuous control.

Learning from Demonstrations

Learning from Demonstration is a crucial part of RL, both for its wide applicability and because of the peculiar theory and its algorithmic result. This field divides into two sub-fields: in imitation learning we start from an expert that we assume to be very close to the optimal policy and we seek for algorithms to mimic its behaviour. Instead, in inverse RL we only assume the demonstration come from someone who is optimizing a reward function, and we have to discover this reward functions, potentially suprpassing the performance of the demonstrator.

Safe Reinforcement Learning

The purpose of safe RL is to prevent risks and hazards, which is any source of potential damage that may either directly or indirectly be caused by the agent during the learning process or after its deployment. Safe learning is paramount to the development of real-life applications, where agents are employed to take critical decisions.

Our Contribution

Our group provided several contributions on the topic of safety in RL by addressing different sources of risk: safe policy updates are employed to achieve monotonic improvement guarantees; the stochastic nature of the environments is instead tackled by optimizing specific risk measures; finally, model uncertainty is addressed via robust reinforcement learning.

Our Contribution

Our group has tackled both theoretical and practical directions in this field, devising specific techniques to handle non-stationarity in the MAB setting providing regret guarantees for the proposed approaches. From a practical perspective, non-stationarity has been tackled by efficiently reusing past data through statistical-based such as Importance Sampling and by also successfully exploiting Meta-RL techniques in real-world domains. The problem of delay in action execution or in state or reward observations has been faced by either making use of Imitation techniques or by proposing heuristics to build belief representations over the unobserved elements.

Non-stationarity, Delays, and Lifelong RL

Unlike the common assumption of stationary enviroments and immediate reward used in RL, many real-world problems experience changes of different scales and patterns in the environment or even delays in geerated rewards. Non-stationary, Delayed and Lifelong learning try to devise new methods in order to cope with these changes and get closer to more challenging and realistic scenarios, even considering (life)longer time horizons.

Sample Efficiency in RL

Sample efficiency refers to the number of interactions with the environment that are required to train a RL agent towards aptimality. Improving the sample efficiency is paramount to apply RL algorithms in the real-world, where collecting interactions is usally costly and time-consuming.

Our Contribution

Our group contributed to the problem of improving the sample efficiency of RL in several directions. Those include especially variance-reduction methods for policy optimization, as well as sample reuse through importance sampling, and leveraging samples from low-quality yet cheap simulators.

Our Contribution

Our group delivered several contributions on the exploration problem, including works on Bayesian exploration for RL, the exploration-exploitation trade-off in various multi-armed bandit settings, and intrinsically-motivated exploration for unsueprvised RL settings.

Exploration

Exploration refers to the problem of deciding when to exploit the current information to maximize the agent's performance rather than exploring the environment to gather additional information. It is a core challenge of reinforcement learning that impacts most of its application domains.

Configurable Environments

Environment configurability concerns with the opportunity of altering some parameters of the environment in order to guide the learning experience of an artificial agent. This line of research includes scenarios in which the configuration is aimed to ease the learning for the agent or performed by an external configuratior with diverging interests.

Our Contribution

Our group formulated for the first time the problem of environment configurability by introducing the notion of configurable Markov decision process. On top of it, both the cooperative and the non-cooperative settings have been investigated by delivering novel learning algorithms.

Our Contribution

Our group contributed to multi-agent and distributed problems in several directions. Those include early contributions in algorithms for extensive-form games and coalition formation, optimization methods for continuous games exploiting geometric decomposition of the game dynamics, more general contributions to regret analysis in general-sum Markov Games and finally some early considerations on disitributed reinforcement learning through the construction of virtual agents at need.

Multi-Agent, Distributed RL

Distributed Reinforcement Learning (DRL) where multiple, and maybe virtual, agents share their experiences to learn from each other and improve their policies. In DRL, the goal is to divide the learning process across multiple agents to improve learning efficiency, scalability, and stability. This can be achieved by dividing the environment into smaller parts, assigning each part to an individual agent, and allowing them to learn locally and communicate their experiences with each other.

Hierarchical RL

Hierarchical Reinforcement Learning (HRL) is a learning framework used to address complex and long-horizon tasks by dividing them into a hierarchy of simpler subtasks that can be solved individually.

Our Contribution

HRL shows relevant empirical performance compared to its standard counterparts in many domains, however, these results haven't been supported by many theoretical studies. Our group focuses on completing this understanding with new theoretical findings, to motivate the benefit of HRL in particular settings, and being able to classify classes of problems in which an HRL approach should be preferred to a standard one.