Temporal Dependency‐Aware Trajectory‐Level Behavioural Metric for Exploration in Reinforcement Learning

ABSTRACT Intrinsic motivation serves as the predominant paradigm of exploration in reinforcement learning. In pursuit of an informative and robust state representation, the behavioural metric groups behaviourally equivalent states together, which share the same single‐step reward and transition distribution. However, due to the presence of uninformative rewards and the dynamic nature of procedurally generated environments, these behavioural metric‐based approaches could limit the effectiveness of the learnt state representations, potentially leading to a representation collapse and an ineffective exploration. Therefore, a more comprehensive and generalisable behavioural metric is needed to overcome the above issues. In this work, we approach the exploration problem from a novel perspective, extending beyond the conventional single‐step assessments to encompass a long‐term consideration of the whole trajectory. Specifically, we propose a novel trajectory‐level behavioural metric (TBM) that exploits temporal dependencies of the trajectory and captures the underlying sequential information of behaviour patterns. To achieve an effective trajectory representation for exploration, we develop a pivotal state identifier (PSI) and a trajectory return estimator (TRE) to distinguish the diverse contributions of individual states in the trajectory. Moreover, an auxiliary representation regulariser is developed to promote the diversity and informativeness of the trajectory representation, mitigating the risk of representation mode collapse. Extensive experiments and empirical analysis conducted on procedurally generated environments showcase the superior performance of our proposed framework.