Application of Reinforcement Learning Techniques in De Novo Drug Design: A Systematic Literature Review

ABSTRACT Background and Aims De novo drug design is the process of generating novel lead compounds that possess desirable pharmacological activities and optimal physicochemical properties for therapeutic development. In recent years, it has evolved into a key computational strategy for discovering and optimizing new therapeutic compounds. Reinforcement learning (RL), a branch of artificial intelligence, has emerged as a powerful tool to address the complex, sequential decision‐making processes involved in molecular generation. This study aims to review recent applications of RL in de novo drug design, highlight commonly used algorithms, identify major challenges, and discuss future research directions. Methods A systematic literature review (SLR) was conducted following standard review procedures. Articles published between January 2017 and January 2024 were retrieved from Google Scholar using the keyword “Reinforcement Learning Techniques in de novo Drug Design.” Studies were screened based on eligibility criteria, including relevance to RL‐based molecular generation, English language, and full‐text availability. Selected papers were analyzed to extract information on RL algorithms, design strategies, and application areas. Results The reviewed studies demonstrate that RL has been successfully applied to molecular generation, optimization, and drug‐target design. Commonly used algorithms include policy‐gradient, actor–critic, and value‐based methods, often integrated with deep generative models such as recurrent neural networks (RNNs), variational autoencoders (VAEs), generative adversarial networks (GANs), and graph neural networks (GNNs). RL frameworks have optimized properties like binding affinity, solubility, and bioavailability, while promoting molecular diversity. Despite these advances, challenges remain in sample efficiency, reward formulation, and interpretability. Conclusion Reinforcement learning provides a robust framework for automated drug design, enabling intelligent exploration of chemical space and the generation of novel, bioactive compounds. However, further improvements in multi‐objective optimization, computational efficiency, and model transparency are essential for broader clinical applicability. Future research should focus on hybrid RL architectures and explainable AI techniques to bridge computational and experimental drug discovery.