PestDetectSim: an integrated approach for crop pest diagnosis using object detection and similarity-based image retrieval

Abstract Background Rapid and accurate pest diagnosis is essential for reducing crop losses and improving agricultural productivity. Most existing methods rely on deep learning-based object detection to classify pests into predefined categories. However, these models typically select the highest-probability class, which can lead to misclassification and limit practical usability due to the lack of user verification or intervention. Therefore, a framework that integrates model-based prediction with user-assisted validation is needed to enhance diagnostic reliability in real-world agricultural environments. Results We propose PestDetectSim, an integrated framework that combines object detection with similarity-based image retrieval. In PestDetectSim, pest objects are first detected using the YOLO v8 model, and detected regions are processed through a backbone network augmented with a Squeeze-and-Excitation (SE-Net) module to extract feature vectors. These feature vectors are then used to retrieve visually similar pest images, enabling users to cross-check and refine the diagnosis. PestDetectSim produces two outputs: object detection results and a ranked list of similar reference images. We evaluated the framework using a real-world field dataset comprising 30 pest species. On a real-world field dataset of 30 pest species, PestDetectSim attains 98.82% end-to-end diagnostic accuracy while providing case-based visual evidence via retrieved similar images, enabling user verification in ambiguous cases. The end-to-end inference time is approximately 60 ms per image, including both detection and retrieval. This value represents the overall performance of the full pipeline, which integrates the detection stage with the similarity based retrieval stage, making the system suitable for real time deployment on resource limited devices. Conclusions PestDetectSim integrates automated detection with user-assisted verification, offering practical reliability through case-based visual evidence, with modest accuracy gains complemented by improved interpretability. Furthermore, a prototype smartphone application was developed to validate its practicality in real-world conditions. This application allows users to capture pest images, receive instant detection results, and review retrieved similar images for verification. These findings demonstrate PestDetectSim’s potential as a reliable and interactive diagnostic tool for real-time pest monitoring and sustainable agricultural management.