A deep learning driven framework with ObjTrackNet for real-time object and human detection in smart video surveillance

Abstract The rising complexity of urban areas, along with growing public safety needs, has made intelligent video surveillance systems more essential. The current surveillance systems that rely on human observation and predefined analytical rules struggle to handle large crowds and active environments, as they cannot scale effectively and lose effectiveness and precision. Deep learning methods, including CNN-based detectors and edge–cloud collaborative models, achieve better object detection and anomaly recognition. Still, they face difficulties achieving both system resilience and immediate processing, as well as a wide system reach. High-detection-accuracy systems operate with reduced time efficiency, but their lightweight deployment systems struggle to maintain accurate performance across multiple applications. To address these gaps, this research introduces SurveilSmartAI, a deep learning-driven framework, powered by the proposed detection model ObjTrackNet, for real-time object and human detection in intelligent surveillance. The system achieves detection performance through its hierarchical feature design, which operates at different scales and its attention-based mechanisms for real-time multi-object tracking and handling objects of various sizes in detection tasks. The system was trained and validated on the COCO, MOTChallenge, and CAVIAR datasets, demonstrating its ability to perform across multiple datasets and to deploy in real-world situations. The quantitative results show that ObjTrackNet achieves 48.2% mAP@[0.5:0.95] on COCO, 55.4% on MOTChallenge, and 44.7% on CAVIAR, while delivering excellent tracking stability, as evidenced by MOTA scores. Finally, we demonstrated real-time deployability through low-latency testing on edge devices. In bridging the accuracy-efficiency gap and translating it into practically scalable solutions, SurveilSmartAI emerges as an end-to-end next-generation surveillance solution, pushing the boundaries of academic research and real-world deployment of security technologies.