Early Anomalus Action Detection in Surveillance Video Using MRCNN-LSTM Classification
Received: 20 February 2025 | Revised: 18 April 2025, 9 May 2025, and 2 June 2025 | Accepted: 6 June 2025 | Online: 2 August 2025
Corresponding author: D. Manju
Abstract
Public space monitoring systems are critical for observing typical human behavior and detecting abnormal activities, especially in high-security environments. With the rise in public space thefts, there is a growing need for intelligent systems capable of detecting suspicious movements early enough to prevent criminal acts. Although Convolutional Neural Networks (CNNs) are widely used in image classification, they are inadequate to differentiate between abnormal and normal behavior and identify criminal activity in its early stage. To overcome these limitations, this study proposes a new hybrid model that combines Mask R-CNN (MRCNN) with Long Short-Term Memory (LSTM) networks for accurate object detection, tracking, and sequential behavior analysis. The main contribution of this study is a multistage anomaly detection pipeline that involves frame conversion, contrast enhancement, background removal, object tracking, and feature extraction. The MRCNN-LSTM framework can extract both spatial and temporal characteristics to allow precise early-stage anomaly detection. Thorough testing on three benchmarking datasets, UCF Crime, Snatch1.0, and CUHK, exhibited excellent performance, with a 93.6% accuracy for the UCF Crime dataset. Performance metrics such as observation ratio and time duration were used to assess the responsiveness and effectiveness of the system in real-time surveillance scenarios. This research advances the field of intelligent surveillance by enabling proactive threat mitigation through the early and precise detection of anomalous behavior.
Keywords:
anomaly detection, classification, LSTM, MRCNN, preprocessing, object detectionDownloads
References
L. Ruff et al., "A Unifying Review of Deep and Shallow Anomaly Detection," Proceedings of the IEEE, vol. 109, no. 5, pp. 756–795, Feb. 2021. DOI: https://doi.org/10.1109/JPROC.2021.3052449
D. Roy and K. M. C., "Snatch theft detection in unconstrained surveillance videos using action attribute modelling," Pattern Recognition Letters, vol. 108, pp. 56–61, Jun. 2018. DOI: https://doi.org/10.1016/j.patrec.2018.03.004
L. Shen, Z. Li, and J. Kwok, "Timeseries Anomaly Detection using Temporal Hierarchical One-Class Network," Advances in Neural Information Processing Systems, vol. 33, pp. 13016–13026, 2020.
S. Das, W. K. Wong, T. Dietterich, A. Fern, and A. Emmott, "Discovering Anomalies by Incorporating Feedback from an Expert," ACM Transactions on Knowledge Discovery from Data, vol. 14, no. 4, Mar. 2020. DOI: https://doi.org/10.1145/3396608
S. Nedelkoski, J. Bogatinovski, A. Acker, J. Cardoso, and O. Kao, "Self-Attentive Classification-Based Anomaly Detection in Unstructured Logs," in 2020 IEEE International Conference on Data Mining (ICDM), Sorrento, Italy, Nov. 2020, pp. 1196–1201. DOI: https://doi.org/10.1109/ICDM50108.2020.00148
R. Wang, F. Kong, H. Sudler, and X. Jiao, "Brief Industry Paper: HDAD: Hyperdimensional Computing-based Anomaly Detection for Automotive Sensor Attacks," in 2021 IEEE 27th Real-Time and Embedded Technology and Applications Symposium (RTAS), Nashville, TN, USA, May 2021, pp. 461–464. DOI: https://doi.org/10.1109/RTAS52030.2021.00052
I. Golan and R. El-Yaniv, "Deep Anomaly Detection Using Geometric Transformations," in Advances in Neural Information Processing Systems, 2018, vol. 31.
C. Lee, J. Kim, and S. Kang, "Semi-supervised Anomaly Detection with Reinforcement Learning," in 2022 37th International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC), Phuket, Thailand, Jul. 2022, pp. 933–936. DOI: https://doi.org/10.1109/ITC-CSCC55581.2022.9895028
C. Zhang, W. Zuo, P. Yang, Y. Li, and X. Wang, "Outsourced privacy-preserving anomaly detection in time series of multi-party," China Communications, vol. 19, no. 2, pp. 201–213, Feb. 2022. DOI: https://doi.org/10.23919/JCC.2022.02.016
Z. Wu, H. Su, and Q. Du, "Low-Rank and Collaborative Representation for Hyperspectral Anomaly Detection," in IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium, Yokohama, Japan, Jul. 2019, pp. 1394–1397. DOI: https://doi.org/10.1109/IGARSS.2019.8900381
S. M. A. Karim, N. Ranjan, and D. Shah, "A Scalable Approach to Time Series Anomaly Detection & Failure Analysis for Industrial Systems," in 2020 10th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA, Jan. 2020, pp. 0678–0683. DOI: https://doi.org/10.1109/CCWC47524.2020.9031262
Y. Jiang, W. Wang, and C. Zhao, "A Machine Vision-based Realtime Anomaly Detection Method for Industrial Products Using Deep Learning," in 2019 Chinese Automation Congress (CAC), Hangzhou, China, Nov. 2019, pp. 4842–4847. DOI: https://doi.org/10.1109/CAC48633.2019.8997079
Y. Zhang, M. Xu, Y. Fan, Y. Zhang, and Y. Dong, "A Kernel Background Purification Based Anomaly Target Detection Algorithm for Hyperspectral Imagery," in IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium, Yokohama, Japan, Jul. 2019, pp. 441–444. DOI: https://doi.org/10.1109/IGARSS.2019.8900606
S. Chen, X. Li, and L. Zhao, "Hyperspectral Anomaly Detection with Data Sphering and Unsupervised Target Detection," in IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium, Kuala Lumpur, Malaysia, Jul. 2022, pp. 1975–1978. DOI: https://doi.org/10.1109/IGARSS46834.2022.9884083
J. Luo, J. Lin, Z. Yang, and H. Liu, "SMD Anomaly Detection: A Self-Supervised Texture–Structure Anomaly Detection Framework," IEEE Transactions on Instrumentation and Measurement, vol. 71, pp. 1–11, 2022. DOI: https://doi.org/10.1109/TIM.2022.3194920
K. W. Cheng, Y. T. Chen, and W. H. Fang, "Video anomaly detection and localization using hierarchical feature representation and Gaussian process regression," in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, Jun. 2015, pp. 2909–2917. DOI: https://doi.org/10.1109/CVPR.2015.7298909
D. Varga and T. Szirányi, "Robust real-time pedestrian detection in surveillance videos," Journal of Ambient Intelligence and Humanized Computing, vol. 8, no. 1, pp. 79–85, Feb. 2017. DOI: https://doi.org/10.1007/s12652-016-0369-0
W. Sultani, C. Chen, and M. Shah, "Real-World Anomaly Detection in Surveillance Videos," in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, Jun. 2018, pp. 6479–6488. DOI: https://doi.org/10.1109/CVPR.2018.00678
D. Xu, Y. Yan, E. Ricci, and N. Sebe, "Detecting anomalous events in videos by learning deep representations of appearance and motion," Computer Vision and Image Understanding, vol. 156, pp. 117–127, Mar. 2017. DOI: https://doi.org/10.1016/j.cviu.2016.10.010
Y. Zhao, B. Deng, C. Shen, Y. Liu, H. Lu, and X. S. Hua, "Spatio-Temporal AutoEncoder for Video Anomaly Detection," in Proceedings of the 25th ACM International Conference on Multimedia, Jul. 2017, pp. 1933–1941. DOI: https://doi.org/10.1145/3123266.3123451
L. Ruff et al., "Deep One-Class Classification," in Proceedings of the 35th International Conference on Machine Learning, Jul. 2018, pp. 4393–4402.
A. Gao and J. Liu, "STEAD: Spatio-Temporal Efficient Anomaly Detection for Time and Compute Sensitive Applications." arXiv, Mar. 11, 2025.
H. Singh, A. Kumar,. L. K. Balyan, and G. K. Singh, "Slantlet filter-bank-based satellite image enhancement using gamma-corrected knee transformation," International Journal of Electronics, vol. 105, no. 10, pp. 1695–1715, Oct. 2018. DOI: https://doi.org/10.1080/00207217.2018.1477199
M. Kumar and A. K. Bhandari, "Contrast Enhancement Using Novel White Balancing Parameter Optimization for Perceptually Invisible Images," IEEE Transactions on Image Processing, vol. 29, pp. 7525–7536, 2020. DOI: https://doi.org/10.1109/TIP.2020.3004036
D. Raj and P. Mamoria, "Comparative analysis of contrast enhancement techniques on different images," in 2015 International Conference on Green Computing and Internet of Things (ICGCIoT), Greater Noida, Delhi, India, Oct. 2015, pp. 27–31. DOI: https://doi.org/10.1109/ICGCIoT.2015.7380422
S. P. Brintha, S. P. Premnath, and S. E. Lawrence, "Contrast enhancement using discrete wavelet transform and adaptive tranfer function," in International Conference on Information Communication and Embedded Systems (ICICES2014), Chennai, India, Feb. 2014, pp. 1–4. DOI: https://doi.org/10.1109/ICICES.2014.7034054
D. Vijayalakshmi and M. K. Nath, "A Novel Contrast Enhancement Technique using Gradient-Based Joint Histogram Equalization," Circuits, Systems, and Signal Processing, vol. 40, no. 8, pp. 3929–3967, Aug. 2021. DOI: https://doi.org/10.1007/s00034-021-01655-3
W. Zhan et al., "An improved Yolov5 real-time detection method for small objects captured by UAV," Soft Computing, vol. 26, no. 1, pp. 361–373, Jan. 2022. DOI: https://doi.org/10.1007/s00500-021-06407-8
A. Benjumea, I. Teeti, F. Cuzzolin, and A. Bradley, "YOLO-Z: Improving small object detection in YOLOv5 for autonomous vehicles." arXiv, Jan. 03, 2023.
M. Anwer, S. M. Khan, M. U. Farooq, and Waseemullah, "Attack Detection in IoT using Machine Learning," Engineering, Technology & Applied Science Research, vol. 11, no. 3, pp. 7273–7278, Jun. 2021. DOI: https://doi.org/10.48084/etasr.4202
Y. Fan, G. Wen, D. Li, S. Qiu, M. D. Levine, and F. Xiao, "Video anomaly detection and localization via Gaussian Mixture Fully Convolutional Variational Autoencoder," Computer Vision and Image Understanding, vol. 195, Jun. 2020, Art. no. 102920. DOI: https://doi.org/10.1016/j.cviu.2020.102920
"Real-world Anomaly Detection in Surveillance Videos." [Online]. Available: https://www.crcv.ucf.edu/projects/real-world/.
"UCF Crime Dataset." [Online]. Available: https://www.kaggle.com/datasets/odins0n/ucf-crime-dataset.
"Avenue Dataset." [Online]. Available: https://www.cse.cuhk.edu.hk/leojia/projects/detectabnormal/dataset.html.
T. Reiss and Y. Hoshen, "An Attribute-based Method for Video Anomaly Detection," Transactions on Machine Learning Research, Jul. 2024.
D. Li, X. Nie, R. Gong, X. Lin, and H. Yu, "Multi-Branch GAN-Based Abnormal Events Detection via Context Learning in Surveillance Videos," IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 5, pp. 3439–3450, Feb. 2024. DOI: https://doi.org/10.1109/TCSVT.2023.3325451
D. Manju, M. Seetha, and P. Sammulal, "Early Action Prediction Using 3DCNN With LSTM and Bidirectional LSTM," SSRN Electronic Journal, 2021. DOI: https://doi.org/10.2139/ssrn.3815963
A. Phapale and S. Bhingarkar, "Deep Context-Aware Feature Extraction for Anomaly Detection in Surveillance Videos," Engineering, Technology & Applied Science Research, vol. 15, no. 2, pp. 21633–21638, Apr. 2025. DOI: https://doi.org/10.48084/etasr.9810
A. B. Altamimi and H. Ullah, "Panic Detection in Crowded Scenes," Engineering, Technology & Applied Science Research, vol. 10, no. 2, pp. 5412–5418, Apr. 2020. DOI: https://doi.org/10.48084/etasr.3347
Downloads
How to Cite
License
Copyright (c) 2025 D. Manju, Kishore K. Kumar, Movva Pavani, Rajesh Kumar Verma, Anand Kumar Saraswathi Rathod, Pavan N. V. S. Kumar, V. S. N. Murthy, Bh. Krishna Mohan
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
