Metrics and Quality Attributes of AI-Based Software as a Medical Device
Received: 12 April 2025 | Revised: 16 May 2025 | Accepted: 24 May 2025 | Online: 2 August 2025
Corresponding author: Shouki A. Ebad
Abstract
Software as a Medical Device (SaMD) refers to software designed for medical purposes that operates independently of any physical hardware. Artificial Intelligence (AI)-based SaMD refers to standalone software utilizing Machine Learning (ML) and other AI techniques to perform medical functions without being part of a physical device. With the growth in research and practical applications of AI-SaMD, a variety of quality attributes have been proposed. However, the existing body of work lacks a comprehensive and cohesive review from a quality assurance perspective. This research paper aims to fill this gap by conducting a systematic analysis and synthesis of literature published between 2015 and 2024 to identify commonly addressed quality attributes, the metrics used to evaluate them, and emerging directions for future research. The identified key quality attributes included accuracy, AI model transparency, safety, performance, and Specificity and Sensitivity (S&S) as the most frequently used criteria for assessing the quality of AI-SaMD. The findings highlight significant gaps in the current research landscape, particularly the absence of a universal metric to comprehensively evaluate the quality of AI-SaMD. Furthermore, there is a lack of alignment between identified quality attributes, their corresponding metrics, and clinical validation standards, such as Food and Drug Administration (FDA) guidelines. The study concludes by offering valuable insights for stakeholders, including clinicians, technologists, and policymakers, and by outlining promising avenues for future research.
Keywords:
Artificial Intelligence (AI), AI-Software as a Medical Device (SaMD), quality, measurement, attributesDownloads
References
IMDRF SaMD Working Group. "Software as a Medical Device (SaMD): Key definitions." International Medical Device Regulators Forum (IMDRF), Dec 2013. [Online]. Available: https://www.imdrf.org/sites/default/files/docs/imdrf/final/technical/imdrf-tech-131209-samd-key-definitions-140901.pdf.
S. J. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, Fourth edition, Global edition. Boston: Pearson, 2022.
H. A. Al-Jamimi and M. Ahmed, "Machine Learning-Based Software Quality Prediction Models: State of the Art," in 2013 International Conference on Information Science and Applications (ICISA), Suwon, Korea (South), Jun. 2013, pp. 1–4. DOI: https://doi.org/10.1109/ICISA.2013.6579473
The Konverge Team, "From code to cure: the potential of software as a medical device", Konverge, Nov. 2023. [Online]. Available: https://www.konverge.com/blog/general/code-to-cure-potential-software-medical-device-samd/.
International Organization for Standardization and International Electrotechnical Commission, ISO/IEC 25010:2011: Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — System and software quality models, First edition, 2011.
B. Kitchenham and P. Brereton, "A systematic review of systematic review process research in software engineering," Information and Software Technology, vol. 55, no. 12, pp. 2049–2075, Dec. 2013. DOI: https://doi.org/10.1016/j.infsof.2013.07.010
S. A. Ebad, A. Alhashmi, M. Amara, A. B. Miled, and M. Saqib, "Artificial Intelligence-Based Software as a Medical Device (AI-SaMD): A Systematic Review," Healthcare, vol. 13, no. 7, Apr. 2025, Art. no. 817. DOI: https://doi.org/10.3390/healthcare13070817
Y. Wang, M. M. Rosli, N. Musa, and F. Li, "Multi-Class Imbalanced Data Classification: A Systematic Mapping Study," Engineering, Technology & Applied Science Research, vol. 14, no. 3, pp. 14183–14190, Jun. 2024. DOI: https://doi.org/10.48084/etasr.7206
N. Ishizuka, "Statistical Review of Regulatory Requirements for AI Diagnosis," in 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC), Osaka, Japan, Jul. 2024, pp. 1975–1976. DOI: https://doi.org/10.1109/COMPSAC61105.2024.00315
A. Ramachandran, P. Malhotra, and D. Soni, "Current Regulatory Landscape of Software as Medical Device in India: Framework for Way Forward," in 2023 10th International Conference on Computing for Sustainable Global Development (INDIACom), New Delhi, India, Mar. 2023, pp. 305–310.
S. Gerke, B. Babic, T. Evgeniou, and I. G. Cohen, "The need for a system view to regulate artificial intelligence/machine learning-based software as medical device," npj Digital Medicine, vol. 3, no. 1, p. 53, Apr. 2020. DOI: https://doi.org/10.1038/s41746-020-0262-2
D. V. Gunasekeran et al., "Acceptance and Perception of Artificial Intelligence Usability in Eye Care (APPRAISE) for Ophthalmologists: A Multinational Perspective," Frontiers in Medicine, vol. 9, Oct. 2022, Art. no. 875242. DOI: https://doi.org/10.3389/fmed.2022.875242
J. P. Campbell et al., "Artificial Intelligence for Retinopathy of Prematurity," Ophthalmology, vol. 129, no. 7, pp. e69–e76, Jul. 2022.
D. M. Hedderich et al., "Artificial intelligence tools in clinical neuroradiology: essential medico-legal aspects," Neuroradiology, vol. 65, no. 7, pp. 1091–1099, Jul. 2023. DOI: https://doi.org/10.1007/s00234-023-03152-7
A. Moglia, K. Georgiou, L. Morelli, K. Toutouzas, R. M. Satava, and A. Cuschieri, "Breaking down the silos of artificial intelligence in surgery: glossary of terms," Surgical Endoscopy, vol. 36, no. 11, pp. 7986–7997, Nov. 2022. DOI: https://doi.org/10.1007/s00464-022-09371-y
V. Zinchenko et al., "Changes in software as a medical device based on artificial intelligence technologies," International Journal of Computer Assisted Radiology and Surgery, vol. 17, no. 10, pp. 1969–1977, Jun. 2022. DOI: https://doi.org/10.1007/s11548-022-02669-1
F. O’Driscoll, N. O’Brien, C. Guo, M. Prime, A. Darzi, and S. Ghafur, "Clinical Simulation in the Regulation of Software as a Medical Device: An eDelphi Study," JMIR Formative Research, vol. 8, Jun. 2024, Art. no. e56241. DOI: https://doi.org/10.2196/56241
D. V. Parums, "Editorial: Global Initiatives Support the Use and Regulation of Digital Health Technology During the COVID-19 Pandemic," Medical Science Monitor, vol. 27, Oct. 2021. DOI: https://doi.org/10.12659/MSM.935123
K. Palaniappan, E. Y. T. Lin, and S. Vogel, "Global Regulatory Frameworks for the Use of Artificial Intelligence (AI) in the Healthcare Services Sector," Healthcare, vol. 12, no. 5, Feb. 2024, Art. no. 562. DOI: https://doi.org/10.3390/healthcare12050562
C. W. L. Ho, "Implementing the human right to science in the regulatory governance of artificial intelligence in healthcare," Journal of Law and the Biosciences, vol. 10, no. 2, Jul. 2023, Art. no. lsad026. DOI: https://doi.org/10.1093/jlb/lsad026
J. A. Smith, R. E. Abhari, Z. Hussain, C. Heneghan, G. S. Collins, and A. J. Carr, "Industry ties and evidence in public comments on the FDA framework for modifications to artificial intelligence/machine learning-based medical devices: a cross sectional study," BMJ Open, vol. 10, no. 10, Oct. 2020, Art. no. e039969. DOI: https://doi.org/10.1136/bmjopen-2020-039969
V. V. Zinchenko et al., "Methodology for Conducting Post-Marketing Surveillance of Software as a Medical Device Based on Artificial Intelligence Technologies," Sovremennye tehnologii v medicine, vol. 14, no. 5, Oct. 2022, Art. no. 15. DOI: https://doi.org/10.17691/stm2022.14.5.02
C. J. Lee et al., "Pivotal trial of a deep-learning-based retinal biomarker (Reti-CVD) in the prediction of cardiovascular disease: data from CMERC-HI," Journal of the American Medical Informatics Association, vol. 31, no. 1, pp. 130–138, Dec. 2023. DOI: https://doi.org/10.1093/jamia/ocad199
J. E. Carolan, J. McGonigle, A. Dennis, P. Lorgelly, and A. Banerjee, "Technology-Enabled, Evidence-Driven, and Patient-Centered: The Way Forward for Regulating Software as a Medical Device," JMIR Medical Informatics, vol. 10, no. 1, Jan. 2022, Art. no. e34038. DOI: https://doi.org/10.2196/34038
D. Giansanti, "The Regulation of Artificial Intelligence in Digital Radiology in the Scientific Literature: A Narrative Review of Reviews," Healthcare, vol. 10, no. 10, Sep. 2022, Art. no. 1824. DOI: https://doi.org/10.3390/healthcare10101824
Z. Angehrn et al., "Artificial Intelligence and Machine Learning Applied at the Point of Care," Frontiers in Pharmacology, vol. 11, Jun. 2020, Art. no. 759. DOI: https://doi.org/10.3389/fphar.2020.00759
S. Muthamilselvan, P. Ramasami Sundhar Baabu, and A. Palaniappan, "Microfluidics for Profiling miRNA Biomarker Panels in AI-Assisted Cancer Diagnosis and Prognosis," Technology in Cancer Research & Treatment, vol. 22, Jan. 2023, Art. no. 15330338231185284. DOI: https://doi.org/10.1177/15330338231185284
A. Arora, "Conceptualising Artificial Intelligence as a Digital Healthcare Innovation: An Introductory Review," Medical Devices: Evidence and Research, vol. Volume 13, pp. 223–230, Aug. 2020. DOI: https://doi.org/10.2147/MDER.S262590
E. Shimron and O. Perlman, "AI in MRI: Computational Frameworks for a Faster, Optimized, and Automated Imaging Workflow," Bioengineering, vol. 10, no. 4, Apr. 2023, Art. no. 492. DOI: https://doi.org/10.3390/bioengineering10040492
K. A. C. Zapata, R. Patil, T. E. Ward, R. Loughran, and F. McCaffery, "Analysis of the Classification of Medical Device Software in the AI Act Proposal," in HHAI Workshops, 2023.
B. Waissengrin et al., "Artificial intelligence (AI) Molecular Analysis Tool Assists in Rapid Treatment Decision in Lung Cancer: A Case Report," Journal of Clinical Pathology, vol. 76, no. 11, pp. 790–792, Nov. 2023. DOI: https://doi.org/10.1136/jcp-2023-208991
K. Ganapathy, "Artificial Intelligence and Healthcare Regulatory and Legal Concerns," Telehealth and Medicine Today, Apr. 2021. DOI: https://doi.org/10.30953/tmt.v6.252
J. M. Woodside, "Decision-Making Model Transparency for SaMD Machine Learning Algorithms," in 2020 Proceedings of the Conference on Information Systems Applied Research Virtual Conference, 2020, vol. 13, Art. no. 5351.
R. Haneef et al., "Innovative use of data sources: a cross-sectional study of data linkage and artificial intelligence practices across European countries," Archives of Public Health, vol. 78, no. 1, Dec. 2020, Art. no. 55.
Å. Ingvar et al., "Minimum Labelling Requirements for Dermatology Artificial Intelligence‐Based Software as Medical Device (SaMD): A consensus statement," Australasian Journal of Dermatology, vol. 65, no. 3, May 2024. DOI: https://doi.org/10.1111/ajd.14222
K. Dortche, G. McCarthy, S. Banbury, and I. Yannatos, "Promoting Health Equity through Improved Regulation of Artificial Intelligence Medical Devices," Journal of Science Policy & Governance, vol. 21, no. 03, Jan. 2023. DOI: https://doi.org/10.38126/JSPG210302
S. H. Park, J.-I. Choi, L. Fournier, and B. Vasey, "Randomized Clinical Trials of Artificial Intelligence in Medicine: Why, When, and How?," Korean Journal of Radiology, vol. 23, no. 12, 2022, Art. no. 1119. DOI: https://doi.org/10.3348/kjr.2022.0834
A. Lal, J. Dang, C. Nabzdyk, O. Gajic, and V. Herasevich, "Regulatory Oversight and Ethical Concerns Surrounding Software as Medical Device (Samd) And Digital Twin Technology In Healthcare," Annals of Translational Medicine, vol. 10, no. 18, pp. 950–950, Sep. 2022. DOI: https://doi.org/10.21037/atm-22-4203
S. T. Suravarapu and K. Yetukuri, "Software Programmes Employed as Medical Devices," World Journal of Current Medical and Pharmaceutical Research, pp. 67–74, Jun. 2023. DOI: https://doi.org/10.37022/wjcmpr.v5i3.268
E. J. Hwang et al., "Use of Artificial Intelligence-Based Software as Medical Devices for Chest Radiography: A Position Paper from the Korean Society of Thoracic Radiology," Korean Journal of Radiology, vol. 22, no. 11, 2021, Art. no. 1743. DOI: https://doi.org/10.3348/kjr.2021.0544
C. W. L. Ho, "When Learning Is Continuous: Bridging the Research–Therapy Divide in the Regulatory Governance of Artificial Intelligence as Medical Devices," in The Cambridge Handbook of Health Research Regulation, 1st ed., G. Laurie, E. Dove, A. Ganguli-Mitra, C. McMillan, E. Postan, N. Sethi, and A. Sorbie, Eds. Cambridge University Press, 2021, pp. 277–286.
F. Chothani, V. Movaliya, K. Vaghela, M. Zaveri, S. Deshpande, and N. Kanki, "Regulatory Prospective on Software as a Medical Device," International Journal of Drug Regulatory Affairs, vol. 10, no. 4, pp. 13–17, Dec. 2022. DOI: https://doi.org/10.22270/ijdra.v10i4.545
J. K. A. Rathod, N. Bhavani, P. P. Saldanha, P. M. Rao, and P. Patil, "A Review of Current Applications of Artificial Intelligence and Machine Learning in Medical Science," International Journal of Advanced Research in Science, Communication and Technology, pp. 316–325, Nov. 2021. DOI: https://doi.org/10.48175/IJARSCT-2125
A. Petzold et al., "Artificial intelligence extension of the OSCAR‐IB criteria," Annals of Clinical and Translational Neurology, vol. 8, no. 7, pp. 1528–1542, Jul. 2021. DOI: https://doi.org/10.1002/acn3.51320
P. Battista et al., "Alzheimer’s Dementia Early Diagnosis, Characterization, Prognosis and Treatment Decision Via a Software‐as‐Medical Device with an Artificial Intelligent Agent," Alzheimer’s & Dementia, vol. 19, no. S18, Dec. 2023, Art. no. e075674. DOI: https://doi.org/10.1002/alz.075674
Y. L. E. Thompson et al., "Applying queueing theory to evaluate wait-time-savings of triage algorithms," Queueing Systems, vol. 108, no. 3–4, pp. 579–610, Dec. 2024. DOI: https://doi.org/10.1007/s11134-024-09927-w
J. T. Megerian et al., "Evaluation of an artificial intelligence-based medical device for diagnosis of autism spectrum disorder," npj Digital Medicine, vol. 5, no. 1, May 2022, Art. no. 57. DOI: https://doi.org/10.1038/s41746-022-00598-6
S. Askin, D. Burkhalter, G. Calado, and S. El Dakrouni, "Artificial Intelligence Applied to clinical trials: opportunities and challenges," Health and Technology, vol. 13, no. 2, pp. 203–213, Mar. 2023. DOI: https://doi.org/10.1007/s12553-023-00738-2
T. J. Loftus et al., "Artificial intelligence-enabled decision support in nephrology," Nature Reviews Nephrology, vol. 18, no. 7, pp. 452–465, Jul. 2022. DOI: https://doi.org/10.1038/s41581-022-00562-3
M. Rassi-Cruz, F. Valente, and M. V. Caniza, "Digital therapeutics and the need for regulation: how to develop products that are innovative, patient-centric and safe," Diabetology & Metabolic Syndrome, vol. 14, no. 1, Dec. 2022, Art. no. 48. DOI: https://doi.org/10.1186/s13098-022-00818-9
H. Rakugi, "Healthy 100-year life in hypertensive patients: messages from the 45th Annual Meeting of the Japanese Society of Hypertension," Hypertension Research, vol. 47, no. 1, pp. 1–3, Jan. 2024. DOI: https://doi.org/10.1038/s41440-023-01533-2
M. I. Miller, L. C. Shih, and V. B. Kolachalama, "Machine Learning in Clinical Trials: A Primer with Applications to Neurology," Neurotherapeutics, vol. 20, no. 4, pp. 1066–1080, Jul. 2023. DOI: https://doi.org/10.1007/s13311-023-01384-2
L. Berliner, "Minimizing possible negative effects of artificial intelligence," International Journal of Computer Assisted Radiology and Surgery, vol. 19, no. 8, pp. 1473–1476, Apr. 2024. DOI: https://doi.org/10.1007/s11548-024-03105-2
K. Zhou and G. Gattinger, "The Evolving Regulatory Paradigm of AI in MedTech: A Review of Perspectives and Where We Are Today," Therapeutic Innovation & Regulatory Science, vol. 58, no. 3, pp. 456–464, May 2024. DOI: https://doi.org/10.1007/s43441-024-00628-3
J. He, S. L. Baxter, J. Xu, J. Xu, X. Zhou, and K. Zhang, "The practical implementation of artificial intelligence technologies in medicine," Nature Medicine, vol. 25, no. 1, pp. 30–36, Jan. 2019. DOI: https://doi.org/10.1038/s41591-018-0307-0
L. Cerdá-Alberich et al., "MAIC–10 brief quality checklist for publications using artificial intelligence and medical images," Insights into Imaging, vol. 14, no. 1, Jan. 2023, Art. no. 11. DOI: https://doi.org/10.1186/s13244-022-01355-9
A. Agrawal, G. D. Khatri, B. Khurana, A. D. Sodickson, Y. Liang, and D. Dreizin, "A survey of ASER members on artificial intelligence in emergency radiology: trends, perceptions, and expectations," Emergency Radiology, vol. 30, no. 3, pp. 267–277, Mar. 2023. DOI: https://doi.org/10.1007/s10140-023-02121-0
D. Dreizin et al., "Artificial intelligence CAD tools in trauma imaging: a scoping review from the American Society of Emergency Radiology (ASER) AI/ML Expert Panel," Emergency Radiology, vol. 30, no. 3, pp. 251–265, Mar. 2023. DOI: https://doi.org/10.1007/s10140-023-02120-1
R. J. Woodman and A. A. Mangoni, "A comprehensive review of machine learning algorithms and their application in geriatric medicine: present and future," Aging Clinical and Experimental Research, vol. 35, no. 11, pp. 2363–2397, Sep. 2023. DOI: https://doi.org/10.1007/s40520-023-02552-2
F. Tettey, S. K. Parupelli, and S. Desai, "A Review of Biomedical Devices: Classification, Regulatory Guidelines, Human Factors, Software as a Medical Device, and Cybersecurity," Biomedical Materials & Devices, vol. 2, no. 1, pp. 316–341, Mar. 2024. DOI: https://doi.org/10.1007/s44174-023-00113-9
E. J. Hwang et al., "2023 Survey on User Experience of Artificial Intelligence Software in Radiology by the Korean Society of Radiology," Korean Journal of Radiology, vol. 25, no. 7, 2024, Art. no. 613. DOI: https://doi.org/10.3348/kjr.2023.1246
J. Feng, S. Emerson, and N. Simon, "Approval policies for modifications to machine learning‐based software as a medical device: A study of bio‐creep," Biometrics, vol. 77, no. 1, pp. 31–44, Mar. 2021. DOI: https://doi.org/10.1111/biom.13379
J. Herington et al., "Ethical Considerations for Artificial Intelligence in Medical Imaging: Deployment and Governance," Journal of Nuclear Medicine, vol. 64, no. 10, pp. 1509–1515, Oct. 2023. DOI: https://doi.org/10.2967/jnumed.123.266110
H. Dickinson, J. Feifel, K. Muylle, T. Ochi, and E. Vallejo-Yagüe, "Learning with an evolving medicine label: how artificial intelligence-based medication recommendation systems must adapt to changing medication labels," Expert Opinion on Drug Safety, vol. 23, no. 5, pp. 547–552, May 2024. DOI: https://doi.org/10.1080/14740338.2024.2338252
P. Li, R. Williams, S. Gilbert, and S. Anderson, "Regulating Artificial Intelligence and Machine Learning-Enabled Medical Devices in Europe and the United Kingdom," Law, Technology and Humans, vol. 5, no. 2, pp. 94–113, Nov. 2023. DOI: https://doi.org/10.5204/lthj.3073
A. Oloruntoba et al., "Examining labelling guidelines for AI ‐based software as a medical device: A review and analysis of dermatology mobile applications in Australia," Australasian Journal of Dermatology, vol. 65, no. 5, pp. 409–422, Aug. 2024. DOI: https://doi.org/10.1111/ajd.14269
D. D. Miller, "Machine Intelligence in Cardiovascular Medicine," Cardiology in Review, vol. 28, no. 2, pp. 53–64, Mar. 2020. DOI: https://doi.org/10.1097/CRD.0000000000000294
D. Hua, N. Petrina, N. Young, J.-G. Cho, and S. K. Poon, "Understanding the factors influencing acceptability of AI in medical imaging domains among healthcare professionals: A scoping review," Artificial Intelligence in Medicine, vol. 147, Jan. 2024, Art. no. 102698. DOI: https://doi.org/10.1016/j.artmed.2023.102698
K. Lim, T.-Y. Heo, and J. Yun, "Trends in the Approval and Quality Management of Artificial Intelligence Medical Devices in the Republic of Korea," Diagnostics, vol. 12, no. 2, Jan. 2022, Art. no. 355. DOI: https://doi.org/10.3390/diagnostics12020355
M. Komatsu, N. Teraya, T. Natsume, N. Harada, K. Takeda, and Hamamoto, "Clinical Application of Artificial Intelligence in Ultrasound Imaging for Oncology," JMA Journal, vol. 8, no. 1, pp. 18–25, 2025. DOI: https://doi.org/10.31662/jmaj.2024-0203
K. M. Adams, Non-functional Requirements in Systems Analysis and Design, First edition. Cham: Springer International Publishing, 2015.
R. B. Grady, Practical software metrics for project management and process improvement. Englewood Cliffs, NJ: Prentice Hall P T R, 1992.
I. Sommerville, Software engineering, Tenth edition. Boston: Pearson, 2016.
S. A. Ebad, "The influencing causes of software unavailability: A case study from industry," Software: Practice and Experience, vol. 48, no. 5, pp. 1056–1076, May 2018. DOI: https://doi.org/10.1002/spe.2569
Downloads
How to Cite
License
Copyright (c) 2025 Shouki A. Ebad, Radhia Zaghdoud, Achraf Ben Miled
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
