Virtual Reality for Competency Assessment: A Usability Study of MUVE PCA with VRSUQ and SUS

Dhega Febiharsa; Syaad Patmanthara; Hakkun Elmunsyah; Dwi Agus Sudjimat

doi:10.48084/etasr.10669

Authors

Dhega Febiharsa Postgraduate School, Universitas Negeri Malang, Malang, Indonesia https://orcid.org/0000-0001-8795-5579
Syaad Patmanthara Department of Electronics and Informatics Engineering, Engineering Faculty, Universitas Negeri Malang, Malang, Indonesia https://orcid.org/0009-0003-4889-4656
Hakkun Elmunsyah Department of Electronics and Informatics Engineering, Engineering Faculty, Universitas Negeri Malang, Malang, Indonesia https://orcid.org/0000-0002-0754-3097
Dwi Agus Sudjimat Department of Mechanical Engineering Education, Engineering Faculty, Universitas Negeri Malang, Malang, Indonesia https://orcid.org/0000-0001-6995-2454

Volume: 15 | Issue: 4 | Pages: 24344-24353 | August 2025 | https://doi.org/10.48084/etasr.10669

Received: 21 February 2025 | Revised: 26 April 2025 | Accepted: 8 May 2025 | Online: 26 May 2025

Corresponding author: Dhega Febiharsa

Abstract

The development of Virtual Reality (VR) technology for competency assessment with multiple users is a breakthrough in the assessment of skills. The Multi-User Virtual Environment for Professional Certification Assessment (MUVE PCA) is a VR-based application that provides a VR experience to users. This study aimed to obtain information on the usefulness of the application and whether it can be accepted by users using the Virtual Reality System Usability Questionnaire (VRSUQ) and the System Usability Scale (SUS). Data collection was carried out by piloting the MUVE PCA application installed on the Oculus Head Mounted Device (HMD) to vocational school students, vocational school teachers, students, and lecturers in the field of Information Technology (IT) in computer network environments. The results showed a SUS score of 70.41, slightly above the global SUS standard, indicating that the system is considered acceptable. The average VRSUQ score of 3.52 is slightly below the standards, indicating that the system needs to be improved in some areas, especially in terms of convenience.

Keywords:

multi-user virtual environment, virtual reality, professional certification assessment, simulation, immersion

Downloads

Download data is not yet available.

References

C. Y. Piaw, "Replacing Paper-based Testing with Computer-based Testing in Assessment: Are we Doing Wrong?," Procedia - Social and Behavioral Sciences, vol. 64, pp. 655–664, Nov. 2012. DOI: https://doi.org/10.1016/j.sbspro.2012.11.077

K. Scalise and B. Gifford, "Computer-Based Assessment in E-Learning: A Framework for Constructing ‘Intermediate Constraint’ Questions and Tasks for Technology Platforms," The Journal of Technology, Learning and Assessment, vol. 4, no. 6, Jan. 2006.

M. Jawaid, F. A. Moosa, F. Jaleel, and J. Ashraf, "Computer Based Assessment (CBA): Perception of residents at Dow University of Health Sciences," Pakistan Journal of Medical Sciences, vol. 30, no. 4, May 2014. DOI: https://doi.org/10.12669/pjms.304.5444

G. Riva and S. Serino, "Virtual Reality in the Assessment, Understanding and Treatment of Mental Health Disorders," Journal of Clinical Medicine, vol. 9, no. 11, Oct. 2020, Art. no. 3434. DOI: https://doi.org/10.3390/jcm9113434

R. C. A. Barrett et al., "Comparing virtual reality, desktop-based 3D, and 2D versions of a category learning experiment," PLOS ONE, vol. 17, no. 10, Oct. 2022, Art. no. e0275119. DOI: https://doi.org/10.1371/journal.pone.0275119

E. Rey-Becerra, L. H. Barrero, R. Ellegast, and A. Kluge, "Improvement of short-term outcomes with VR-based safety training for work at heights," Applied Ergonomics, vol. 112, Oct. 2023, Art. no. 104077. DOI: https://doi.org/10.1016/j.apergo.2023.104077

P. Malega, N. Daneshjo, J. Kovac, and P. Korba, "Enhancing Filter Assembly and Printing Efficiency through Automation and Virtual Reality Integration," Engineering, Technology & Applied Science Research, vol. 14, no. 6, pp. 17555–17563, Dec. 2024. DOI: https://doi.org/10.48084/etasr.8409

I. Ismail, A. Permanasari, and W. Setiawan, "Stem virtual lab: An alternative practical media to enhance student’s scientific literacy," Jurnal Pendidikan IPA Indonesia, vol. 5, no. 2, pp. 239–246, Nov. 2016. DOI: https://doi.org/10.21831/jipi.v2i2.8570

M. Bima, H. Saputro, and A. Efendy, "Virtual Laboratory to Support a Practical Learning of Micro Power Generation in Indonesian Vocational High Schools," Open Engineering, vol. 11, no. 1, pp. 508–518, Jan. 2021. DOI: https://doi.org/10.1515/eng-2021-0048

E. Paxinou, M. Georgiou, V. Kakkos, D. Kalles, and L. Galani, "Achieving educational goals in microscopy education by adopting virtual reality labs on top of face-to-face tutorials," Research in Science & Technological Education, vol. 40, no. 3, pp. 320–339, Jul. 2022. DOI: https://doi.org/10.1080/02635143.2020.1790513

C. J. Dede, J. Jacobson, and J. Richards, "Introduction: Virtual, Augmented, and Mixed Realities in Education," in Virtual, Augmented, and Mixed Realities in Education, D. Liu, C. Dede, R. Huang, and J. Richards, Eds. Singapore: Springer Singapore, 2017, pp. 1–16. DOI: https://doi.org/10.1007/978-981-10-5490-7_1

E. L. Isenstein et al., "Rapid assessment of hand reaching using virtual reality and application in cerebellar stroke," PLOS ONE, vol. 17, no. 9, Sep. 2022, Art. no. e0275220. DOI: https://doi.org/10.1371/journal.pone.0275220

N. Dawood, "VR-roadmap: a vision for 2030 in the built environment," Journal of Information Technology in Construction (ITcon), vol. 14, no. 32, pp. 489–506, Aug. 2009. DOI: https://doi.org/10.1002/9781444302349.ch19

F. G. Santori, "The future of mobility in the metaverse is not (only) what you imagine," InfraJournal. https://www.infrajournal.com/en/w/the-future-of-mobility-in-the-metaverse-is-not-only-what-you-imagine.

M. Alcañiz Raya, J. Marín-Morales, M. E. Minissi, G. Teruel Garcia, L. Abad, and I. A. Chicchi Giglioli, "Machine Learning and Virtual Reality on Body Movements’ Behaviors to Classify Children with Autism Spectrum Disorder," Journal of Clinical Medicine, vol. 9, no. 5, Apr. 2020, Art. no. 1260. DOI: https://doi.org/10.3390/jcm9051260

G. Riva, B. K. Wiederhold, and F. Mantovani, "Neuroscience of Virtual Reality: From Virtual Exposure to Embodied Medicine," Cyberpsychology, Behavior, and Social Networking, vol. 22, no. 1, pp. 82–96, Jan. 2019. DOI: https://doi.org/10.1089/cyber.2017.29099.gri

Y. K. Dwivedi et al., "Metaverse beyond the hype: Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy," International Journal of Information Management, vol. 66, Oct. 2022, Art. no. 102542. DOI: https://doi.org/10.1016/j.ijinfomgt.2022.102542

F. Yang and Y. Miang Goh, "VR and MR technology for safety management education: An authentic learning approach," Safety Science, vol. 148, Apr. 2022, Art. no. 105645. DOI: https://doi.org/10.1016/j.ssci.2021.105645

A. F. Karami, H. Nurhayati, Y. M. Arif, "Design and Evaluation of Maliki V-Lab: A Metaverse-Based Virtual Laboratory for Computer Assembly Learning in Higher Education," International Journal of Information and Education Technology, vol. 14, no. 6, pp. 814–821, 2024. DOI: https://doi.org/10.18178/ijiet.2024.14.6.2106

X. Lei, H. H. Chen, P. L. P. Rau, L. Dong, and X. Liu, "Learning in virtual reality: Effects of instruction type and emotional arousal on learning performance," Learning and Motivation, vol. 80, Nov. 2022, Art. no. 101846. DOI: https://doi.org/10.1016/j.lmot.2022.101846

F. De Lorenzis, F. G. Pratticò, M. Repetto, E. Pons, and F. Lamberti, "Immersive Virtual Reality for procedural training: Comparing traditional and learning by teaching approaches," Computers in Industry, vol. 144, Jan. 2023, Art. no. 103785. DOI: https://doi.org/10.1016/j.compind.2022.103785

M. Coban, Y. I. Bolat, and I. Goksu, "The potential of immersive virtual reality to enhance learning: A meta-analysis," Educational Research Review, vol. 36, Jun. 2022, Art. no. 100452. DOI: https://doi.org/10.1016/j.edurev.2022.100452

R. Liu, B. Becerik-Gerber, and G. M. Lucas, "Effectiveness of VR-based training on improving occupants’ response and preparedness for active shooter incidents," Safety Science, vol. 164, Aug. 2023, Art. no. 106175. DOI: https://doi.org/10.1016/j.ssci.2023.106175

J. N. Bailenson, N. Yee, J. Blascovich, A. C. Beall, N. Lundblad, and M. Jin, "The Use of Immersive Virtual Reality in the Learning Sciences: Digital Transformations of Teachers, Students, and Social Context," Journal of the Learning Sciences, vol. 17, no. 1, pp. 102–141, Feb. 2008. DOI: https://doi.org/10.1080/10508400701793141

H. Song, T. Kim, J. Kim, D. Ahn, and Y. Kang, "Effectiveness of VR crane training with head-mounted display: Double mediation of presence and perceived usefulness," Automation in Construction, vol. 122, Feb. 2021, Art. no. 103506. DOI: https://doi.org/10.1016/j.autcon.2020.103506

A. Menin, R. Torchelsen, and L. Nedel, "The effects of VR in training simulators: Exploring perception and knowledge gain," Computers & Graphics, vol. 102, pp. 402–412, Feb. 2022. DOI: https://doi.org/10.1016/j.cag.2021.09.015

S. Hadam, N. Rahayu, and A. N. Ariyadi, Strategi Implementasi Revitalisasi SMK (10 Langkah Revitalisasi SMK). Jakarta: Direktorat Pembinaan Sekolah Menengah Kejuruan, Direktorat Jenderal Pendidikan Dasar dan Menengah, Kementerian Pendidikan dan Kebudayaan, 2017.

Persyaratan Umum Lembaga Sertifikasi Profesi. BNSP Republik Indonesia, 2014.

Pedoman BNSP 210 Pedoman Pengembangan dan Pemeliharaan Skema Sertifikasi Profesi, Badan Nasional Sertifikasi Profesi, Indonesia, 2017.

L. Bareišytė et al., "Questionnaires for evaluating virtual reality: A systematic scoping review," Computers in Human Behavior Reports, vol. 16, Dec. 2024, Art. no. 100505. DOI: https://doi.org/10.1016/j.chbr.2024.100505

Y. M. Kim and I. Rhiu, "Development of a virtual reality system usability questionnaire (VRSUQ)," Applied Ergonomics, vol. 119, Sep. 2024, Art. no. 104319. DOI: https://doi.org/10.1016/j.apergo.2024.104319

S. C. Peres, T. Pham, and R. Phillips, "Validation of the System Usability Scale (SUS): SUS in the Wild," Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 57, no. 1, pp. 192–196, Sep. 2013. DOI: https://doi.org/10.1177/1541931213571043

J. Baumgartner, N. Frei, M. Kleinke, J. Sauer, and A. Sonderegger, "Pictorial System Usability Scale (P-SUS): Developing an Instrument for Measuring Perceived Usability," in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, UK, May 2019, pp. 1–11. DOI: https://doi.org/10.1145/3290605.3300299

S. Thiagarajan and A. Others, Instructional Development for Training Teachers of Exceptional Children: A Sourcebook. Council for Exceptional Children, 1974.

M. Won et al., "Diverse approaches to learning with immersive Virtual Reality identified from a systematic review," Computers & Education, vol. 195, Apr. 2023, Art. no. 104701. DOI: https://doi.org/10.1016/j.compedu.2022.104701

R. Webster and J. Dues, "System Usability Scale (SUS): Oculus Rift® DK2 and Samsung Gear VR®," in 2017 ASEE Annual Conference & Exposition Proceedings, Columbus, OH, USA, Jun. 2017, Art. no. 28899.

K. Ito, M. Tada, H. Ujike, and K. Hyodo, "Effects of the Weight and Balance of Head-Mounted Displays on Physical Load," Applied Sciences, vol. 11, no. 15, Jul. 2021, Art. no. 6802. DOI: https://doi.org/10.3390/app11156802

T. Chihara and A. Seo, "Evaluation of physical workload affected by mass and center of mass of head-mounted display," Applied Ergonomics, vol. 68, pp. 204–212, Apr. 2018. DOI: https://doi.org/10.1016/j.apergo.2017.11.016

S. Palmisano and R. Constable, "Reductions in sickness with repeated exposure to HMD-based virtual reality appear to be game-specific," Virtual Reality, vol. 26, no. 4, pp. 1373–1389, Dec. 2022. DOI: https://doi.org/10.1007/s10055-022-00634-6

A. Bangor, P. T. Kortum, and J. T. Miller, "An Empirical Evaluation of the System Usability Scale," International Journal of Human-Computer Interaction, vol. 24, no. 6, pp. 574–594, Jul. 2008. DOI: https://doi.org/10.1080/10447310802205776

T. Tullis and B. Albert, "Self-Reported Metrics," in Measuring the User Experience, Elsevier, 2013, pp. 121–161. DOI: https://doi.org/10.1016/B978-0-12-415781-1.00006-6

S. Palmisano, R. Mursic, and J. Kim, "Vection and cybersickness generated by head-and-display motion in the Oculus Rift," Displays, vol. 46, Jan. 2017. DOI: https://doi.org/10.1016/j.displa.2016.11.001

A. Da Silva Marinho, U. Terton, and C. M. Jones, "Cybersickness and postural stability of first time VR users playing VR videogames," Applied Ergonomics, vol. 101, May 2022, Art. no. 103698. DOI: https://doi.org/10.1016/j.apergo.2022.103698

E. Nalivaiko, S. L. Davis, K. L. Blackmore, A. Vakulin, and K. V. Nesbitt, "Cybersickness provoked by head-mounted display affects cutaneous vascular tone, heart rate and reaction time," Physiology & Behavior, vol. 151, pp. 583–590, Nov. 2015. DOI: https://doi.org/10.1016/j.physbeh.2015.08.043