Evidence-based assessment of acupuncture skills using a newly developed simulation-based test: A validation study

Objective: Acupuncture is an efficient treatment modality for many different training, but the technique can be challenging to master. Identification of the different acupuncture points can now be practiced in a standardized and patient-safe simulation-based environment. We aim to gather additional evidence regarding simulation-based training of acupuncture skills and hope that the simulation-based test developed and will be used for simulation-based mastery learning in future develop. Methods: The aim of this study was to develop a test of acupuncture skills using the TCM (Traditional Chinese Medicine) Meridians & Acupoints Anatomy Teaching Platform and gather validity evidence using Messick's framework. Novice medical students without prior experience in acupuncture (n = 43), intermediate medical students who had finished the mandatory acupuncture course (n = 24), and skilled acupuncture doctors (n = 17) were included in the study. One-hundred and seven acupuncture points were presented to the participants in identical randomized order and they were asked to identify each point by touching the correct anatomical location on the simulator. The simulator automatically registered the accuracy of participants’ answers. Results: The reliability of the final test (66 acupuncture points) was at a high level with a Cronbach's alpha of 0.98 and only 16 acupuncture points were needed to ensure a reliability of 0.90. The test could discriminate significantly between all three groups and a credible pass/fail-score of 34 points was determined which resulted in eight out of 43 (19%) novices passing the test and only two out of 17 (12%) experienced failing the test. These consequences were statistically significant (P < 0.001). Conclusion: Based on Messick's framework we demonstrated strong validity evidence for a newly developed simulation-based test of acupuncture points localization skills. The test can be used as an aid to decide when trainees are ready to progress to more advanced training and clinical practice. Compared with the traditional point-point test, the point-point evaluation of the simulator can be more objective and reduce the effort of teachers.