Ensuring competence in ultrasound-guided procedures—a validity study of a newly developed assessment tool

Niklas Kahr Rasmussen*, Jonathan Frederik Carlsen, Beth Hærstedt Olsen, Dorte Stærk, Trine Lise Lambine, Birthe Henriksen, Maja Rasmussen, Mattis Jørgensen, Elisabeth Albrecht-Beste, Lars Konge, Michael Bachmann Nielsen, Leizl Joy Nayahangan

*Corresponding author af dette arbejde

Publikation: Bidrag til tidsskriftTidsskriftartikelForskningpeer review

3 Citationer (Scopus)

Abstract

Objectives: To investigate the validity of the Interventional Ultrasound Skills Evaluation (IUSE) tool for assessment of procedural competence in ultrasound-guided procedures in a clinical environment, including a pass/fail score. Methods: Novices and experienced radiologists were recruited from four hospitals and were observed and assessed while performing ultrasound-guided procedures. Performances were assessed using the IUSE tool by two independent raters. Validity evidence was gathered in accordance with Messick’s framework: response process was ensured by standardisation of written rater instructions. Internal structure was explored using Cronbach’s alpha for internal consistency reliability; inter-rater reliability was calculated as Pearson’s r independently across all ratings, and test-retest reliability was reported using Cronbach’s alpha. Relationship to other variables was investigated by comparing performances of the participants in each group. Consequences evidence was explored by calculating a pass/fail standard using the contrasting groups method. Results: Six novices and twelve experienced radiologists were enrolled. The IUSE tool had high internal consistency (Cronbach’s alpha = 0.96, high inter-rater reliability (Pearson’s r = 0.95), and high test-retest reliability (Cronbach’s alpha = 0.98), and the mean score was 33.28 for novices and 59.25 for experienced with a highly significant difference (p value < 0.001). The pass/fail score was set at 55 resulting in no false positives or false negatives. Conclusions: Validity evidence from multiple sources supports the use of the IUSE tool for assessment of competence in ultrasound-guided procedures in a clinical environment and its use in high-stakes assessment such as certification. A credible pass/fail criterion was established to inform decision-making. Key Points: • A multi-site validity investigation established that the Interventional Ultrasound Skills Evaluation (IUSE) tool can be used to assess procedural competence in ultrasound-guided procedures. • Validity evidence was gathered according to Messick’s framework validity from the following sources: response process, internal structure, relationship to other variables, and consequences evidence. • The IUSE tool can be used for both formative and summative assessment, and a credible pass/fail score was established to help inform decision-making such as certification.

OriginalsprogEngelsk
TidsskriftEuropean Radiology
Vol/bind32
Udgave nummer7
Sider (fra-til)4954-4966
Antal sider13
ISSN0938-7994
DOI
StatusUdgivet - 2022

Bibliografisk note

Correction: 10.1007/s00330-022-08542-8
Publisher Copyright:
© 2022, The Author(s), under exclusive licence to European Society of Radiology.

Citationsformater