TY - JOUR
T1 - Ensuring competence in ultrasound-guided procedures—a validity study of a newly developed assessment tool
AU - Rasmussen, Niklas Kahr
AU - Carlsen, Jonathan Frederik
AU - Olsen, Beth Hærstedt
AU - Stærk, Dorte
AU - Lambine, Trine Lise
AU - Henriksen, Birthe
AU - Rasmussen, Maja
AU - Jørgensen, Mattis
AU - Albrecht-Beste, Elisabeth
AU - Konge, Lars
AU - Nielsen, Michael Bachmann
AU - Nayahangan, Leizl Joy
N1 - Correction: 10.1007/s00330-022-08542-8
Publisher Copyright:
© 2022, The Author(s), under exclusive licence to European Society of Radiology.
PY - 2022
Y1 - 2022
N2 - Objectives: To investigate the validity of the Interventional Ultrasound Skills Evaluation (IUSE) tool for assessment of procedural competence in ultrasound-guided procedures in a clinical environment, including a pass/fail score. Methods: Novices and experienced radiologists were recruited from four hospitals and were observed and assessed while performing ultrasound-guided procedures. Performances were assessed using the IUSE tool by two independent raters. Validity evidence was gathered in accordance with Messick’s framework: response process was ensured by standardisation of written rater instructions. Internal structure was explored using Cronbach’s alpha for internal consistency reliability; inter-rater reliability was calculated as Pearson’s r independently across all ratings, and test-retest reliability was reported using Cronbach’s alpha. Relationship to other variables was investigated by comparing performances of the participants in each group. Consequences evidence was explored by calculating a pass/fail standard using the contrasting groups method. Results: Six novices and twelve experienced radiologists were enrolled. The IUSE tool had high internal consistency (Cronbach’s alpha = 0.96, high inter-rater reliability (Pearson’s r = 0.95), and high test-retest reliability (Cronbach’s alpha = 0.98), and the mean score was 33.28 for novices and 59.25 for experienced with a highly significant difference (p value < 0.001). The pass/fail score was set at 55 resulting in no false positives or false negatives. Conclusions: Validity evidence from multiple sources supports the use of the IUSE tool for assessment of competence in ultrasound-guided procedures in a clinical environment and its use in high-stakes assessment such as certification. A credible pass/fail criterion was established to inform decision-making. Key Points: • A multi-site validity investigation established that the Interventional Ultrasound Skills Evaluation (IUSE) tool can be used to assess procedural competence in ultrasound-guided procedures. • Validity evidence was gathered according to Messick’s framework validity from the following sources: response process, internal structure, relationship to other variables, and consequences evidence. • The IUSE tool can be used for both formative and summative assessment, and a credible pass/fail score was established to help inform decision-making such as certification.
AB - Objectives: To investigate the validity of the Interventional Ultrasound Skills Evaluation (IUSE) tool for assessment of procedural competence in ultrasound-guided procedures in a clinical environment, including a pass/fail score. Methods: Novices and experienced radiologists were recruited from four hospitals and were observed and assessed while performing ultrasound-guided procedures. Performances were assessed using the IUSE tool by two independent raters. Validity evidence was gathered in accordance with Messick’s framework: response process was ensured by standardisation of written rater instructions. Internal structure was explored using Cronbach’s alpha for internal consistency reliability; inter-rater reliability was calculated as Pearson’s r independently across all ratings, and test-retest reliability was reported using Cronbach’s alpha. Relationship to other variables was investigated by comparing performances of the participants in each group. Consequences evidence was explored by calculating a pass/fail standard using the contrasting groups method. Results: Six novices and twelve experienced radiologists were enrolled. The IUSE tool had high internal consistency (Cronbach’s alpha = 0.96, high inter-rater reliability (Pearson’s r = 0.95), and high test-retest reliability (Cronbach’s alpha = 0.98), and the mean score was 33.28 for novices and 59.25 for experienced with a highly significant difference (p value < 0.001). The pass/fail score was set at 55 resulting in no false positives or false negatives. Conclusions: Validity evidence from multiple sources supports the use of the IUSE tool for assessment of competence in ultrasound-guided procedures in a clinical environment and its use in high-stakes assessment such as certification. A credible pass/fail criterion was established to inform decision-making. Key Points: • A multi-site validity investigation established that the Interventional Ultrasound Skills Evaluation (IUSE) tool can be used to assess procedural competence in ultrasound-guided procedures. • Validity evidence was gathered according to Messick’s framework validity from the following sources: response process, internal structure, relationship to other variables, and consequences evidence. • The IUSE tool can be used for both formative and summative assessment, and a credible pass/fail score was established to help inform decision-making such as certification.
KW - Certification
KW - Education, medical
KW - Educational measurement
KW - Reproducibility of results
KW - Ultrasonography, interventional
UR - https://link.springer.com/article/10.1007/s00330-022-08695-6
U2 - 10.1007/s00330-022-08542-8
DO - 10.1007/s00330-022-08542-8
M3 - Journal article
C2 - 35195745
AN - SCOPUS:85125346756
VL - 32
SP - 4954
EP - 4966
JO - European Radiology
JF - European Radiology
SN - 0938-7994
IS - 7
ER -