Details

Autor(en) / Beteiligte

• Thoma, Brent, MD, MA
• Sebok-Syer, Stefanie S., PhD
• Krishnan, Keeth, BHSc
• Siemens, Marshall
• Trueger, N. Seth, MD, MPH
• Colmers-Gray, Isabelle, MD, MSc
• Woods, Rob, MD, MMEd
• Petrusa, Emil, PhD
• Chan, Teresa, MD, MHPE

Titel

Individual Gestalt Is Unreliable for the Evaluation of Quality in Medical Education Blogs: A METRIQ Study

Ist Teil von

• Annals of emergency medicine, 2017-09, Vol.70 (3), p.394-401

Ort / Verlag

United States: Elsevier Inc

Erscheinungsjahr

2017

Quelle

MEDLINE

Beschreibungen/Notizen

• Study objective Open educational resources such as blogs are increasingly used for medical education. Gestalt is generally the evaluation method used for these resources; however, little information has been published on it. We aim to evaluate the reliability of gestalt in the assessment of emergency medicine blogs. Methods We identified 60 English-language emergency medicine Web sites that posted clinically oriented blogs between January 1, 2016, and February 24, 2016. Ten Web sites were selected with a random-number generator. Medical students, emergency medicine residents, and emergency medicine attending physicians evaluated the 2 most recent clinical blog posts from each site for quality, using a 7-point Likert scale. The mean gestalt scores of each blog post were compared between groups with Pearson’s correlations. Single and average measure intraclass correlation coefficients were calculated within groups. A generalizability study evaluated variance within gestalt and a decision study calculated the number of raters required to reliably (>0.8) estimate quality. Results One hundred twenty-one medical students, 88 residents, and 100 attending physicians (93.6% of enrolled participants) evaluated all 20 blog posts. Single-measure intraclass correlation coefficients within groups were fair to poor (0.36 to 0.40). Average-measure intraclass correlation coefficients were more reliable (0.811 to 0.840). Mean gestalt ratings by attending physicians correlated strongly with those by medical students ( r =0.92) and residents ( r =0.99). The generalizability coefficient was 0.91 for the complete data set. The decision study found that 42 gestalt ratings were required to reliably evaluate quality (>0.8). Conclusion The mean gestalt quality ratings of blog posts between medical students, residents, and attending physicians correlate strongly, but individual ratings are unreliable. With sufficient raters, mean gestalt ratings provide a community standard for assessment.