Details

Autor(en) / Beteiligte

• Palea, Dustin
• Vadhul, Giridhar
• Lee, David T

Titel

Annota: Peer-based AI Hints Towards Learning Qualitative Coding at Scale

Ist Teil von

• Proceedings of the 29th International Conference on Intelligent User Interfaces, 2024, p.455-470

Ort / Verlag

New York, NY, USA: ACM

Erscheinungsjahr

2024

Quelle

ACM Digital Library Complete

Beschreibungen/Notizen

• Learning qualitative analysis requires personalized feedback and in-depth discussion not possible for educators to provide in a large course, resulting in many students obtaining only a shallow exposure to qualitative user research and interpretative skills. To overcome this challenge, we introduce a learnersourcing method that builds on the Dawid-Skene expectation maximization (EM) algorithm to generate peer-based AI hints that support students in one aspect of qualitative analysis: determining what sentences are relevant to the research question. After one annotation round, class-wide annotations are used to predict relevant sentences and to generate hints prompting students to revisit missed or incorrectly annotated sentences. An in-the-wild deployment within a large course (N=122) showed that our algorithm converged to comparatively high accuracy despite noisy student labels, and after only ∼ 20 students. An analysis of student interviews found that peer-based AI hints helped improve understanding of research questions, led to more careful examination of transcript annotations, and improved understanding of when they were over-annotating or under-annotating.