Details

Autor(en) / Beteiligte

• Alsharid, Mohammad
• Cai, Yifan
• Sharma, Harshita
• Drukker, Lior
• Papageorghiou, Aris T.
• Noble, J. Alison

Titel

Gaze-assisted automatic captioning of fetal ultrasound videos using three-way multi-modal deep neural networks

Ist Teil von

• Medical image analysis, 2022-11, Vol.82, p.102630-102630, Article 102630

Ort / Verlag

Elsevier B.V

Erscheinungsjahr

2022

Quelle

Alma/SFX Local Collection

Beschreibungen/Notizen

• In this work, we present a novel gaze-assisted natural language processing (NLP)-based video captioning model to describe routine second-trimester fetal ultrasound scan videos in a vocabulary of spoken sonography. The primary novelty of our multi-modal approach is that the learned video captioning model is built using a combination of ultrasound video, tracked gaze and textual transcriptions from speech recordings. The textual captions that describe the spatio-temporal scan video content are learnt from sonographer speech recordings. The generation of captions is assisted by sonographer gaze-tracking information reflecting their visual attention while performing live-imaging and interpreting a frozen image. To evaluate the effect of adding, or withholding, different forms of gaze on the video model, we compare spatio-temporal deep networks trained using three multi-modal configurations, namely: (1) a gaze-less neural network with only text and video as input, (2) a neural network additionally using real sonographer gaze in the form of attention maps, and (3) a neural network using automatically-predicted gaze in the form of saliency maps instead. We assess algorithm performance through established general text-based metrics (BLEU, ROUGE-L, F1 score), a domain-specific metric (ARS), and metrics that consider the richness and efficiency of the generated captions with respect to the scan video. Results show that the proposed gaze-assisted models can generate richer and more diverse captions for clinical fetal ultrasound scan videos than those without gaze at the expense of the perceived sentence structure. The results also show that the generated captions are similar to sonographer speech in terms of discussing the visual content and the scanning actions performed. [Display omitted] •Captioning describes spatio-temporal ultrasound content with sonographer spoken words•Word2vec embedding outperforms BioWordVec embedding in gaze-assisted captioning•Gaze helps achieve higher scores on evaluation metrics, specifically B1 to B4 and F1