Title
Cognitive load in multimodal and multilingual text processing. The impact of ASR and post-editing in transcription processes
Conference name
Media for All 8
City
Country
Sweden
Modalities
Date
19/06/2019
Abstract
Substantial gains in quality driven by Neural Machine Translation (NMT) for literary texts (Toral et al. 2018) and advances in Automatic Speech Recognition (ASR) suggest that post-editing could boost productivity while maintaining quality standards also in subtitling. Compass (Computer-Assisted Subtitling), a joint project of ZDF Digital and University of Mainz funded by the European Commission aims at optimizing the multilingual subtitling processes for public TV programs by developing an innovative subtitling platform.
We present results from an eyetracking and keylogging study evaluating transcription and translation processes as two main subprocesses involved in the complex subtitling process. With established gaze and typing measures indicating cognitive effort (de Sousa et al 2011), we investigate the impact of ASR and NMT on students’ and professional subtitlers’ transcription and translation behavior and triangulate the results with data from questionnaires and quality evaluations.
The experiment in which 13 professional translators and 13 translation students performed 8 tasks on 2-minute video clips is motived by the planned pipeline of the Compass tool featuring support via ASR, NMT and translation with English as relay language. The tasks include intralingual transcription with and without ASR, translation from English into German directly from the video, with an additional English ASR script and with a correct English transcript.
Finally, participants performed post-editing on German NMT output of a Swedish TV series previously transcribed and translated into English. To assist their post-editing, they first had both the Swedish video and English transcript, and then either the Swedish video or English transcript only.
We expected ASR to help in intralingual and interlingual transcription but find that only a reliable transcript and the NMT resulted in the expected efficiency gains. Regarding cognitive load, we observed different strategies in the two groups and found access to the images in the video to be essential.
We present results from an eyetracking and keylogging study evaluating transcription and translation processes as two main subprocesses involved in the complex subtitling process. With established gaze and typing measures indicating cognitive effort (de Sousa et al 2011), we investigate the impact of ASR and NMT on students’ and professional subtitlers’ transcription and translation behavior and triangulate the results with data from questionnaires and quality evaluations.
The experiment in which 13 professional translators and 13 translation students performed 8 tasks on 2-minute video clips is motived by the planned pipeline of the Compass tool featuring support via ASR, NMT and translation with English as relay language. The tasks include intralingual transcription with and without ASR, translation from English into German directly from the video, with an additional English ASR script and with a correct English transcript.
Finally, participants performed post-editing on German NMT output of a Swedish TV series previously transcribed and translated into English. To assist their post-editing, they first had both the Swedish video and English transcript, and then either the Swedish video or English transcript only.
We expected ASR to help in intralingual and interlingual transcription but find that only a reliable transcript and the NMT resulted in the expected efficiency gains. Regarding cognitive load, we observed different strategies in the two groups and found access to the images in the video to be essential.