Title
“That’s not what they said!” The impact of mismatches between the dialogue and the intralingual English subtitles on native and foreign viewers
Conference name
Languages & the Media 2024
City
Country
Hungary
Modalities
Date
13/11/2024-15/11/2024
Abstract
Professional subtitlers often implement strategies like text reduction, condensation and omission due to well-known spatial (typically 1-2 lines, 39-42 characters per line) and temporal (reading speed) constraints inherent to the subtitling medium. Despite condensation being a common practice and a source of pride for many subtitlers (Szarkowska et al., 2021), surprisingly little is known about the effects of such strategies on the reading patterns of both native (L1) and non-native (L2) viewers, particularly in intralingual subtitles, where the same language appears in the dialogues and in the subtitles, and therefore the resulting mismatches are naturally more visible.
On one hand, verbatim subtitles are not only generally welcome by deaf and hard-of-hearing viewers (Szarkowska et al., 2011), but also potentially helpful for language learners. Prior research has shown that presenting the foreign language both aurally and in writing helps learners chunk and decode the aural code, recognise known words, understand novel ones, and resolve ambiguity, as subtitles facilitate mapping continuous L2 speech onto separate L2 words (see, e.g., Winke et al., 2013). On the other, with edited (non-verbatim) subtitles, seeing one word while hearing another involves simultaneously processing two mismatched or ‘incongruous’ items competing for attention, which may increase processing effort and possibly be particularly disruptive for L2 viewers, as reading and listening are typically harder in a foreign language.
The present study uses eye tracking to examine whether, and if so how, these common subtle incongruencies affect the reading process, and whether native and non-native viewers process such incongruities differently. To this end, L1-English speakers (UK-based) and L2-English speakers (L1-Polish) watched two English clips with English subtitles from the Netflix TV series “The Chair”. Two versions of each clip were created, one where the same words were heard (auditory input—dialogues) and read (written input—subtitles) and one where a dialogue-subtitle mismatch was introduced. In the mismatch version, one word per subtitle was changed with a synonym or near-synonym that did not substantially alter the meaning of the utterance. We used (near-)synonyms to resemble what subtitlers often do in their professional practice and to avoid introducing deeper semantic discrepancies, which are more likely to attract attention and disrupt reading if the meaning of the overall sentence is changed. To maximise comparability, target pairs (matched vs. mismatched) were always the same part of speech (e.g., nouns or verbs), with comparable length and frequency, appearing in the same subtitle line and position (never the first or last word of a line). Only self-contained sentences that did not span over multiple subtitles were selected, and subtitles had the same number of words and lines. These criteria also allowed us to maintain comparable subtitle speeds in both versions.
To assess the effect of incongruities specifically on subtitle reading, and to understand whether the effect takes place at an early or late stage of processing, several fixation measureswere analysed via mixed-effect models, e.g., first fixation duration (duration of the first fixation on a word), total reading time (duration of all fixations on a word), and go-past time (sum of all fixations from the first fixation on a word up to but not including the first fixation to the right of that word, i.e., including any reinspection time of a word before moving past it).
We also measured viewers’ comprehension, recall, cognitive load, and enjoyment using self-reports (written questionnaires) and semi-structured in-person interviews. Using a two-way mixed design to analyse the questionnaires – with mismatches as the main within-subject variable and language group as the between-subject factor – we found that mismatches negatively affected viewers' cognitive load and enjoyment. However, there was no significant effect of mismatches on comprehension or recall.
The presentation will provide details of the study methodology, present a subset of key results from both the eye-tracking and the questionnaire analyses, and discuss the findings in light of current industry practices and guidelines, as well as the verbatim vs. edited debate in subtitling. The steady increase in subtitling use has led to an expansion of the subtitle user base. Our findings are timely as they come to inform multiple stakeholders (subtitlers, LSPs, producers and trainers, among others) about viewers’ comprehensive reactions to same-language subtitling. As more and more non-native speakers watch original English productions, and growing numbers of English L1 viewers – especially young ones – also watch content with English subtitles (YPulse, 2022), understanding both L1 and L2 processing of subtitled audiovisual content is essential to ensure the production of subtitles optimised for the widest and most diverse audiences.
On one hand, verbatim subtitles are not only generally welcome by deaf and hard-of-hearing viewers (Szarkowska et al., 2011), but also potentially helpful for language learners. Prior research has shown that presenting the foreign language both aurally and in writing helps learners chunk and decode the aural code, recognise known words, understand novel ones, and resolve ambiguity, as subtitles facilitate mapping continuous L2 speech onto separate L2 words (see, e.g., Winke et al., 2013). On the other, with edited (non-verbatim) subtitles, seeing one word while hearing another involves simultaneously processing two mismatched or ‘incongruous’ items competing for attention, which may increase processing effort and possibly be particularly disruptive for L2 viewers, as reading and listening are typically harder in a foreign language.
The present study uses eye tracking to examine whether, and if so how, these common subtle incongruencies affect the reading process, and whether native and non-native viewers process such incongruities differently. To this end, L1-English speakers (UK-based) and L2-English speakers (L1-Polish) watched two English clips with English subtitles from the Netflix TV series “The Chair”. Two versions of each clip were created, one where the same words were heard (auditory input—dialogues) and read (written input—subtitles) and one where a dialogue-subtitle mismatch was introduced. In the mismatch version, one word per subtitle was changed with a synonym or near-synonym that did not substantially alter the meaning of the utterance. We used (near-)synonyms to resemble what subtitlers often do in their professional practice and to avoid introducing deeper semantic discrepancies, which are more likely to attract attention and disrupt reading if the meaning of the overall sentence is changed. To maximise comparability, target pairs (matched vs. mismatched) were always the same part of speech (e.g., nouns or verbs), with comparable length and frequency, appearing in the same subtitle line and position (never the first or last word of a line). Only self-contained sentences that did not span over multiple subtitles were selected, and subtitles had the same number of words and lines. These criteria also allowed us to maintain comparable subtitle speeds in both versions.
To assess the effect of incongruities specifically on subtitle reading, and to understand whether the effect takes place at an early or late stage of processing, several fixation measureswere analysed via mixed-effect models, e.g., first fixation duration (duration of the first fixation on a word), total reading time (duration of all fixations on a word), and go-past time (sum of all fixations from the first fixation on a word up to but not including the first fixation to the right of that word, i.e., including any reinspection time of a word before moving past it).
We also measured viewers’ comprehension, recall, cognitive load, and enjoyment using self-reports (written questionnaires) and semi-structured in-person interviews. Using a two-way mixed design to analyse the questionnaires – with mismatches as the main within-subject variable and language group as the between-subject factor – we found that mismatches negatively affected viewers' cognitive load and enjoyment. However, there was no significant effect of mismatches on comprehension or recall.
The presentation will provide details of the study methodology, present a subset of key results from both the eye-tracking and the questionnaire analyses, and discuss the findings in light of current industry practices and guidelines, as well as the verbatim vs. edited debate in subtitling. The steady increase in subtitling use has led to an expansion of the subtitle user base. Our findings are timely as they come to inform multiple stakeholders (subtitlers, LSPs, producers and trainers, among others) about viewers’ comprehensive reactions to same-language subtitling. As more and more non-native speakers watch original English productions, and growing numbers of English L1 viewers – especially young ones – also watch content with English subtitles (YPulse, 2022), understanding both L1 and L2 processing of subtitled audiovisual content is essential to ensure the production of subtitles optimised for the widest and most diverse audiences.