Title
Live subtitling quality on Polish news TV across six quality metrics
Conference name
Media for All 10 Conference
City
Country
Belgium
Modalities
Date
06/07/2023-07/07/2023
Abstract
The past few years saw a growth in how much live subtitling is available in Poland across television, live events and online streaming. As 108 Polish TV channels regulated by National Broadcasting Council increase their provision of subtitles, with the overall mandatory quota for access services set to rise to 50% by 2024, many channels will have to start providing live subtitles and continue doing so at scale, sustaining and improving quality so as to make sure that this service addresses the needs of the viewers. Now that the quantity of live subtilting is getting quite high, the focus should turn to ensuring fit-for-purpose quality.
The objective of this research was to investigate the quality of live subtitling in Polish on news TV channels. The following research questions have been posed: (1) Do the Deaf and the hard of hearing viewers have effective access to news television thanks to live subtitling, that is, whether subtitles reflect the content accurately and are intelligible? (2) Does the quality of live subtitling differ between broadcasters? (3) Does the quality differ between live subtitling and semi-live subtitling? (4) Does the quality increase or decrease over time? It was hypothesised that semi-live subtitling delivers better quality than live subtitling and that the quality of live subtitling is increasing over time as broadcasters gain more experience in providing live subtitles. The research was carried out by the Media Accessibility Observatory at the University of Warsaw in partnership with Dostepni.eu and the Polish National Broadcasting Council.
The quality of live and semi-live subtitles was analysed based on samples collected from 96 TV shows from three Polish news TV channels (TVP Info, Polsat News and TVN24) across three quarters (Q2 2021, Q3 2021 and Q1 2022). The total duration of all samples was 970 minutes and they included 13,620 subtitles. The study used established metrics of live subtitling quality: NER score (as a measure of accuracy), latency, subtitle speed and reduction rate. Two new metrics were proposed as well: segmentation score and gaps between consecutive subtitles. The speech rate in the subtitled TV shows was also measured. The results were analysed with various statistical methods including analysis of variance, analysis of covariance, correlation and linear mixed models.
- The study has found that the quality of live subtitling is quite diverse and varies between broadcasters. While the results have shown that respeaking with parallel correction is an effective method of creating live subtitles in Polish and it can be used in the broadcasting settings to obtain acceptable quality results, live subtitles for some of the TV shows had substandard quality. The subtitles were presented at excessive subtitle speeds and with high latency, Also, they were inaccurate and at times unintelligible. The study has confirmed that semi-live subtitling produces better quality results than live subtitling. However, the quality of both live and semi-live subtitling is decreasing over time in the case of all three news TV channels. On a 10-point scale, across the three quarters analysed in the study, the accuracy of live subtitling has decreased from acceptable 5/10 to substandard 2/10 for both TVP Info and Polsat News, while it has stayed at 0/10 in the case of TVN24. The latency has increased; so have the subtitle speeds as well as the number of text segmentation errors.
As a comprehensive review of multiple dimensions of live subtitling quality on Polish news TV channels, the study may be the first step towards a regular monitoring of the quality of live subtitling on Polish TV. It can also serve as an inspiration on how multiple dimensions of live subtitling quality can be measured.
The objective of this research was to investigate the quality of live subtitling in Polish on news TV channels. The following research questions have been posed: (1) Do the Deaf and the hard of hearing viewers have effective access to news television thanks to live subtitling, that is, whether subtitles reflect the content accurately and are intelligible? (2) Does the quality of live subtitling differ between broadcasters? (3) Does the quality differ between live subtitling and semi-live subtitling? (4) Does the quality increase or decrease over time? It was hypothesised that semi-live subtitling delivers better quality than live subtitling and that the quality of live subtitling is increasing over time as broadcasters gain more experience in providing live subtitles. The research was carried out by the Media Accessibility Observatory at the University of Warsaw in partnership with Dostepni.eu and the Polish National Broadcasting Council.
The quality of live and semi-live subtitles was analysed based on samples collected from 96 TV shows from three Polish news TV channels (TVP Info, Polsat News and TVN24) across three quarters (Q2 2021, Q3 2021 and Q1 2022). The total duration of all samples was 970 minutes and they included 13,620 subtitles. The study used established metrics of live subtitling quality: NER score (as a measure of accuracy), latency, subtitle speed and reduction rate. Two new metrics were proposed as well: segmentation score and gaps between consecutive subtitles. The speech rate in the subtitled TV shows was also measured. The results were analysed with various statistical methods including analysis of variance, analysis of covariance, correlation and linear mixed models.
- The study has found that the quality of live subtitling is quite diverse and varies between broadcasters. While the results have shown that respeaking with parallel correction is an effective method of creating live subtitles in Polish and it can be used in the broadcasting settings to obtain acceptable quality results, live subtitles for some of the TV shows had substandard quality. The subtitles were presented at excessive subtitle speeds and with high latency, Also, they were inaccurate and at times unintelligible. The study has confirmed that semi-live subtitling produces better quality results than live subtitling. However, the quality of both live and semi-live subtitling is decreasing over time in the case of all three news TV channels. On a 10-point scale, across the three quarters analysed in the study, the accuracy of live subtitling has decreased from acceptable 5/10 to substandard 2/10 for both TVP Info and Polsat News, while it has stayed at 0/10 in the case of TVN24. The latency has increased; so have the subtitle speeds as well as the number of text segmentation errors.
As a comprehensive review of multiple dimensions of live subtitling quality on Polish news TV channels, the study may be the first step towards a regular monitoring of the quality of live subtitling on Polish TV. It can also serve as an inspiration on how multiple dimensions of live subtitling quality can be measured.