While it is relatively simple to transcribe a few minutes of audio where one person speaks, transcribing several hours of multi-speaker exchanges, with speech overlaps and business terms, is a less straightforward task. Especially when the goal is then to produce a report reliable, where speakers must be continuously identified (speaker recognition is called diarisation).
In this article, we will see why classic transcriptions have their limits, and how to combine different approaches to obtain a transcription that is both consistent, precise, and diarised.
Why classic transcriptions can fail
To address the problem of transcription in general, two families of models exist:
Continuous Automatic Speech Recognition (ASR)
On one hand, we have continuous ASR (Automatic Speech Recognition) systems, which only have access to a local audio context for the recording, and therefore rely mainly on phonetic / short-term aspects. They have stable performance with the length of the recording. This also makes them fallible: some phonetically similar expressions can be difficult to discriminate without global semantic information.
These systems can precisely account for the temporality of each word, which allows for syncing the text with the audio, but especially for coupling them with a diarisation, which allows each utterance to be attributed to a speaker, consistently over long durations. A timestamp and an identity can thus be assigned to any utterance.
A list of keywords can optionally be provided to artificially increase the probability of certain words, but this only allows for limited consideration of the external context to the recording.
In summary: average to good performance, but independent of input length, with reliable long-term diarisation.
Simplified diagram of a continuous ASR model
Large Multimodal Language Model (MLLM)
On the other hand, we have multimodal LLMs, which take into account the entire context to transcribe each word, and thus ensure a long-term contextual consistency.
Their strong performance in language-related tasks ensures a subtle understanding of the recording and avoids numerous misunderstandings. They can be enriched with comprehensive additional information and also perform accurate diarisation.
However, their performance degrades with input length:
When the context becomes too rich, predictions become noisy.
Some precise information may be lost, and generation quality deteriorates.
The cost of inference increases with input size.
Here, interventions cannot be precisely timestamped. It is possible to request timestamps, but they will very often be hallucinated by the model.
In short: excellent performance on moderate contexts, but which degrades and becomes costly on long recordings.
Simplified diagram of a MLLM in the context of transcription
It can be noted that ASR systems alone often have much better latency than multimodal LLMs and are therefore often more appropriate for real-time transcription. In our case, latency is not an issue, so this criterion is not taken into account.
Rather than finding a compromise between the two, both systems can be used simultaneously.
Step 1: Chunking for the MLLM
To maintain optimal MLLM performance, the recording is split into fixed-size chunks (e.g., 10-15 minutes), with overlap (overlap), each processed separately, by injecting global contextual information each time.
Thus, the model can use rich contextual information without the input length degrading its quality. It is asked to provide a diarised transcription quite freely.
Limitation: long-term tracking of speaker identities is lost, as chunks are processed independently: it is impossible to know if speaker n°2 from the first chunk is the same as speaker n°2 from the last chunk.
Step 2: Full analysis via continuous ASR + diarisation
For the continuous ASR system with diarisation, the audio can be analysed in one go. The result will take the form of segments, associated with a speaker and a timestamp. These timestamps allow the result to be split into chunks synchronised with those of the MLLM.
Step 3: Chunk-by-chunk merging
These synchronised chunks are then merged two by two with a lightweight and inexpensive LLM.
Merging chunks corresponding to the same portion of the recording:
Step 4: Final merging
Finally, all these improved chunks are merged to produce a unified transcription.
A lightweight LLM is used again to limit resource consumption.
Adjacent chunks are merged two by two, with overlap to facilitate transitions.
The model receives the instruction to produce a single text having a common prefix with the beginning of the previous context, and a common suffix with the end of the next context.
The overlap between chunks ensures that no information is lost in the process
Result: a reliable, consistent, and diarised transcription over the entire duration.
Merging two consecutive chunks, so that all chunks can then be easily merged:
Merging all chunks:
Cost analysis
The transcription and merging methods are independent of input length, so the quality remains constant. The cost is also linear with input length, making the process viable for arbitrarily long recordings. The MLLM cost is dominant overall, with the rest adding about 30-50%. From an economic standpoint, we are looking at around 0.50€ - 1€ per hour of audio.
In terms of resource consumption, we are below a single call to an MLLM (as one might do on a chatbot) when the recording lasts several hours. Indeed, MLLMs (and LLMs in general) have a cost per token that increases with input or output length. By restricting to 10-15 minute chunks, token consumption remains moderate.
In practice, some providers for the ASR + diarisation part limit processing to 4–6 hours of audio, which can be a limitation. An analogous system can be manually designed using the pyannote library and a local model.
Metrics
The different methods were compared on the FLEURS fr dataset (short recordings with a single speaker) to highlight the quality transfer from the MLLM to our approach. And on the SUMM-RE dataset (meetings with multiple speakers), where the number of errors was counted using LLM-as-a-judge.
FLEURS - FR
| Mode | Raw WER |
|---|
| azure speech recognition (diarised) | 9.6% |
| gemini 3 flash | 3.1% |
| dual transcription (ours) | 3.1% |
SUMM-RE - FR
| Mode | semantic discrepancies (minor) | semantic discrepancies (major) | speaker attribution precision |
|---|
| azure speech recognition (diarised) | 12 | 2 | 86% |
| gemini 3 flash | 3 | 0 | |
| dual transcription (ours) | 2 | 0 | 88% |
Conclusion
The method described here allows the transcription quality achievable on short audio to be transposed to arbitrarily long recordings. This comes with a substantial increase in costs, but which should be put into perspective with the total cost of the system, of which transcription is only one step. Thus, many errors that would be incorrigible when moving from transcription to the final report are avoided.
Best practices
Include an overlap between chunks to facilitate merging (around 30s - 1 minute).
Use chunks long enough for the MLLM to benefit from contextual information, but short enough not to degrade its performance (10-20 mins is a good compromise).
If possible, set the temperature to a value close to 0 in the MLLM to limit noise
A simpler system than that will be a better option if the aim is to transcribe short audios (a few tens of minutes).
