Avoiding Catastrophic Forgetting in Text-Only Adaptation of LLM-based ASR via Multi-View Text Denoising

Quick Links: The idea · Reproducing the paper · Expected results · Scripts · Installation · Citation

This repo contains the code to reproduce our Interspeech 2026 paper (pdf).

In LLM-based ASR, an speech encoder is wired to an LLM by one trainable module, the projector (also called a "connector" or "speech adapter"), which learns to map speech into the LLM's input embedding space. The LLM reads those embeddings in place of tokens and writes out the transcript.

But what do those embeddings actually look like to the LLM? They are continuous vectors, not tokens, so we asked which vocabulary entry each one lands closest to. For the utterance "yes that would be":

audio input         🔊 "yes that would be"
      |                         |
      v                         v
projector output    📝 "mmy Z YesssS S SGS that Will B be S S"
                       (each embedding mapped to its nearest LLM vocabulary entry)
      |                         |
      v                         v
LLM output          📝 "yes that would be"

It reads like a garbled version of what was said: the fragments in bold survive intact, while "would" comes through as "Will B".

This suggests the LLM may be treating speech recognition as something closer to cleaning up corrupted text. We take that as our working hypothesis and adapt on its terms: corrupt target-domain transcripts with a textual noise function, noise(transcript), and train the LLM to recover them. The new domain is learned from text alone, with no target-domain audio anywhere in the pipeline.

Training on that text by itself would eventually pull the LLM away from what the projector actually emits, collapsing the speech-text alignment it depends on: the catastrophic forgetting in the title. Each batch therefore mixes four views of the data, three drawn from the source domain the base model already knows and one from the target domain being adapted to, so the LLM keeps denoising real projector output while it picks up the new domain. The only audio involved is source-domain audio you already have.

Batch composition used for fine-tuning the LLM during text-only adaptation to a target domain. Here, σ_a, σ_ta and σ_t denote the source-domain batch proportions, and τ denotes the target-domain proportion.

Check out our paper for all the details. The code here does more than reproduce the tables: every parameter is exposed as an environment variable, so you can change any part of the setup and try your own. Want to play with the batch proportions shown above, for instance? Those knobs are listed in scripts/README.md.

Text alone still leaves a gap to fine-tuning on real target audio. Our follow-up Interspeech 2026 paper asks how much target audio it takes to close it. The answer: surprisingly little. Mixing just 10% of the target audio into these same batches matches or beats fine-tuning on all of it. See README-speech-text-gap.md to reproduce it with this code.

Everything runs in one conda environment:

conda env create -f environment.yml
conda activate slam_llm

The scripts/ folder holds one sub-folder per table in the paper:

The *-fang-et-al and *-ma-et-al folders alongside tables 2 to 4 reproduce the two baselines.

Every folder runs the same way. Taking Table 3 and its agriculture target as an example, once the environment is ready a whole row is four steps:

# 1. point the scripts at your speech encoder and LLM
#    (edit the "Model and data paths" block)
$EDITOR scripts/common/config_defaults.sh

# 2. point the bundled SlideSpeech splits at your copy of the audio
bash scripts/prepare_slidespeech.sh /path/to/slidespeech/audio

# 3. run one target domain end to end
TARGET_DOMAIN=agriculture bash scripts/table3/run_all.sh

# 4. read the WER once the jobs finish
cat exp/table3/TOA/prompt_llama/agriculture_tau_0.47/epoch_5/WER.txt

Step 3 queues the whole pipeline (base training, base decoding, adaptation, final decoding) as SLURM jobs chained by dependency, and skips any stage whose output already exists. The base model is therefore trained once per table and reused by every row, including the baselines.

Each folder has its own README with the exact commands for its rows, and scripts/README.md walks through the whole workflow and every option you can change.

Training data is a JSONL file with one utterance per line:

{"key": "agriculture_0001-00000", "source": "/path/to/agriculture_0001-00000.wav", "target": "AND THE LAST BUT NOT LEAST"}

During text-only adaptation only the target field of the target-domain file is read; its source paths are never opened, so they may point at audio you do not have.

SlideSpeech. The exact partitions used in the paper ship with this repository under data/slidespeech/. No audio is distributed: those files contain only our domain splits, the original utterance identifiers, and the ground-truth transcripts after our preprocessing and normalisation, included to the minimal extent needed for reproducibility. The transcript text remains subject to the original corpus license and terms of use. Download the corpus from the SlideSpeech project page, then point the files at your copy of the audio:

bash scripts/prepare_slidespeech.sh /path/to/slidespeech/audio

See data/slidespeech/README.md for how the partitions were built, the expected audio layout, and the citation.

DefinedAI. A commercial corpus (defined.ai) that we cannot redistribute. Prepare your own JSONL files in the format above and set the paths in scripts/table{2,4,5,6}*/config.sh.

• Speech encoder: WavLM-Large
• LLM: Llama-3.2-3B-Instruct

Point DEFAULT_SPEECH_ENCODER_PATH and DEFAULT_LLM_PATH at your copies in scripts/common/config_defaults.sh and you are set.

Word error rate (%) and relative improvement Δ (%) over the corresponding base model, so you can check a run against the paper without opening it. Every run writes its own WER.txt next to the checkpoint it evaluated.

Table 2, in-domain (scripts/table2/), DefinedAI source B/I/H:

Table 3, out-of-domain (scripts/table3/), SlideSpeech source L/T/E:

Table 4, cross-domain (scripts/table4/), DefinedAI source, SlideSpeech targets:

The two ablations (scripts/table5/, scripts/table6/) report their numbers in the respective folder READMEs.

This repository is built on top of idiap/llm-asr-prompt, the code for our ICASSP 2026 paper on prompt sensitivity in LLM-based ASR, which in turn extends the SLAM-LLM framework (specifically its ASR example) and ships a modified copy of the HuggingFace PEFT library.

The LLM-based ASR setup itself, the prompt configuration syntax used in conf/, the training pipeline and the PEFT modifications all come from there, and that repository documents them.

What this repository adds is the text-only adaptation method:

• train_config.use_text_only_adaptation (bool) and train_config.text_only_adaptation_config (tau_t, tau_a, sigma_a, sigma_ta, sigma_t, noise_type), which drive the multi-view batch composition described above.
• dataset_config.train_aux_data_path: the source-domain audio-text data mixed into every adaptation batch.
• Noise functions and batching in slam_llm/datasets/speech_dataset.py: noise_text(), plus SpeechAndTextDatasetJsonl and SpeechAuxDatasetJsonl, which sample target- and source-domain items by τ and by the three σ's, and merge_batch(), which joins the two streams.
• Audio-free batch items: the collator and the WavLM forward pass now accept batches in which some items carry no audio at all.
• Deterministic training: seeds for Python, NumPy and CUDA, and torch.use_deterministic_algorithms(True).

If you find our work useful, please consider citing it:

@inproceedings{burdisso26_interspeech,
  title     = {{Avoiding Catastrophic Forgetting in Text-Only Adaptation of LLM-based ASR via Multi-View Text Denoising}},
  author    = {Sergio Burdisso and Esaú Villatoro-Tello and Thibault Bañeras-Roux and Shashi Kumar and Srikanth Madikeri and Pradeep Rangappa and Manjunath K E and Petr Motlicek and Andreas Stolcke},
  year      = {2026},
  booktitle = {{Interspeech 2026}},
  pages     = {4597--4602},
  doi       = {10.21437/Interspeech.2026-3422},
  issn      = {2958-1796},
}

If you use the SlideSpeech partitions shipped here, please also cite the original corpus; the entry is in data/slidespeech/README.md.

This repository is REUSE compliant: every file states its copyright and license, either in its own header or in REUSE.toml, and the full texts live in LICENSES/.

Our own additions (the text-only adaptation method, its multi-view batching, the noise functions, and the related configuration, scripts and documentation) are licensed under the MIT License. Other parts keep the license they arrived with:

The SlideSpeech partitions carry transcripts derived from the SlideSpeech corpus, so they keep its CC BY-SA 4.0 license. Redistributing them, or anything derived from them, means doing so under CC BY-SA 4.0 with attribution to the SlideSpeech authors. That covers the data files; the code alongside them is MIT. See data/slidespeech/README.md.

Files we modified carry a header listing the changes; only those listed modifications are ours.