Code-Switched Audio: How to Transcribe and Translate Mixed-Language Conversations

For production speech-to-text workflows, see our AI Transcription service for the delivery path behind the guidance in this article.

Code-switching occurs when speakers alternate between languages within a conversation, utterance or even a sentence. It is common in multilingual communities and workplaces, yet it remains one of the hardest cases for automatic speech recognition and translation.

The challenge is that the system cannot assume one language model controls the entire utterance.

Do not normalize mixed-language speech into a monolingual transcript before the source has been preserved.

What code-switching is

A speaker alternates languages.

Example:

“I sent the invoice jana, so please check your email.”

This may be natural bilingual speech.

It is not necessarily a transcription error.

Inter-sentential switching

Language changes between sentences.

Easier:

English sentence → Swahili sentence.

Segment-level LID can handle this.

Intra-sentential switching

Language changes inside one sentence.

Harder:

grammar and vocabulary interact.

A model may not know which tokenizer/language model to prioritize.

Borrowing versus code-switching

Borrowed terms can become part of a language.

Example:

technology terms.

Research annotation may distinguish:

• established borrowing;
• active language switch.

For ordinary transcription, exact sociolinguistic labeling may not be necessary.

Why monolingual ASR fails

A monolingual English model may force Swahili words into similar-sounding English words.

The transcript becomes fluent-looking nonsense.

Use multilingual ASR.

Multilingual is not enough

Many multilingual models are trained on:

• many languages;
• mostly monolingual examples.

DECM research notes that code-switched spontaneous speech remains challenging even for multilingual models.

Explicit code-switch training data matters.

Language identification challenge

2024 EACL research found code-switched LID harder than common assumptions and reported no evaluated approach was adequate for the broad realistic task they formulated.

This means production systems should expose uncertainty.

Candidate languages

If you know the pair:

• English + Hindi;
• English + Swahili.

Provide it.

A known pair is easier than open-world detection.

Token-level language tags

For research/QA, annotate:

[EN] I sent the invoice [SW] jana [EN] so check email.

This supports:

• language-specific WER;
• switch analysis.

For client-readable output, tags may be hidden.

Transcription first

Create source transcript preserving:

• original words;
• script.

Do not translate minority-language terms while transcribing.

Transcription should answer “what was said,” not “what does it mean.”

Native script

If speaker uses Hindi, record Devanagari when the project expects native orthography.

Romanization can lose distinctions.

Define script policy.

Punctuation

Code-switching can confuse punctuation.

Review sentence boundaries.

Translation needs logical sentence/turn context.

Speaker diarization

Code-switching and multi-speaker are separate.

Store:

• speaker;
• language spans.

A person can use two languages.

Do not infer language = speaker.

Translation strategy

Possible target:

All speech → English.

Translate each source span according to meaning, but generate one natural target turn.

Do not translate fragments separately and concatenate them blindly.

Preserve intentional source words

Sometimes a code-switched word carries identity/emphasis.

Research or literary translation may preserve it with explanation.

Business translation may normalize it.

Define use case.

Named entities

Names can resemble code-switches.

Use entity list.

Avoid translating organization names.

Terminology

Technical teams often keep English terms inside another language.

Treat recurring product vocabulary as glossary items rather than detecting every term as a language switch.

Human review

Ideal reviewer understands both languages.

If not available:

• Language A reviewer;
• Language B reviewer;
• adjudicator.

For three-language conversation, review complexity grows.

Error types

Language substitution

Foreign word becomes dominant-language word.

Omission

Minority phrase dropped.

Language-tag error

Text correct but language label wrong.

Translation flattening

Target loses contrast/emphasis of switch.

Track these separately.

Code-switch WER

Standard WER can be calculated on full transcript.

Also compute:

• WER by language;
• code-switched word error rate.

Minority language may be only 10% of words but 60% of errors.

Switch-point accuracy

For linguistic tasks, evaluate:

• switch boundaries.

For ordinary transcription, correct text may matter more.

Synthetic data

Research explores creating code-switch training data from monolingual sources.

Dhawan et al. proposed methods for generating code-switching ASR data and a tokenizer producing language ID per token for English-Hindi and English-Spanish.

Synthetic data can help but needs realistic evaluation.

Real conversations

Benchmark on spontaneous speech.

Read sentences with inserted foreign words do not capture:

• hesitation;
• interaction.

DECM explicitly uses spontaneous speech from diverse domains for more realistic evaluation.

Customer service

Code-switching may be a sign of comfort, not error.

Support systems should not force the customer to choose one language.

A multilingual transcript can improve agent understanding.

Research interviews

Preserve switching because it may be analytically meaningful.

Do not normalize without documentation.

Translation for publication

Decide whether to show switch.

Example:

• italics;
• retain term.

An editor/native reviewer should decide.

Privacy

Language choice can be identity-linked.

Do not infer ethnicity/nationality from code-switching.

Use only necessary language metadata.

Designing a code-switch annotation guide

A research or evaluation project should define:

• what counts as borrowing;
• how named entities are labeled;
• how multiword switches are marked;
• which script is used;
• how ambiguous tokens are handled.

Without these rules, two annotators can disagree substantially even when both understand the speech perfectly.

Example of translation preserving emphasis

Source bilingual speech may switch languages for a key phrase. A unified English translation can add a note such as:

[speaker switches to Swahili for emphasis]

when the switch itself is analytically or editorially important.

Do not add such notes to ordinary business transcripts unless the audience benefits. The point is to preserve meaningful discourse, not to annotate every linguistic event.

Model evaluation by switch position

Measure errors:

• immediately before a switch;
• on switched words;
• immediately after.

Models may perform well in long monolingual spans and fail specifically at boundaries. This analysis tells whether the problem is language recognition or general ASR.

Real-time code-switch systems

Live systems cannot wait for a human to label each switch. A robust design may use one multilingual recognizer rather than constantly swapping monolingual models. Candidate-language metadata can still constrain decoding.

The tradeoff is that multilingual models may have lower peak performance on one language than a specialized monolingual model. Benchmark the real mixed use case.

Community language behavior

Do not “correct” code-switching out of training data because it seems informal. If users genuinely speak that way, removing it creates a less representative dataset and a worse product.

Language technology should model real communicative behavior, including multilingual behavior.

Matrix language and translation

In code-switched speech, one language may provide much of the grammatical frame while another supplies inserted words or phrases. Researchers call this the matrix language in some theoretical approaches.

Translation usually should not mechanically translate each token according to its language label. The translator needs the utterance-level grammatical meaning first.

This is another reason a unified multilingual model or bilingual reviewer is valuable.

Code-switch datasets need realistic domains

A dataset of bilingual social-media or scripted examples may not predict performance on:

• medical consultations;
• customer service;
• family conversations.

Collect code-switch data in the intended domain because switching patterns and borrowed terminology vary substantially by context.

Automatic code-switch tagging

Token-level LID can provide tags, but production transcript displays do not always need them. Store tags in metadata and render them only when the user or research purpose requires.

This keeps the transcript readable while preserving analytical information.

Translation of untranslatable local concepts

A code-switched word may have been chosen precisely because the dominant language lacks an equivalent. Options include:

• retain term;
• gloss at first use;
• descriptive translation.

A bilingual reviewer should decide based on audience and purpose.

Error-analysis example

If the model transcribes an English word correctly but assigns it a Swahili language tag, the lexical transcript may still be usable while the language-ID metric records an error. Separate:

• text accuracy;
• tag accuracy.

This prevents system designers from treating every LID annotation error as a transcription failure.

Human review priority

Review:

• switch boundaries;
• low-resource language words;
• names;
• culturally significant phrases.

Common monolingual spans can rely more heavily on proven ASR performance.

Code-switching and automatic translation metrics

A target translation can be correct even if the source transcript’s language tags are imperfect. Evaluate separately:

• lexical source accuracy;
• language-tag accuracy;
• target adequacy.

This prevents one research annotation metric from obscuring the practical usability of the transcript.

Glossary handling across code-switched words

A product term may appear in English regardless of the surrounding language. Store it as a concept with:

• source aliases;
• do-not-translate or target policy.

This is better than treating every occurrence as a new language switch requiring special translation logic.

Speaker-aware context

A bilingual speaker may consistently use one language for technical terms and another for general conversation. Passing previous turns to the model can help predict these patterns.

However, the system should not force a learned pattern when the speaker changes behavior. The audio and current words remain authoritative.

Quality escalation rules

Escalate code-switched segments when:

• minority-language confidence low;
• switch changes legal/medical meaning;
• target model drops the switched phrase;
• language identification disagrees with reviewer.

Routine well-recognized switches can remain automated.

Ethical product design

Do not pressure users to “speak one language at a time” simply because the ASR struggles. Improving multilingual robustness and providing correction paths is a better long-term product strategy for communities where code-switching is normal.

Code-switching regression set

Every production pair should keep examples of:

• short one-word switches;
• long phrase switches;
• named entities;
• borrowed technical words;
• noisy mixed speech.

Run the set after ASR or tokenizer changes. Code-switch performance can regress even when monolingual WER improves.

Final principle

Treat code-switching as expected input, not malformed speech. The quality objective is to preserve the speaker’s meaning and language choices well enough for the transcript’s intended use.

Implementation takeaway

For production use, treat switch-boundary evaluation, bilingual context and minority-language preservation as measurable parts of the workflow rather than assumptions. Define the relevant policy before processing content, keep the model or data configuration versioned, and verify results on representative material from the actual language, locale and domain. If automation produces a result that conflicts with a deterministic rule, authoritative terminology or qualified human evidence, escalate the item instead of averaging the disagreement away. The most durable multilingual systems preserve source-to-target traceability, record exceptions and feed reviewed corrections back into benchmarks, glossaries or routing rules. That discipline makes quality improvements reproducible when models, providers, reviewers or source content change.

Frequently asked questions

Is code-switching the same as speaking several languages in one file?

It is specifically alternation by speakers, often within discourse/utterances. A file with separate monolingual sections is easier.

Can Whisper or multilingual ASR handle code-switching?

Multilingual models can handle some, but performance must be benchmarked for your language pair.

Should mixed-language words be translated during transcription?

No. Preserve the source first.

How should translation handle code-switching?

Translate the whole turn in context while deciding whether intentional source-language elements should remain.

What is the best reviewer?

A bilingual speaker familiar with both languages/varieties and the domain.

Conclusion

Code-switched audio exposes the limits of one-language speech pipelines.

Use a constrained multilingual model, preserve source language spans, measure minority-language performance and translate conversation-level meaning rather than disconnected fragments.

Most importantly, treat code-switching as normal human language behavior. The system should adapt to speakers, not force speakers to adapt to a monolingual system.

Research references

• Ugan et al., DECM (LREC-COLING 2024):
• Burchell et al., Code-Switched LID (EACL 2024):
• Dhawan et al., Unified Code-Switch ASR/LID:
• Iakovenko & Hain, Matrix Language (EMNLP 2024):