How to Evaluate Terminology Accuracy in AI Translation

For production multilingual workflows, see our AI Translation service for the delivery path behind the guidance in this article.

A translation can be excellent overall and still fail because one important term is wrong.

Terminology errors are especially costly in legal, medical, technical, financial and product content. A model may produce a natural synonym when the organization requires one approved term. It may translate a product name that should remain unchanged or use the correct dictionary form in a grammatically incorrect way.

Terminology therefore needs its own quality evaluation.

The central principle is to evaluate concept correctness, not only string matching.

Build the evaluation resource first

You cannot measure terminology accuracy without a source of truth.

A termbase should contain:

• source term;
• definition;
• target term;
• locale;
• domain;
• status;
• accepted variants;
• forbidden variants;
• do-not-translate rule.

Definitions are important because one source word can represent several concepts.

High-priority terminology

Not every word belongs in a termbase.

Prioritize:

• product names;
• feature names;
• legal defined terms;
• medical concepts;
• technical components;
• financial categories;
• regulated terminology;
• recurring ambiguous terms.

The value of terminology management comes from controlling high-impact concepts, not micromanaging ordinary language.

Exact-match evaluation

The simplest QA rule is:

If source term X occurs, target term Y must occur.

This works well for:

• product names;
• acronyms;
• model numbers;
• fixed interface labels.

It is less reliable for highly inflected languages.

Morphological variation

Languages can change the surface form of a term according to:

• case;
• gender;
• number;
• agreement.

A correct translation may not match the glossary string exactly.

For example, a termbase may store a nominative singular form while the sentence requires a different case.

Evaluation should therefore support:

• lemmatization;
• morphological matching;
• acceptable forms;
• human review.

Do not penalize grammatical correctness for failing a simplistic exact-match test.

Multiword terms

Terminology can appear:

• inflected;
• reordered;
• discontinuous;
• abbreviated.

A multiword term may require language-specific pattern matching.

For strict legal defined terms, exactness may be appropriate.

For technical prose, semantic term matching may be more useful.

WMT terminology research

The WMT 2023 terminology shared task explicitly evaluated both general translation quality and specialized terminology usage.

Its findings show that providing terminology dictionaries can improve systems but also highlight limitations in terminology-centric evaluation metrics.

This reinforces the need to measure term behavior separately rather than assuming an overall metric captures it.

Term accuracy metric

A simple metric is:

correct required terms / total source term occurrences

Example:

• 500 term occurrences;
• 485 acceptable target terms.

Term accuracy = 97%.

But the metric needs rules for:

• repeated occurrences;
• ambiguous source terms;
• acceptable variants.

Document these before evaluation.

Term consistency

Accuracy and consistency are different.

If a source concept occurs 20 times and the model uses:

• approved term 18 times;
• synonym A once;
• synonym B once;

the document may have high accuracy but visible inconsistency.

Track:

consistency rate = dominant approved rendering / comparable occurrences

Contextual exceptions should be excluded.

Forbidden-term rate

A termbase can list deprecated or misleading forms.

Track:

• number of forbidden occurrences;
• severity;
• pages/segments affected.

This is especially useful during terminology migrations.

Do-not-translate compliance

Some terms should remain unchanged:

• brand names;
• APIs;
• SKU codes;
• commands.

Check whether:

• spelling changed;
• script transliteration occurred unexpectedly;
• capitalization changed;
• spaces/hyphens changed.

A product name that remains semantically “similar” is still wrong if brand policy requires exact preservation.

Contextual correctness

A target term may be approved in one sense but wrong in another.

Source:

“charge”

Possible domains:

• electrical;
• legal;
• financial;
• battery.

A glossary should have domain or definition metadata.

The QA system should not force one universal target string.

Term extraction for evaluation

If no termbase exists, AI can help extract candidate terms from:

• source corpus;
• product documentation;
• previous translations.

Human experts then approve the high-value list.

Automatic extraction is a discovery tool, not the final source of truth.

Reference translations

Reference translations can provide evidence of preferred terminology.

However, do not assume every reference term is mandatory.

A reference may contain:

• old terminology;
• stylistic variant;
• translator error.

Approved termbase status should take priority.

Human terminology review

A reviewer should ask:

1. Is this the correct concept?
2. Is the approved target term used?
3. Is the form grammatically correct?
4. Is this an allowed exception?
5. Is the same decision used elsewhere?

This catches errors that exact matching cannot.

Terminology severity

Classify errors.

Minor

Cosmetic variant with little impact.

Major

Wrong approved term creates confusion.

Critical

Term changes safety, legal or technical meaning.

A medical drug or legal-defined-term error may be critical even if only one occurrence exists.

Automated terminology QA

A production system can:

• identify source term spans;
• align target segments;
• search accepted variants;
• flag missing required terms;
• flag forbidden variants.

Use an LLM or bilingual embedding model for semantic cases, but keep deterministic matching for strict terms.

Terminology-aware LLM evaluation

LLM judges can compare:

• source;
• translation;
• glossary.

They are useful when:

• terms are inflected;
• source is ambiguous;
• wording is not exact.

However, the judge should not invent approved alternatives.

The glossary remains authoritative.

Terminology coverage

Coverage asks:

How much of the important domain vocabulary is governed?

Measure:

• candidate high-priority terms;
• approved entries;
• target-language entries.

A 99% term-accuracy score is less reassuring if the termbase covers only 20% of critical vocabulary.

Cross-language terminology

At scale, one concept should have a stable concept ID.

Example:

concept_00412 = Workspace

Then each locale has:

• target term;
• status;
• notes.

This is stronger than maintaining independent spreadsheets for 50 languages.

Quality dashboard

Useful terminology metrics:

• accuracy;
• consistency;
• forbidden term count;
• do-not-translate violations;
• coverage;
• unresolved candidate terms;
• reviewer override rate.

Review by language and domain.

Reviewer override rate

If reviewers repeatedly reject glossary terms, investigate.

Possible causes:

• wrong translation;
• overly rigid entry;
• missing sense distinction;
• outdated product terminology.

A glossary is a living controlled resource.

A practical evaluation workflow

Step 1

Freeze the glossary version.

Step 2

Extract source term occurrences.

Step 3

Automatically verify strict terms.

Step 4

Flag morphological/semantic cases.

Step 5

Human-review flagged segments.

Step 6

Calculate term accuracy and severity.

Step 7

Update terminology only after approval.

This keeps evaluation reproducible.

Example: evaluating a technical term with inflection

Suppose a glossary stores a target-language noun in its dictionary form. In a sentence, grammar requires that noun in another case. An exact-string checker flags the segment even though the translation is correct.

A stronger pipeline first checks whether the target contains an acceptable lemma or morphological form. If the term is high risk, a bilingual reviewer verifies that the inflected form still refers to the intended concept. This prevents two opposite mistakes: rejecting correct grammar and accepting a wrong term that merely looks similar.

Build terminology challenge sets

A terminology benchmark should contain more than easy exact matches. Include:

• one source term with several senses;
• long multiword terms;
• inflected target forms;
• acronyms;
• product names that must remain unchanged;
• deprecated variants;
• terms next to similar competing concepts.

This tests whether the system understands term use rather than just whether a glossary string can be copied.

Context windows for terminology QA

Some term decisions depend on a definition several sentences earlier. A segment-level checker may therefore misclassify the target. When the source term is ambiguous, provide the evaluator with the surrounding paragraph and termbase definition.

The QA system can classify entries by enforcement type:

• strict: exact brand/API name;
• morphological: approved lemma with grammatical inflection;
• semantic: approved concept but flexible wording.

Different term types deserve different validators.

Closing the terminology feedback loop

Every reviewer override should become data. If a term is repeatedly overridden because it sounds unnatural, investigate the termbase rather than labeling every reviewer correction as model error. Conversely, if reviewers repeatedly ignore an approved product term, improve the reviewer interface and governance.

The most useful terminology accuracy metric is therefore paired with an override and exception report.

Terminology acceptance criteria by risk

A terminology program should define different acceptance rules for different term classes.

Class A: identity and regulated terms: product names, drug names, legal defined terms and safety-critical component names. Require exact or formally approved forms and treat violations as major or critical.

Class B: controlled business terminology: feature names, service categories and recurring technical concepts. Require the approved lemma or allowed inflection. Deviations are normally major if they create inconsistency or ambiguity.

Class C: recommended language: preferred marketing or editorial terms. Alternatives may be acceptable if meaning and style remain correct.

This classification prevents a QA tool from treating a harmless stylistic synonym with the same severity as a wrong drug name.

Measuring terminology by document, not only by segment

A term can be correct in each individual sentence yet inconsistent across a document. Run a document-level pass that groups all source occurrences of a concept and lists every target rendering. Reviewers can then see whether the model uses three variants for one feature.

For long technical manuals, document-level consistency can be more important to users than one isolated sentence-level term score.

Glossary conflicts and precedence

Large projects may contain several resources:

• global corporate glossary;
• client/project glossary;
• legal defined terms;
• locale-specific exceptions.

Define precedence explicitly, for example:

1. document-specific defined term;
2. project/client term;
3. product/global term;
4. general language preference.

The AI and QA system should use the same precedence. Otherwise the translator can follow one glossary while the checker enforces another.

Terminology quality review meeting

For strategic products, review terminology metrics monthly or after major releases. Examine:

• high-frequency misses;
• reviewer overrides;
• newly emerging source vocabulary;
• deprecated terms still appearing;
• languages with low term coverage.

Terminology accuracy improves fastest when errors lead to termbase decisions rather than repeated one-off corrections.

Implementation takeaway

For production use, treat term-class risk, morphology-aware validation and document-level consistency as measurable parts of the workflow rather than assumptions. Define the relevant policy before processing content, keep the model or data configuration versioned, and verify results on representative material from the actual language, locale and domain. If automation produces a result that conflicts with a deterministic rule, authoritative terminology or qualified human evidence, escalate the item instead of averaging the disagreement away. The most durable multilingual systems preserve source-to-target traceability, record exceptions and feed reviewed corrections back into benchmarks, glossaries or routing rules. That discipline makes quality improvements reproducible when models, providers, reviewers or source content change.

Frequently asked questions

Is exact string matching enough?

Only for strict terms. Inflected languages and contextual variants require more flexible evaluation.

What terminology accuracy should a translation achieve?

There is no universal threshold. Critical product, legal or medical terms may require effectively 100% compliance.

Can AI create the glossary automatically?

It can propose terms and translations, but domain owners should approve them.

Should terminology be evaluated separately from COMET or BLEU?

Yes. Overall metrics can miss important specialized term errors.

What is the most important glossary field after the translation itself?

A definition or context note, because it distinguishes different senses of similar source terms.

Conclusion

Terminology accuracy is measurable when the organization first defines what “correct terminology” means.

Combine a governed termbase, automated detection, morphology-aware matching and human contextual review. Track both accuracy and consistency, and treat critical-domain terms according to their risk.

The result is a quality signal that is far more actionable than a general translation score.

Research references

• WMT 2023 Terminology Shared Task:
• Moslem et al., Domain Terminology Integration (WMT 2023):
• Bogoychev & Chen, Terminology-aware LLM prompting (WMT 2023):
• Google Cloud Translation glossaries:
• Amazon Translate custom terminology: