AI Translation Quality Evaluation: Accuracy, Fluency and Adequacy

For production multilingual workflows, see our AI Translation service for the delivery path behind the guidance in this article.

Evaluating AI translation is harder than asking whether a sentence “looks good.”

Modern systems can produce text that is highly fluent even when part of the meaning is wrong. That makes translation evaluation a multidimensional problem. A strong review separates accuracy, adequacy, fluency, terminology and critical-error risk rather than collapsing everything into one impression.

The key principle is that a translation can be fluent and still be inaccurate.

Accuracy

Accuracy asks whether the translation represents the source correctly.

Errors include:

• mistranslation;
• addition;
• omission;
• wrong number;
• wrong name;
• wrong relationship;
• incorrect negation.

Accuracy should be evaluated against the source, not only by reading the target.

A monolingual target-language proofreader cannot reliably verify semantic accuracy.

Adequacy

Adequacy is closely related to accuracy but focuses on whether the target conveys enough of the source meaning.

A translation can be broadly correct but omit:

• qualification;
• detail;
• nuance;
• emphasis.

For example:

Source: “The product may cause temporary irritation in a small number of users.”

Weak target: “The product can cause irritation.”

The broad idea survives, but “temporary” and “small number” disappear. Adequacy has decreased.

Fluency

Fluency asks whether the target sounds natural and grammatically correct.

Reviewers consider:

• grammar;
• word order;
• punctuation;
• idiomatic usage;
• cohesion;
• readability.

Fluency is target-language quality.

It should not be mistaken for accuracy.

LLMs are especially capable of producing fluent language, which makes this distinction important.

Terminology

Domain terminology deserves its own evaluation.

Check whether:

• approved terms are used;
• product names are preserved;
• technical concepts are correct;
• repeated terms stay consistent.

WMT terminology research has demonstrated that specialized terminology remains a distinct evaluation challenge even when general translation quality is strong.

Completeness

Completeness checks whether every source segment has a corresponding translation.

Common issues include:

• missing sentence;
• untranslated paragraph;
• duplicated segment;
• skipped footnote;
• missing table cell.

Automated alignment can detect many of these.

Style

Style evaluation asks whether the translation follows project requirements.

Examples:

• formal versus informal;
• brand voice;
• technical style;
• plain language;
• legal register.

Style is not universal. The translation brief defines the target.

Locale fit

A linguistically correct translation can still be wrong for the target locale.

Check:

• spelling;
• date format;
• number format;
• formality;
• local terminology;
• units.

This is part of localization quality.

Criticality matters

Not all errors have the same consequence.

A missing comma in internal notes is not equal to a wrong dosage in medical instructions.

Quality frameworks such as MQM use error types and severity levels.

A practical severity model is:

Minor

Does not significantly affect meaning or usability.

Major

Meaning or usability is materially affected.

Critical

Could create legal, safety, financial or reputational harm.

This helps prioritize review.

MQM

Multidimensional Quality Metrics (MQM) is a widely used framework for analytic translation evaluation.

The MQM approach categorizes errors such as:

• accuracy;
• fluency;
• terminology;
• style;
• locale convention.

Reviewers mark the span, category and severity.

Recent WMT human evaluations have used MQM, and 2024 work on MQM scoring highlights its role as both an error typology and scoring framework.

MQM is powerful but expensive because trained reviewers need time to classify errors in detail.

Direct Assessment

Direct Assessment (DA) asks reviewers to assign an overall quality score.

It is faster than detailed error annotation.

However, it offers less diagnostic information.

If a translation scores 70/100, the team still needs to know why.

Error Span Annotation

WMT 2024 research proposed Error Span Annotation (ESA), combining a continuous quality score with error-span severity marking.

The research found ESA could offer faster annotation than full MQM while retaining strong quality.

This illustrates an important production principle: evaluation design should balance information value and reviewer cost.

Automatic metrics

Automatic metrics are useful for:

• benchmarking systems;
• monitoring changes;
• triaging output;
• comparing large batches.

They are not direct substitutes for business-specific human approval.

BLEU

BLEU, introduced in 2002, measures n-gram overlap between machine output and reference translations.

It helped standardize MT research.

Limitations include:

• sensitivity to reference wording;
• weak interpretability;
• poor segment-level reliability;
• inability to directly understand meaning.

BLEU can still be useful in controlled benchmarks but should not be treated as a universal quality score.

COMET

COMET uses multilingual pretrained representations and is trained on human quality judgments.

The original COMET research reported stronger correlation with human assessments than prior automatic approaches.

Modern WMT metric evaluations continue to use human MQM data to assess automatic metrics.

COMET is often more semantically aware than BLEU, but no metric is infallible.

Reference-based versus reference-free evaluation

Reference-based

Uses:

• source;
• candidate translation;
• reference translation.

Reference-free

Estimates quality without a human reference, sometimes using only source and target.

Reference-free approaches are useful when no gold translation exists.

However, research has shown that quality-estimation datasets can contain artifacts and skew, so results should be interpreted cautiously.

LLM-as-a-judge

Large language models can evaluate translations by:

• assigning scores;
• identifying errors;
• applying MQM-like rubrics.

Recent research shows promising performance, while also finding that references and detailed guidelines improve reliability.

LLM evaluation should be calibrated against human judgments for the language pairs and domains that matter.

Document-level quality

Sentence-level evaluation can miss:

• inconsistent pronouns;
• terminology drift;
• contradictions;
• discourse issues;
• repeated names.

As translation systems increasingly handle longer context, quality evaluation should move beyond isolated sentences.

WMT research has studied paragraph-level metric evaluation because sentence-only scoring can miss document phenomena.

Challenge sets

A general test set may hide important weaknesses.

Build targeted challenge sets for:

• numbers;
• negation;
• terminology;
• names;
• idioms;
• low-resource language;
• gender;
• word order.

WMT 2024 metric evaluation included challenge-set testing for exactly this reason: metrics should be tested on known error categories, not only overall correlations.

Human evaluator qualifications

Reviewers need:

• strong target-language proficiency;
• source-language understanding for accuracy review;
• domain knowledge when required;
• clear guidelines.

Native language alone is not always enough for specialized finance, medical or legal content.

Inter-annotator agreement

Two reviewers can disagree.

Reduce inconsistency with:

• examples;
• severity definitions;
• calibration sessions;
• adjudication.

Quality evaluation is itself a measurement process and should be designed carefully.

Sampling

High-volume workflows may not review every segment.

Sampling options include:

• random;
• risk-weighted;
• low-confidence;
• new language;
• high-value content.

Combine sampling with automated QA.

If sample error rates rise, increase review coverage.

A practical scorecard

For business translation, score:

The weights should change by use case.

Legal translation may weight accuracy higher. Marketing may weight style higher.

Acceptance thresholds

Define thresholds before evaluation.

Examples:

• zero critical errors;
• no major errors in defined terms;
• terminology accuracy ≥ 98%;
• average human score above agreed threshold.

Avoid choosing a threshold after seeing the results.

Root-cause analysis

Quality evaluation should improve the workflow.

For each recurring error, ask whether the cause is:

• source ambiguity;
• model weakness;
• missing glossary;
• segmentation;
• bad prompt;
• OCR error;
• reviewer inconsistency.

Fix the system, not just the sentence.

Calibration against real user harm

A translation evaluation program should identify which errors actually matter to the organization. For a travel marketplace, an incorrect location or cancellation condition may be more damaging than a minor style defect. For an AI-data company, mislabeled languages or terminology in dataset documentation may create downstream engineering errors.

Build a critical-error list from incidents, reviewer feedback and business risk. Then make sure both automatic and human evaluation can detect those cases. Aggregate quality scores are useful only when they do not hide severe local failures.

Confidence intervals and statistical uncertainty

When teams compare translation systems, small score differences may not be meaningful. Research on metric meta-evaluation has repeatedly cautioned against over-interpreting marginal improvements without statistical testing.

For production experiments:

• use the same test set;
• keep sampling consistent;
• calculate confidence intervals or significance tests where practical;
• inspect individual error categories.

A 0.2-point improvement on a metric is not automatically a business improvement.

Regression testing

Translation quality should be tested whenever the system changes:

• new model;
• new prompt;
• new glossary;
• new preprocessing;
• new OCR engine;
• new segmentation logic.

Maintain a fixed regression set with difficult examples. Run it before deployment and compare not just overall scores but critical categories such as numbers, named entities and negation.

Human feedback as evaluation data

Every human correction can become evaluation data if it is captured systematically. Store:

• source;
• original AI output;
• approved correction;
• error category;
• severity.

Over time, this creates an organization-specific benchmark that is more valuable than a generic public test set because it reflects the actual content, languages and risks the system encounters.

Quality evaluation for different content risks

The relative importance of evaluation dimensions changes by use case. A legal translation may tolerate slightly formal target prose but cannot tolerate a missing obligation. Marketing content can accept freer restructuring but cannot tolerate an invented claim. Product UI may have excellent semantic accuracy yet still fail because the target label no longer fits or uses a different term from the help center.

For this reason, build a weighting matrix by content type rather than one universal score. Keep a common definition of critical errors across the organization, then adjust weights for fluency, style, terminology and formatting according to the audience.

Example of a misleading aggregate score

Imagine a 20-page financial translation receives excellent fluency scores across 19 pages but changes a percentage from 1.5% to 15% in one table. An average sentence-level metric could still look strong. A risk-based evaluation correctly treats the number error as dominant.

This is why deterministic checks should sit beside semantic metrics. Numbers, identifiers, placeholders and required terminology can often be validated exactly, while human or learned evaluators assess language quality. The systems complement rather than replace one another.

Building an enterprise evaluation dataset

A useful internal benchmark should include:

• typical high-volume content;
• known historical errors;
• difficult names and numbers;
• domain terminology;
• long-context examples;
• rare but critical scenarios.

Keep the test set private where practical, and add new examples after incidents. Over time, the benchmark becomes a record of what “good translation” means for the organization rather than an abstract score from a public dataset.

Frequently asked questions

What is the difference between accuracy and fluency?

Accuracy compares meaning to the source. Fluency measures how natural the target language is.

What is adequacy?

Adequacy measures how completely the target conveys the source meaning.

Which automatic metric is best?

There is no universal metric. Modern learned metrics such as COMET often correlate well with human judgments, but they should be validated for the relevant languages and domains.

Is BLEU obsolete?

No, but it has important limitations and should not be the only evaluation method.

Do I need human evaluation?

For high-stakes translation, yes. For high-volume low-risk content, automatic metrics and sampling can reduce the amount of human review.

Conclusion

AI translation quality is not a single score.

A reliable evaluation program separates meaning, fluency, terminology, completeness, locale fit and error severity. Automatic metrics provide scale; human evaluation provides judgment and accountability.

The best evaluation system is one that not only tells you whether translation quality is acceptable, but also explains where the workflow needs to improve.

Research references

• Papineni et al., BLEU (ACL 2002):
• Rei et al., COMET (EMNLP 2020):
• WMT 2024 Metrics Shared Task:
• Kocmi et al., Error Span Annotation (WMT 2024):
• Lommel et al., MQM scoring models (AMTA 2024):