BLEU, COMET and Human Review: How Translation Quality Is Measured

For production multilingual workflows, see our AI Translation service for the delivery path behind the guidance in this article.

Machine translation has always needed a way to answer a deceptively hard question:

Is this translation better than that one?

Human review can answer it, but human evaluation is expensive and slow. Automatic metrics make large-scale experiments possible, but every metric simplifies translation quality into a number.

BLEU and COMET represent two generations of automatic evaluation. Human review remains the reference point they are ultimately trying to approximate.

No automatic score should be treated as “percent accurate.”

Why automatic metrics exist

Imagine testing ten translation models on 100,000 sentences.

Human review of every output would be extremely expensive.

Automatic metrics allow researchers and engineers to:

• compare models;
• measure improvements;
• detect regressions;
• tune systems;
• rank experiments.

The tradeoff is that a metric is only a proxy for human quality.

BLEU: the classic metric

BLEU was introduced by Papineni and colleagues in 2002.

It compares machine output with reference translations using n-gram precision plus a brevity penalty.

Simplified idea:

If the candidate uses many of the same word sequences as the reference, it gets a higher score.

BLEU became popular because it was:

• fast;
• reproducible;
• language-agnostic;
• useful for corpus-level comparison.

It was a major step for machine translation research.

What BLEU measures well

BLEU can be useful when:

• test data is fixed;
• reference style is consistent;
• systems are compared on the same benchmark;
• evaluation is corpus-level.

It is particularly good at identifying gross lexical overlap differences.

BLEU helped make MT experimentation scalable.

BLEU limitations

Translation is not a one-answer task.

Source:

“The meeting was postponed.”

Valid translations might use different target-language words for:

• delayed;
• rescheduled;
• put off.

A reference may contain one.

BLEU can penalize another valid translation because the surface text differs.

Other limitations include:

• weak segment-level interpretability;
• sensitivity to tokenization;
• inability to reason directly about meaning;
• limited document context.

Meta-evaluation research has warned against drawing overly confident conclusions from small metric differences.

BLEU is not a percentage

A BLEU score of 40 does not mean “40% accurate.”

It is a metric value based on overlap statistics.

Scores are only meaningful with context:

• dataset;
• language pair;
• tokenization;
• implementation.

Do not compare unrelated BLEU scores casually.

COMET: learned semantic evaluation

COMET was introduced in 2020 as a neural framework trained to predict human MT quality judgments.

The model uses multilingual pretrained representations.

Depending on the COMET model, inputs may include:

• source;
• translation;
• reference.

The original research reported strong correlation with human assessments.

Why COMET can outperform lexical overlap

Consider two candidate translations:

A: uses different wording but preserves meaning.

B: shares more words with the reference but changes meaning.

A lexical metric may prefer B.

A learned semantic metric can potentially prefer A because it models meaning and multilingual relationships.

This is the major advantage of metrics such as COMET.

COMET is not infallible

Learned metrics can have weaknesses.

They may:

• over-reward certain styles;
• miss rare critical errors;
• behave differently across languages;
• inherit biases from training data.

WMT now evaluates metrics themselves using human MQM judgments and challenge sets.

The 2024 Metrics Shared Task specifically examined how metrics perform on newer LLM-generated translations.

Human review

Human review can evaluate dimensions that are difficult to encode into one score.

Reviewers can identify:

• critical mistranslation;
• brand mismatch;
• illegal or unsafe wording;
• context errors;
• cultural problems;
• terminology violations.

Human evaluation is therefore still central for production acceptance.

MQM human evaluation

MQM provides an error taxonomy.

Reviewers annotate errors by:

• type;
• span;
• severity.

Typical categories include:

• accuracy;
• fluency;
• terminology;
• style.

The result is more diagnostic than an overall impression score.

Human review is expensive

Expert MQM evaluation can be slow.

WMT research on Error Span Annotation notes that full MQM requires trained experts and can be costly.

This is why businesses should not assume every sentence needs the same level of annotation.

Sampling and tiered review can control cost.

Direct Assessment

Direct Assessment asks reviewers to score overall quality.

It is faster than MQM.

However, it does not explain what went wrong.

A production program may use:

• broad scoring for monitoring;
• detailed error annotation for investigations.

Metrics and human review answer different questions

BLEU answers roughly:

How much does this output overlap with the references?

COMET asks:

How likely is this translation to align with human quality judgments?

Human review asks:

Is this translation correct and acceptable for this purpose?

Those are not identical questions.

A layered evaluation stack

A strong production workflow can use four layers.

Layer 1: deterministic QA

Check:

• numbers;
• placeholders;
• missing segments;
• glossary terms.

Layer 2: automatic metric

Use COMET or another validated metric to detect distribution changes.

Layer 3: human sampling

Review representative content.

Layer 4: mandatory expert review

Apply to:

• legal;
• medical;
• financial;
• safety-critical;
• flagship marketing.

This combines scale and risk control.

System-level versus segment-level

Automatic metrics are usually more reliable when comparing large groups of translations than when judging one sentence.

A single segment may receive a surprising score because:

• reference wording differs;
• sentence is short;
• named entity dominates.

Do not automatically reject one sentence because a metric score falls below a threshold unless the model is validated for that use.

Reference quality

Automatic evaluation depends on reference quality.

A poor reference can cause:

• valid output to be penalized;
• errors to be rewarded.

References should be:

• human-quality;
• relevant to domain;
• stylistically suitable.

Some newer metrics operate without references, but reference-free estimation has its own limitations.

Metric drift

When translation systems change, evaluation metrics may need revalidation.

LLM-based translation can produce more natural variation than traditional MT, challenging older overlap metrics.

WMT 2024 explicitly asked whether LLM translation was “breaking” existing metrics.

This means evaluation is not a one-time setup.

Challenge sets

A challenge set contains deliberately difficult examples.

Examples:

• negation;
• dates;
• named entities;
• pronouns;
• terminology;
• word order.

A metric that scores well overall can still fail to penalize a critical error class.

Use challenge sets alongside normal test corpora.

Business metrics

Translation quality also affects:

• conversion;
• support contacts;
• customer satisfaction;
• task completion.

These are downstream measures, not linguistic metrics.

But they matter in localization.

A technically improved COMET score that produces worse customer outcomes should trigger investigation.

When to use BLEU

BLEU remains useful for:

• legacy benchmarks;
• controlled research comparisons;
• reproducibility.

Use the same implementation and dataset when comparing systems.

Avoid presenting BLEU alone as a business-quality guarantee.

When to use COMET

COMET is useful for:

• modern model benchmarking;
• large-scale quality monitoring;
• semantic evaluation.

Validate the model for:

• languages;
• domains;
• output type.

When human review is mandatory

Human review is especially important when:

• error impact is high;
• style matters;
• source is ambiguous;
• target will be published;
• regulation applies.

Metrics help decide where to look. They do not replace accountable approval.

Metric ensembles

No single metric has complete visibility into translation quality. Organizations sometimes combine:

• lexical metrics;
• learned semantic metrics;
• terminology checks;
• quality-estimation models;
• deterministic validators.

An ensemble can be useful for triage. For example, a translation may receive a strong COMET score but fail a number-preservation check. The deterministic failure should override the reassuring semantic score for a financial document.

Cost of evaluation

Evaluation itself has a cost. A business should calculate:

• model inference cost;
• reference creation cost;
• reviewer time;
• adjudication time;
• engineering effort.

This matters when choosing between a sophisticated metric stack and a simpler risk-based sampling program. The “best” evaluation system is one that provides enough confidence for the business decision without consuming more resources than the translation workflow itself.

Metric versioning

Learned metrics change. COMET has multiple checkpoints and architectures; different versions can produce different absolute scores. Store the exact metric version, configuration and model checkpoint used for every benchmark.

Without versioning, a dashboard may appear to improve or decline simply because the evaluation model changed.

Error analysis after a metric disagreement

If BLEU, COMET and humans disagree, do not average the scores mechanically. Inspect the examples.

Disagreement can reveal:

• valid paraphrase punished by overlap metrics;
• fluent hallucination rewarded by a semantic model;
• weak reference translation;
• human reviewer inconsistency.

These cases are particularly useful for improving an evaluation strategy.

Why business teams should avoid metric shopping

It is tempting to run five metrics and report whichever one gives the most favorable result. That reverses the purpose of evaluation. The metric should be selected before the comparison based on evidence that it correlates with the kinds of human judgments the organization cares about.

A defensible benchmark report records:

• metric name and exact version;
• reference set;
• tokenization or preprocessing;
• sample size;
• language pair;
• human calibration results.

This makes later model comparisons reproducible.

Example: BLEU and COMET disagree

Suppose a candidate translates an English sentence into fluent Spanish using a natural paraphrase. The human reference uses different wording. BLEU may fall because several n-grams do not match, while COMET and bilingual reviewers score the meaning highly.

In another segment, the candidate copies reference-like vocabulary but reverses a logical relationship. BLEU may remain comparatively strong, while a semantic metric or human reviewer catches the meaning error.

Neither example proves one metric is always correct. It illustrates why disagreement should trigger inspection rather than automatic averaging.

A practical enterprise metric policy

For a mature program, define:

• one primary learned metric for system monitoring;
• deterministic checks for non-negotiable data;
• human sample review as the calibration source;
• challenge sets for critical error categories.

Keep historical BLEU if continuity with earlier benchmarks matters, but avoid turning it into a customer-facing “accuracy percentage.” Metrics are engineering instruments, not guarantees.

Implementation takeaway

For production use, treat metric versioning, reference quality and human calibration as measurable parts of the workflow rather than assumptions. Define the relevant policy before processing content, keep the model or data configuration versioned, and verify results on representative material from the actual language, locale and domain. If automation produces a result that conflicts with a deterministic rule, authoritative terminology or qualified human evidence, escalate the item instead of averaging the disagreement away. The most durable multilingual systems preserve source-to-target traceability, record exceptions and feed reviewed corrections back into benchmarks, glossaries or routing rules. That discipline makes quality improvements reproducible when models, providers, reviewers or source content change.

Frequently asked questions

Is COMET better than BLEU?

Often for correlation with human judgments, but performance depends on the evaluation setting.

What is a good BLEU score?

There is no universal threshold. BLEU values depend on language, domain, tokenization and dataset.

Is COMET a percentage?

No. It is a learned metric score whose scale depends on the model.

Can automatic metrics replace human review?

Not for high-stakes acceptance. They are best used as scalable monitoring and benchmarking tools.

Why keep BLEU if newer metrics exist?

It remains widely understood, fast and useful for controlled historical comparisons.

Conclusion

Translation quality cannot be reduced safely to one number.

BLEU measures lexical overlap. COMET models semantic quality from human judgments. Human reviewers evaluate meaning, risk, style and context.

The best evaluation strategy uses each where it is strongest: automatic metrics for scale, deterministic checks for hard constraints, and human review for the decisions that require judgment.

Research references

• Papineni et al., BLEU (ACL 2002):
• Rei et al., COMET (EMNLP 2020):
• Mathur et al., Tangled up in BLEU (ACL 2020):
• WMT 2024 Metrics Shared Task:
• Kocmi et al., Error Span Annotation (WMT 2024):