Human Evaluation of AI Translation: A Practical Scoring Framework

For production multilingual workflows, see our AI Translation service for the delivery path behind the guidance in this article.

Automatic translation metrics are useful, but most organizations eventually need a human answer to a practical question:

Can we publish this?

Human evaluation turns that judgment into a repeatable process. Without a framework, reviewers tend to rely on intuition. One person penalizes awkward style heavily; another focuses only on mistranslations. Scores become inconsistent and difficult to act on.

A practical framework defines what reviewers should evaluate, how serious each error is, and what quality level is acceptable.

MQM is a widely used framework for detailed human translation evaluation, while newer approaches such as Error Span Annotation aim to reduce cost while preserving useful diagnostic information.

Start with the purpose

A scoring framework must reflect what the translation is for.

A medical instruction and an internal marketing brainstorm should not share the same acceptance policy.

Define:

• audience;
• content type;
• risk;
• publication status.

This determines weights and severity.

Step 1: define evaluation dimensions

A practical framework can use six core dimensions.

Accuracy

Does the target preserve source meaning?

Errors:

• mistranslation;
• addition;
• wrong relation;
• wrong negation.

Completeness

Is every relevant source element translated?

Errors:

• omission;
• duplicate;
• untranslated segment.

Terminology

Are approved terms correct and consistent?

Fluency

Is the target grammatical and natural?

Style

Does the translation follow the brief?

Locale/formatting

Are local conventions correct?

This compact framework is easier to train than a very deep taxonomy.

Step 2: define severity

Severity should reflect impact.

Minor error

Limited impact.

Examples:

• punctuation preference;
• slight awkwardness;
• non-critical style issue.

Major error

Materially affects meaning or usability.

Examples:

• wrong technical term;
• missing qualification;
• confusing instruction.

Critical error

Could cause serious harm or invalidate the content.

Examples:

• wrong medical dosage;
• reversed legal obligation;
• changed financial amount;
• unsafe instruction.

Critical errors should usually trigger automatic rejection.

Step 3: define weights

A scoring formula could assign:

• minor = 1;
• major = 5;
• critical = automatic fail.

Or:

• minor = 1;
• major = 5;
• critical = 25.

The exact numbers are less important than consistency.

MQM scoring similarly combines error categories and severity into measurable results.

Step 4: normalize by volume

Long documents naturally contain more errors.

Normalize by:

• words;
• segments;
• characters.

For example:

weighted error points per 1,000 words

This allows comparison across documents.

Example

A 5,000-word translation contains:

• 5 minor errors;
• 2 major errors;
• 0 critical.

Using:

• minor = 1;
• major = 5;

Total error points:

5 + 10 = 15

Normalized:

3 error points per 1,000 words.

The project can define whether that passes.

Step 5: create an error taxonomy

Keep the taxonomy useful.

A practical set is:

Accuracy

• mistranslation;
• omission;
• addition;
• entity/number.

Fluency

• grammar;
• word order;
• punctuation;
• spelling.

Terminology

• wrong term;
• inconsistent term.

Style

• register;
• brand;
• tone.

Locale

• date;
• currency;
• formatting.

Reviewers should not spend more time choosing a category than evaluating the translation.

Step 6: define examples

Guidelines become much more useful when they include examples.

For each category, provide:

• source;
• bad target;
• correct target;
• severity;
• explanation.

Examples reduce reviewer disagreement.

Step 7: calibrate reviewers

Before production, give reviewers the same sample.

Compare:

• error counts;
• categories;
• severity.

Discuss disagreements.

Calibration is especially important for:

• major versus minor;
• style;
• ambiguity.

Human evaluation is a measurement instrument. Reviewers need calibration just as metrics do.

Step 8: separate language and domain expertise

A native speaker may not recognize a technical error.

For specialized content, reviewers may need:

• target language;
• source language;
• domain expertise.

Options include:

• bilingual subject expert;
• linguist plus specialist;
• second-stage domain review.

MQM in practice

MQM offers a deeper taxonomy than many teams need initially.

Its strengths:

• standardized categories;
• severity;
• rich diagnostics;
• broad industry use.

Its weakness:

• review cost.

The 2024 AMTA work on MQM scoring emphasizes that the framework combines error typology with a scoring model.

For enterprise QA, a reduced MQM subset is often a practical compromise.

Error Span Annotation

WMT 2024 proposed Error Span Annotation as a middle ground.

Reviewers:

• mark error spans;
• assign severity;
• provide overall scoring.

The research reported that ESA could be faster and cheaper than full MQM while maintaining high evaluation quality.

This is relevant for businesses that need scalable human evaluation.

Holistic scoring

Some projects use a 1–5 scale.

Example:

5: Excellent

Publication-ready, no meaningful corrections.

4: Good

Minor edits only.

3: Usable after review

Several material edits.

2: Poor

Substantial retranslation.

1: Unusable

Meaning is unreliable.

Holistic scoring is easy but less diagnostic.

Combine it with error tagging.

Pairwise comparison

Another evaluation method asks:

Which translation is better, A or B?

Pairwise review is useful for:

• model selection;
• A/B testing;
• vendor comparison.

It avoids forcing reviewers to understand a numerical scale.

However, it does not provide an absolute acceptance threshold.

Sampling strategy

For high-volume AI translation, review a sample.

Sampling approaches:

Random

Good for unbiased estimation.

Risk-based

Prioritize legal, medical, pricing and safety content.

Confidence-based

Review low-confidence output.

New language

Increase review during launch.

New model

Increase review after system changes.

Combine approaches.

Statistical thinking

Do not review five sentences and claim a model is “99% accurate.”

Quality estimates need enough data.

Track:

• sample size;
• error rate;
• severity;
• language pair;
• content type.

The more variable the content, the larger the sample required.

Critical error register

Maintain a separate list of critical errors.

Examples:

• number corruption;
• negation reversal;
• safety instruction;
• legal obligation;
• personal name.

Even if average quality is high, recurring critical errors can make the system unsuitable.

Reviewer comments

Require comments for:

• critical errors;
• ambiguous source;
• proposed terminology changes.

Do not require long explanations for every missing comma.

The review system should capture useful information without wasting reviewer time.

Adjudication

When reviewers disagree on important errors, use adjudication.

An adjudicator can:

• choose the correct classification;
• update guidance;
• improve examples.

Repeated disagreements indicate unclear guidelines.

Measuring reviewer agreement

Formal projects may calculate inter-annotator agreement.

Business teams can also use simpler measures:

• percentage of shared errors;
• severity agreement;
• pass/fail agreement.

The goal is not academic perfection. It is operational consistency.

Translation quality scorecard

A sample scorecard:

Reviewers can assign 0–100 per dimension.

Alternatively use error points.

Choose one system and keep it consistent.

Acceptance policies

Example:

Low-risk internal

• no critical errors;
• average score ≥ 80.

Customer-facing

• no critical;
• major error rate below threshold;
• score ≥ 90.

High-risk

• zero critical;
• zero major accuracy errors;
• specialist signoff.

Do not apply one threshold to every content type.

Use human data to improve AI

Reviewer edits can improve:

• prompts;
• glossaries;
• model routing;
• training data;
• quality checks.

Human evaluation should create operational learning, not only pass/fail reports.

Reviewer productivity matters

A quality framework should be designed around the reviewer’s actual work. Too many categories slow annotation and increase inconsistency. Too few categories produce reports that are hard to improve from.

Pilot the rubric with real reviewers and measure:

• minutes per 1,000 words;
• agreement;
• percentage of errors categorized as “other”;
• number of guideline questions.

If reviewers repeatedly debate the taxonomy instead of the translation, simplify it.

Blind evaluation for vendor or model comparisons

When comparing two providers or models, hide the system identity from reviewers where practical. This reduces expectation bias. Randomize the order of outputs and use the same source segments.

For pairwise evaluation, reviewers can select:

• A better;
• B better;
• tie;
• both unacceptable.

Then separately annotate severe errors. This produces a useful ranking while still protecting against a “less bad” winner that fails the acceptance standard.

Quality dashboards

A production dashboard should show more than one total score. Useful views include:

• error severity by language;
• terminology accuracy;
• critical-error count;
• reviewer edit rate;
• quality by content type;
• trend after model changes.

This allows managers to see whether overall quality is stable while one language or domain deteriorates.

Reviewer wellbeing and fatigue

Long translation-evaluation sessions can reduce attention. Critical errors are easier to miss when reviewers are working through thousands of near-identical segments. Use reasonable batch sizes, breaks and rotation for monotonous high-volume work.

For high-stakes review, do not schedule the most consequential content only after a reviewer has completed hours of routine annotation. Human quality controls need human-centered workflow design too.

A reviewer worksheet example

A practical review form can include one row per segment with:

• source;
• AI target;
• corrected target;
• category;
• severity;
• reviewer note;
• glossary change required? yes/no.

This structure captures both the final correction and the reason for it. It also enables later analysis: if 40% of major errors are terminology-related, the organization knows that improving the termbase may deliver more value than switching the base model.

Calibration before a production launch

Before a new language launches, select 100–300 representative segments and ask at least two qualified reviewers to score them independently. Compare major/critical error agreement and discuss the segments with the largest differences.

Typical disagreements reveal unclear rules such as:

• whether one regional word is acceptable;
• when an omission becomes major;
• whether overly formal style is minor or major.

Update the guidelines before scaling. This investment makes later quality data far more stable.

When an overall score should not override an error

Some failure types should be “gating” conditions. A translation can score 96/100 overall and still fail if it contains:

• one wrong medical dosage;
• one inverted legal obligation;
• one changed bank-account number;
• one unsafe operating instruction.

Define these gates explicitly. Reviewers then know that their role is not merely to optimize an average score but to protect the properties that make the translation usable and safe.

Practical rule for borderline errors

When a reviewer is unsure whether an issue is minor or major, ask what would happen if the target were published unchanged. If a typical reader would still understand the same factual meaning and complete the intended task, the issue is likely minor. If the error could change a decision, instruction, obligation or interpretation, classify it as major. This consequence-based test keeps severity judgments tied to real use rather than personal sensitivity to wording.

Implementation takeaway

For production use, treat reviewer calibration, severity consistency and critical-error gating as measurable parts of the workflow rather than assumptions. Define the relevant policy before processing content, keep the model or data configuration versioned, and verify results on representative material from the actual language, locale and domain. If automation produces a result that conflicts with a deterministic rule, authoritative terminology or qualified human evidence, escalate the item instead of averaging the disagreement away. The most durable multilingual systems preserve source-to-target traceability, record exceptions and feed reviewed corrections back into benchmarks, glossaries or routing rules. That discipline makes quality improvements reproducible when models, providers, reviewers or source content change.

Frequently asked questions

What is the best human translation scoring framework?

MQM is a strong standardized framework, but many businesses benefit from a simplified MQM-inspired taxonomy.

How many reviewers are needed?

One reviewer may be sufficient for routine content. High-stakes or benchmarking work often benefits from multiple reviewers or adjudication.

Should reviewers see the source?

Yes for accuracy evaluation.

How do you score critical errors?

Many organizations treat any critical error as an automatic failure.

Can LLM judges replace human reviewers?

They can support evaluation and triage, but human calibration remains important for high-risk decisions.

Conclusion

Human translation evaluation becomes much more valuable when it is structured.

Define dimensions, severity, examples, thresholds and reviewer calibration. Normalize scores, track critical errors and feed recurring problems back into the translation workflow.

The purpose of a scoring framework is not to create bureaucracy. It is to make quality decisions repeatable, explainable and improvable.

Research references

• Lommel et al., MQM scoring models (AMTA 2024):
• Kocmi et al., Error Span Annotation (WMT 2024):
• WMT 2024 Metrics Shared Task:
• ISO 17100:2015 Translation services: