You Can't Read Your Japanese Documentation, So How Do You Know It's Any Good?

The Japanese documentation shipped. The vendor delivered on time; we paid the invoice, and the handoff note said everything passed QA. Then someone asked the obvious question in a planning meeting: Is it any good? The room went quiet, because nobody there could read a word of it.

This is the normal situation for software companies selling into Japan. You are spending real money on a deliverable that no one in the building can inspect, and the only quality signal you have is the word of the people you paid to produce it. You usually find out how good it was months later, when a partner's engineer in Osaka mentions that their team just reads the English pages instead.

You don't have to wait for that conversation. You can evaluate Japanese documentation without reading Japanese yourself, as long as you stop relying on the wrong kinds of evidence.

"It passed QA" is not evidence

When a localization vendor says the work passed QA, ask what was actually checked. Much of what gets called QA is automated: tags are intact, numbers match the source, placeholders weren't broken, and approved glossary terms appear where expected. Those checks are worth running, but they say almost nothing about whether a Japanese engineer can read the page and do what it describes.

Human review varies just as much. ISO 17100, the international standard for translation services, requires that a second person revise the translation, not just the translator checking their own work. That is a sensible floor, so ask your vendor who that second person was. A reviser who is a capable linguist but has never configured an OAuth client will catch awkward Japanese and miss a sentence that confidently describes the wrong step. A certification tells you a process exists. It doesn't tell you the reviser understood your product.

Back-translation answers the wrong question

The most common workaround is back-translation: pay someone to translate the Japanese back into English and compare it with the original. It feels rigorous, and it does catch gross errors such as omitted paragraphs or a reversed instruction.

What it can't catch is most of what Japanese readers actually notice. Stiff, word-for-word Japanese often back-translates beautifully, precisely because it followed the English so closely. Inconsistent terminology, the wrong level of formality, and phrasing no Japanese engineer would ever write all disappear on the way back into English. Back-translation tells you whether the meaning survived. It doesn't tell you whether anyone would want to read it.

Run an independent, sampled review

You don't need to review every page. You need a sample that represents the documentation's risk and a reviewer who has no stake in the answer.

1. Choose the pages that matter most. A getting-started guide, your authentication docs, a troubleshooting or error-reference page, your highest-traffic page in Japan, and a slice of UI strings. A dozen well-chosen pages will tell you more than a hundred random ones.
2. Hire a reviewer who is independent of the vendor. They should be a native Japanese writer who understands your domain. Give them the English source, your termbase if one exists, and access to the product itself, not just the text.
3. Score against a shared error typology. MQM (Multidimensional Quality Metrics) is an open framework for analytic translation quality evaluation, maintained by the MQM Council. Its top-level categories include terminology, accuracy, linguistic conventions, style, locale conventions, audience appropriateness, and design and markup. Have the reviewer log each problem with a category and a severity. A spreadsheet is enough.
4. Add a task test. Ask the reviewer to complete one real task, such as creating an API key and making a first authenticated call, using only the Japanese documentation. This is where accuracy problems surface, because the product doesn't care how fluent the sentence was.
5. Ask for patterns, not just a list of fixes. The most valuable output is a judgment: are the problems isolated slips, or systemic issues with terminology, register, or technical understanding? Isolated slips mean you have a good vendor having a normal week. Systemic issues mean you have a vendor problem.

Checks you can run without reading Japanese

Some of the best signals are already in your own systems.

• Support tickets from Japan. Look for questions the Japanese docs supposedly answer, and for Japanese customers writing to you in English while citing your English pages.
• Locale switching. If your analytics show Japanese visitors landing on Japanese pages and then switching to English, they are telling you which version they trust.
• Terminology consistency. You can count term variants with a short script even if you can't read them. Japanese technical writing has common spelling variants for loanwords, and a documentation set should pick one and stick to it.

import pathlib, re

variants = {
    "user": [r"ユーザー", r"ユーザ(?!ー)"],
    "interface": [r"インターフェース", r"インタフェース", r"インターフェイス"],
}

text = "".join(p.read_text(encoding="utf-8")
               for p in pathlib.Path("docs/ja").rglob("*.md"))

for term, patterns in variants.items():
    print(term, {p: len(re.findall(p, text)) for p in patterns})

If every form shows up in volume, nobody is enforcing a termbase, and that tells you something about how the rest of the work was managed. Ask your vendor for the termbase directly. If one doesn't exist, that is your answer.

Put quality in the contract

Once you know what good looks like, write it down before the next engagement. Agree on the error typology, the severity levels that make a page unacceptable, how often an independent sample will be reviewed, and who pays for rework when a sample fails. Strong vendors welcome this, because it separates them from cheaper competitors. Vendors who resist clear acceptance criteria are telling you something too.

Why the reviewer needs to understand the technology

A linguist can tell you whether a sentence is natural Japanese. Only someone who understands the product can tell you whether it's true. The most expensive errors in Japanese documentation tend to be fluent, polite, well-formatted sentences describing behavior the software doesn't have, often because the English was ambiguous and the translator had no way to know. I wrote about what Japanese readers notice in Why Your Japanese Documentation Is Quietly Hurting Your Product. An evaluation is how you find out whether your docs have those problems before your customers decide for you.

I spent decades as a software engineer before I became a technical writer, and Japanese has been part of my life since I moved to Japan in 1989. When I review a Japanese documentation set, I compare it against the English source and the running product. This is the only way to tell a translation problem from a product problem.

Knowing what you paid for

If you are investing in the Japanese market and have no reliable way to judge the documentation your customers there are reading, I'd be glad to help you scope a sampled review or set up acceptance criteria for your next localization contract. You can reach me through the contact page.