I Tried to Automate Blog Translation. The Real Product Was QA.

I started with a simple wish: I had English articles on dzarlax.dev, and I wanted Russian and Serbian versions without turning every post into a weekend project.

The shape looked automatable. The source was already Markdown. The site already had language-aware routing. I write in an Obsidian vault. Article bundles are predictable:

articles/<slug>/index.md
articles/<slug>/index.ru.md
articles/<slug>/index.rs.md

So the initial question sounded technical: which translation model should I use?

That turned out to be the wrong question. The real question was: which parts of translation should be automated, and which parts should stay editorial?

The final answer was not a heroic end-to-end translator. It was a much less romantic thing: a draft generator surrounded by deterministic QA. The model translates. The toolchain protects Markdown, links, footnotes, code spans, language constraints, run budgets, and review state. The final prose pass happens in the ChatGPT app and in my editor, because that is where the text actually gets good.

This is how I got there.

The translator I thought I wanted

The first version was the obvious one: send the article to a capable local LLM with a glossary and a tone-of-voice prompt. I already had Ollama. I could run aya-expanse:32b. It was large enough to feel plausible and multilingual enough to be tempting.

For Russian, it was often acceptable. For Serbian Latin, it was not.

The failures were not dramatic in the way benchmark failures usually are. It did not print nonsense every time. It printed text that was close enough to be dangerous: Slavic-looking prose with Czech, Polish, Croatian, or Bosnian residue in places where I needed Serbian Latin. A fluent reader would catch it. I am not fluent enough in Serbian to catch all of it reliably.

Then I tried the classic two-stage design:

1. Use OPUS-MT / Marian for raw translation.
2. Use a larger LLM as a polish pass.

That was attractive on paper. A specialized NMT model should know the grammar; the LLM should know style.

In practice, the boundary was messy. The NMT stage mangled brand names and damaged inline Markdown. The polish stage sometimes repaired tone, but also invented structure. One run added a phantom top-level heading. Another changed punctuation rules it had been explicitly told to preserve. The LLM was not behaving like a mechanical copy editor. It was behaving like a writer.

That is useful when you want a writer. It is expensive when you want a patch.

The first useful baseline was boring

TranslateGemma changed the baseline.

The translategemma:12b run was not perfect, but it did the important boring things well: preserved brands, preserved inline code, kept Markdown shape, avoided Serbian salad, and finished quickly.

The early experiment archive in _meta/translation-experiments ended up looking like this:

The surprising result was not that TranslateGemma was the best writer. It was not. The result was that it was the best draft producer. That is a narrower job, and a much easier one to make reliable.

At this point I still thought I might add a second model for polish. The benchmark run killed most of that idea.

The benchmark answered a different question

I ran a broader comparison on two articles:

• a short, brand-heavy post: sleeping-terminal-1000-dollars;
• a longer, term-heavy Health Dashboard post: part-4-readiness-stopped-being-one-number.

The candidates were the TranslateGemma family, Gemma 3, Aya 32B, and NLLB. The probes were intentionally unglamorous: headings, inline code spans, brand tokens, em dashes, number formatting, Serbian Cyrillic leaks, and obvious Czech/Polish/Bosnian/Croatian residue.

The latency table alone was enough to reset expectations:

There were tempting details. Gemma 3 sometimes picked a better Serbian term. TranslateGemma 27B sometimes needed fewer manual glossary fixes. But every improvement came with another regression: Western-style currency formatting, English nouns declined into Serbian suffixes, or much longer runtime.

The most important benchmark result was about what not to automate.

The glossary had conditional rules like: keep readiness in English when it refers to the concrete recovery score; translate it otherwise. No model handled that consistently. The table in the comparison note made the conclusion hard to avoid: this was not a translation-model problem. It was an editorial decision with local context. A reviewer should make it.

That became the first design rule:

Do not spend automation budget on judgment calls that still need human review.

Then the bulk run broke in the old-fashioned way

The next failure was not linguistic. It was mechanical.

A bulk run over the eight Health Dashboard parts used an output cap that was too small for long articles. The model did not throw an error. It just stopped.

The damage pattern was very specific:

• footnote definitions disappeared, because they were at the end of the source;
• final sections disappeared in longer articles;
• some code blocks disappeared;
• Serbian Cyrillic leaked into Latin-script output.

This was useful because it clarified the second design rule:

Translation output is not valid just because it looks like prose. It is valid only after structural checks pass.

That pushed the pipeline toward chunking, placeholder protection, retries, and budgets.

The tool started protecting fenced code blocks, inline code, and footnote references before sending chunks to the model. It started recording run profiles. It gained hard limits like TRANSLATION_MAX_MODEL_CALLS and TRANSLATION_MAX_CHUNK_RETRIES, because a retry loop without a budget is just a future billing incident with nicer logs.

The quality gate also stopped being a vibe check. It had to fail the run if a placeholder disappeared, a footnote definition was orphaned, or Serbian Latin contained Cyrillic outside allowed quoted examples.

Hosted GPT nano made the economics boring

After the local-model experiments, I tried openai/gpt-4.1-nano through OpenRouter.

The first production-shaped run was deliberately not all-in. It covered all articles except the already-published sleeping-terminal post, in Russian and Serbian.

The summary was awkward but informative:

The important line is not the cost, although the cost was low enough to stop being the main concern. The important line is that the failed jobs usually failed after chunk translation, at the local quality gate.

Most Russian jobs passed. Serbian jobs repeatedly hit Serbian Latin output contains Cyrillic outside code blocks. One Russian job failed on a missing footnote placeholder. These were not reasons to abandon the hosted model. They were reasons to improve the harness.

The next runs added stricter retry prompts, Serbian normalization, placeholder-specific retries, precheck artifacts under _meta/translation-precheck, and debug output that did not clutter article directories.

The remaining Serbian strict-v2 run completed the missing pieces:

This was the point where the model choice became less interesting than the system around it.

GPT nano was cheap enough. TranslateGemma was local enough. Neither removed the need for final editing. Both became acceptable as draft engines if the surrounding toolchain could tell me when they broke structure.

The embarrassing part: the app still wrote better prose

After all that, the final article polish still often worked better in the ChatGPT app.

That could sound like a failure. I do not think it is.

The local pipeline had been trying to do two jobs at once:

1. produce a structurally safe translation;
2. produce final publishable prose in my voice.

The first job is an engineering problem. The second is editorial.

When I pushed both into the same automated pass, I got bad trade-offs. Stronger rewrite models improved some sentences but damaged Markdown or drifted terminology. Translation-focused models preserved structure but produced text that sometimes read like a correct machine translation. A deterministic script can detect a missing footnote. It cannot tell whether a Russian title feels like a phrase a human would actually write.

The most productive workflow was therefore not end-to-end automation. It was division of labor:

English canonical article
   -> draft translation
   -> structural QA
   -> ChatGPT app editorial pass
   -> post-edit structural QA
   -> draft:false + mark-reviewed

The app is allowed to be a prose editor. The repo is responsible for making that edit safe to paste back.

The final tool is a harness

The current blog-translator is not a magic translator. It is a harness with sharp boundaries.

It does these things:

• generates draft translations for index.ru.md and index.rs.md;
• protects Markdown structures before model calls;
• chunks long articles and records per-chunk profiles;
• enforces retry and model-call budgets;
• archives runs under _meta/translation-runs and _meta/translation-experiments;
• tracks translated/reviewed/stale state in _meta/translation-registry.json;
• runs deterministic QA over articles and translated series pages.

The newest piece is translation-qa.py. It is intentionally boring. It checks frontmatter, draft state when requested, footnote refs and definitions against the English source, Markdown URLs, raw URLs, code fences, broken bold/code artifacts, Serbian Cyrillic leakage, and git diff --check.

It also knows that articles and series pages are different shapes:

python bin/translation-qa.py --vault /path/to/blog --slug part-8-asking-before-telling --lang rs
python bin/translation-qa.py --vault /path/to/blog --kind series --slug health-dashboard
python bin/translation-qa.py --vault /path/to/blog --kind all --expect-draft false

That last command matters because the page I almost forgot was not an article. It was the translated series page.

What I would automate again

I would absolutely automate the draft.

I would automate code-block protection, inline-code protection, footnote checks, link checks, script checks, run manifests, retries, cost summaries, and review state. I would keep every one of those.

I would not try to make the same pipeline responsible for final voice.

The better lesson is narrower: use models where they are cheap and useful, then build deterministic systems around the failure modes they actually have.

For my blog, that means the canonical source stays English. Russian and Serbian start as model-generated drafts. The toolchain catches structural damage. The ChatGPT app and I handle prose. Then the toolchain checks again before the translation is marked reviewed.

It is less magical than the translator I thought I wanted.

It is also the first version I trust.