How We Fixed the TranslateGemma Template Bug in llama.cpp

i can haz fix again!

Session Summary

Date: July 5, 2026 Issue: ggml-org/llama.cpp#20305 Branch: ali0une-fixes

Discovery

The bug was discovered while running the translategemma-12B-Instruct model through the llama.cpp router. The server would fail at startup with:

common_chat_verify_template: failed to apply template:
While executing CallExpression at line 601, column 31 in source:
Error: Jinja Exception: User role must provide `content` as an iterable with exactly one item.
That item must be a `mapping(type:'text' | 'image', source_lang_code:string, target_lang_code:string, text:string | none, image:string | none)`.

The model had worked fine at build 8226 (commit 34df42f7b) but failed at build 8461 (commit cea560f48). The initial suspicion was commit 34df42f7b itself ("hexagon: add f32 ssm_conv op"), but that commit is unrelated to chat template handling.

Investigation

Root Cause Analysis

Tracing the 235 commits between 34df42f7b and cea560f48, the breaking change was identified as 566059a26 ("Autoparser - complete refactoring of parser architecture", PR #18675). This commit removed the dedicated TranslateGemma handler (common_chat_params_init_translate_gemma) from common/chat.cpp. The old code path was:

common_chat_templates_apply_jinja
  -> render_message_to_json (plain string content)
  -> detect [source_lang_code] / [target_lang_code] in template source
  -> common_chat_params_init_translate_gemma (transforms messages to required schema)
  -> apply Jinja template with transformed messages -> works

After the refactoring:

common_chat_templates_apply_jinja
  -> render_message_to_json (plain string content)
  -> common_chat_try_specialized_template (no TranslateGemma detection)
  -> autoparser fallback
  -> apply Jinja template with untransformed messages -> fails

The TranslateGemma Jinja template requires user message content to be an array with objects containing type, text, source_lang_code, and target_lang_code fields. The new autoparser passes plain string content, so the template's validation check throws.

Why The Old Handler Worked

The removed handler transformed messages before applying the template:

For each user message with string content, it wrapped that string into an array item
Added source_lang_code and target_lang_code fields (defaulting to en-GB, overridable via chat_template_kwargs)
Applied the Jinja template with the transformed messages

This transformation is not something the generic autoparser can do -- it requires model-specific knowledge of the expected schema.

Related Upstream References

Issue #20305 -- the upstream issue tracking this bug, closed as "not planned"; fix shared with maintainers
PR #18675 -- the refactoring that removed the handler (commit 566059a26)
PR #20956 -- open, attempting to fix by supporting extra fields on content parts

Fix

Approach

Restore the TranslateGemma handler in common_chat_try_specialized_template() so it intercepts the template before the autoparser fallback. Detection: Check for [source_lang_code] and [target_lang_code] in the template source string. These substrings appear in languages[source_lang_code] / languages[target_lang_code] within the TranslateGemma template and are unique to it. Transformation: For each user message with string content, wrap into the required array format:

{
  "type": "text",
  "text": "<original content>",
  "source_lang_code": "<from chat_template_kwargs or en-GB>",
  "target_lang_code": "<from chat_template_kwargs or en-GB>"
}

Placement: Function defined before common_chat_try_specialized_template (line ~975) so no forward declaration is needed. Detection added as the last check in the specialized template chain, just before return std::nullopt.

Safety Verification

Check	Result
Detection strings unique to TranslateGemma	Yes -- no other handler or template uses `[source_lang_code]` / `[target_lang_code]`
No conflict with Gemma4 detection	Yes -- TranslateGemma template does not contain `'<\\|tool_call\>call:'`
Pattern matches existing handlers	Yes -- same pattern as LFM2 (build `adjusted_messages`, pass as `messages_override`)
API compatible	Yes -- `common_chat_template_direct_apply_impl(tmpl, params, adjusted_messages)` is the current signature
Format enum valid	Yes -- uses `COMMON_CHAT_FORMAT_PEG_NATIVE` like all other specialized handlers

Testing

Metric	Before Fix (`cea560f48`)	After Fix (`5740bd414`)
Startup	Fails with Jinja exception	Clean, no errors
Template verification	`common_chat_verify_template: failed`	Passes
Example format	N/A (crashes before)	Renders correctly with language codes
Translation response	N/A	Model returns correct translation output

Test with translategemma-12B-Instruct-Q4_K_M.gguf:

Server starts cleanly, no template errors
Translation request returns: <start_of_turn>model\nJohn est dans la cuisine. (correct French translation)
The <start_of_turn>model\n prefix is expected template output (turn delimiter)

Deliverables

Files Created/Modified

common/chat.cpp -- fix patch (+51 lines, 2 hunks: handler function + detection in specialized template chain)
llama.cpp-how-we-fixed-translategemma-bug.md -- this file
restore-translategemma-specialized-template-handler.diff -- clean unified diff for maintainers
translategemma-llama.cpp-34df42f7b.log -- working log (build 8226) for reference
translategemma-llama.cpp-cea560f48.log -- failing log (build 8461) for reference
translategemma-12B-Instruct.jinja -- custom Jinja template file with language mappings

Git Commit

5740bd414 -- common : restore TranslateGemma specialized template handler

How It Started

The -fit sleep/wake fix and the checkpoint clustering fix both taught us to look for state being overwritten by refactoring. This time, the pattern was simpler: a dedicated handler that was removed during the autoparser overhaul and never replaced. The failing log made it obvious -- the same "Neither string content nor typed content is supported" warning appeared in both working and failing builds, but only the working build had the handler to transform the messages before the template saw them. What made this fix straightforward was recognizing that TranslateGemma's message schema requirement (array with language codes) is fundamentally different from any other template -- it cannot be handled by generic parsing rules alone.

Key Takeaways

Refactoring can lose model-specific handlers: The autoparser overhaul was comprehensive but dropped TranslateGemma's dedicated handler without a replacement.
Detection strings matter: [source_lang_code] in the template source is unique enough to serve as a reliable fingerprint for this model family.
Message transformation belongs in specialized handlers: When a template expects non-standard message schemas, the autoparser cannot adapt -- a dedicated handler is needed.
AI as assistive tool: Bug discovery, investigation, and solution design were human-led. AI helped trace commit history, verify API compatibility, and format documentation.

Non-native English speaker, French Assisted-by: llama.cpp:local pi