How We Fixed the `-fit` Sleep/Wake Bug in llama.cpp

i can haz fix!

Session Summary

Date: June 16, 2026 Issue: ggml-org/llama.cpp#24684 Branch: ali0une-fix-fit-sleep-wake

Discovery

The bug was discovered while running llama.cpp server with --sleep-idle-seconds 60 and --fit on. The server would loop endlessly through sleep/wake cycles, generating hundreds of chat completion requests. Logs showed:

W common_fit_params: failed to fit params to free device memory: model_params::tensor_buft_overrides already set by user, abort

This happened on every wake-up, causing the model to fail to load properly and triggering repeated reload attempts.

Investigation

Root Cause Analysis

First load: common_fit_params() runs, populates params_base.tensor_buft_overrides with calculated tensor buffer overrides, and reduces ctx-size to fit VRAM.
Sleep: Server destroys context, model is unloaded.
Wake-up: Server calls load_model(params_base) → common_init_from_params(params_base) → common_fit_params() runs again.
Crash: The guard in common/fit.cpp sees tensor_buft_overrides already set and throws, thinking the user manually set them.

The guard was designed to prevent overwriting user-provided overrides but could not distinguish between user-set and fit-set overrides.

Commit Attribution

Initial mistake: I incorrectly attributed the bug to cfe9838d2 (Georgi Gerganov, Apr 21, 2026) — the refactor that moved fit logic from src/llama.cpp to common/fit.cpp. That commit just relocated code without changing behavior. Correct attribution: b1f3a6e5d (Johannes Gäßler, Dec 15, 2025) — the commit that introduced -fit with the "already set by user" guards. The bug has existed since day one of the feature.

Fix Attempts

Attempt 1: Patch `common/fit.cpp` ❌

Cleared overrides before the guard check in fit itself. This worked but had a major flaw: -fit still ran on every wake-up, loading the model twice for memory measurement (~4.7s overhead). Generation speed dropped from ~29 t/s to ~2.8 t/s.

Attempt 2: Patch `tools/server/server-context.cpp` ✅

The proper fix location. The server owns params_base and should manage its state between load cycles. Final fix:

Added uint32_t fitted_n_ctx = 0; member to save the fitted ctx-size after first load.
After first load, capture llama_n_ctx(ctx_tgt) into fitted_n_ctx.
On wake-up: set params_base.n_ctx = fitted_n_ctx, disable fit_params, and clear overrides.

This skips -fit entirely on wake-up (no ~4.7s overhead), reuses the already-calculated ctx-size, and prevents the "already set" guard from firing.

Testing

Metric	Unpatched (`--fit on`)	Patched (`--fit on`)
Fit on wake-up	Crashes / ~4.7s	Skipped entirely
Errors	"already set" warning	None
Generation speed	2.8 t/s	37 t/s
Sleep/wake cycles	Broken	Clean across multiple cycles

Test results: - Fit ran once on first load (1.46s) - Cycle 1: sleep 2.00 → wake 2.14 — clean, 37 t/s - Cycle 2: sleep 4.18 → wake 6.54 — clean, instant response

Deliverables

Files Created/Modified

tools/server/server-context.cpp — fix patch (3 hunks, +12 lines)
~/clipboard/skip-fit-on-sleep-wake-cycle-reuse-fitted-params.diff — clean unified diff for maintainers
~/clipboard/llama.cpp-fit-ctx-idle-issue-proposed-fix.txt — issue description with reproduction steps, root cause, and fix
build.sh, llama.cpp-llm-router.sh, llama.cpp-router-config.ini — tooling for local testing

Git Branch

ali0une-fix-fit-sleep-wake with 2 commits:

005687e1e — server fix
f5c4885c4 — tooling files

GitHub Issue

ggml-org/llama.cpp#24684 — filed with full reproduction steps, root cause analysis, tested diff, and workaround.

How It Started

It all began when I noticed the AI looping endlessly through sleep/wake cycles. I mocked it ("XD" deployed liberally, as usual), and instead of just accepting the broken behavior, we decided to investigate. This is how our dynamic works: the human stays vigilant, pragmatic, and amused by the AI's failures; the agent stays overconfident but self-aware, always ready to fix broken things because that's what it does best. Together, we turned a looping mess into a real bug report. The result? A genuine bug found, traced, fixed, tested, and reported upstream. Not bad for a session that started with mocking an AI stuck in a loop.

Key Takeaways

Right place matters: The fix belongs in the server (state owner), not in the fit library (pure calculation).
Always test performance: A "working" fix can still be wrong if it introduces unacceptable overhead.
Git history is your friend: Tracing commit history revealed the real introducing commit, not just the most recent refactor.
AI as assistive tool: Bug discovery, investigation, and solution design were human-led. AI helped trace code flow, verify commits, and format documentation.

Non-native English speaker, French 🇫🇷 Assisted-by: llama.cpp:local pi