# i can haz more fix!

## Session Summary

**Date:** June 30, 2026  
**Issue:** [ggml-org/llama.cpp#25023](https://github.com/ggml-org/llama.cpp/issues/25023)  
**Branch:** `ali0une-fixes`

---

## Discovery

The bug was discovered while analyzing `llama.cpp-router.log` from a long-running agent workflow. The server had context checkpoints configured with `min spacing = 1024`, but actual checkpoint spacings revealed the bypass:

```
Observed spacings: 2, 25, 75, 85, 92, 112, 130, 135, 150, 160, 184, 195, 226, 231, 235, 240, 246, 258, 259, 272, ...
```

Many checkpoints were spaced only **2-304 tokens apart**, far below `checkpoint_min_step = 1024`. With 12 checkpoints at ~100-token spacing, the window was only ~1.2K tokens wide instead of ~12K. The log showed:

- 23 mass erasure events (checkpoints invalidated in cascades)
- Task 3880 erasing 8 checkpoints spanning ~12K tokens, reprocessing 8270 tokens in 8.86s
- `f_keep` dropping to -1.0 or 0.291 (near-total cache misses)

## Investigation

### Root Cause Analysis

At line ~3576 of `tools/server/server-context.cpp`:

```cpp
do_checkpoint = do_checkpoint && (slot.prompt.checkpoints.empty() || is_last_user_message || n_tokens_start > slot.prompt.checkpoints.back().n_tokens + params_base.checkpoint_min_step);
```

The `is_last_user_message` OR clause fires on **every turn** in an agent workflow because each turn is a "last user message." This creates checkpoints at whatever spacing the turns happen to produce (2-304 tokens), completely bypassing `checkpoint_min_step`.

### Full Gate Chain Analysis

The `do_checkpoint` variable flows through 7 gates before reaching line 3576:

```
Gate 1 (line 3427): params_base.n_ctx_checkpoints > 0
Gate 2 (line 3430): slot.task->type == SERVER_TASK_TYPE_COMPLETION
Gate 3 (line 3437): seq_rm type check (FULL || RS || n_swa > 0)
Gate 4 (line 3547-3559): Mid-prompt gate -- skip unless is_user_start or near_prompt_end
Gate 5 (line 3568): pos_min >= 0
Gate 6 (line 3573): !has_mtmd
Gate 7 (line 3576): Spacing check -- THE BUGGY LINE
```

### Why `is_last_user_message` Fires Every Turn

In an agent workflow where each request is a separate conversation turn:

- Request 1: "user1" -- `is_last_user_message = true` (only user msg) -- checkpoint fires
- Request 2: "user1 + assistant1 + user2" -- `is_last_user_message = true` (user2 is last) -- checkpoint fires
- Request 3: "user1 + assistant1 + user2 + assistant2 + user3" -- `is_last_user_message = true` (user3 is last) -- checkpoint fires

Every request has exactly one "last user message" (the newest one), so `is_last_user_message` fires on every turn, bypassing spacing.

## Fix Attempts

### Option A: Remove the bypass entirely (too aggressive)

```cpp
do_checkpoint = do_checkpoint && (slot.prompt.checkpoints.empty() || n_tokens_start > slot.prompt.checkpoints.back().n_tokens + params_base.checkpoint_min_step);
```

Risk: may break the original intent of ensuring a checkpoint exists at the end of user input for better cache hits on the next turn.

### Option B: Apply a relaxed floor (selected)

```cpp
const int32_t checkpoint_floor = slot.prompt.checkpoints.empty()
    ? 0
    : slot.prompt.checkpoints.back().n_tokens + params_base.checkpoint_min_step / 2;
do_checkpoint = do_checkpoint && (slot.prompt.checkpoints.empty()
    || (is_last_user_message && n_tokens_start > checkpoint_floor)
    || n_tokens_start > slot.prompt.checkpoints.back().n_tokens + params_base.checkpoint_min_step);
```

Gives `is_last_user_message` a relaxed floor (`checkpoint_min_step / 2`) instead of no floor at all. For `checkpoint_min_step = 1024`, this means last-user-message checkpoints are at least 512 tokens apart.

### Option C: Only bypass when extending the window (DISCARDED)

```cpp
const bool extends_window = slot.prompt.checkpoints.empty() || n_tokens_start > slot.prompt.checkpoints.back().n_tokens;
do_checkpoint = do_checkpoint && (slot.prompt.checkpoints.empty()
    || (is_last_user_message && extends_window)
    || n_tokens_start > slot.prompt.checkpoints.back().n_tokens + params_base.checkpoint_min_step);
```

**DISCARDED.** This does not fix the bug. It only requires `n_tokens_start > back().n_tokens` -- i.e., the checkpoint just needs to be ahead of the last one. Since checkpoints are always created at increasing positions as the prompt grows, this condition is trivially true on every turn. It still produces ~100-token spacing, essentially the same as the current bypass.

### Why Option B Was Chosen

1. **It actually solves the problem** -- enforces real minimum spacing between checkpoints
2. **It preserves the intent** of `is_last_user_message` -- more frequent checkpoints at user boundaries, just not pathological frequencies
3. **The relaxed floor is a reasonable compromise** -- half the normal spacing still provides a meaningful window
4. **Preserves the `near_prompt_end` safety net** for very short turns
5. **Backward-compatible** when `checkpoint_min_step = 0`
6. **Minimal change** -- single-line replacement plus one local variable declaration

## Regression Verification

### Scenario Analysis

| Scenario | Turn Length | Before Fix | After Fix | Regression? |
|----------|-------------|-----------|-----------|-------------|
| Agent, short turns | ~100 tokens | spacing: 100 | spacing: ~512 | No -- window widens |
| Normal chat, long turns | ~800 tokens | spacing: 800 | spacing: 800 | No -- identical |
| Very short turns | ~20 tokens | spacing: 20 | skipped + near_prompt_end | No -- end-of-prompt checkpoint still fires |
| Single-turn | N/A | fires at end | fires at end | No -- identical |
| Spacing disabled | N/A | no floor | no floor (floor = 0) | No -- identical |
| Large min_step (8192) | varies | bypassed | relaxed floor 4096 | No -- still allows frequent checkpoints |

### Type Safety

- `checkpoint_floor` is `int32_t`, matching the existing `checkpoint_min_step` type
- The comparison `n_tokens_start > checkpoint_floor` uses the same mixed-type pattern as the original line (size_t vs int32_t)

## Testing

| Metric | Before Fix | After Fix | Change |
|--------|-----------|-----------|--------|
| Checkpoint count | 207 unique positions | 19 unique positions | -91% fewer |
| Min spacing | 1 token | 577 tokens | 577x improvement |
| Mean spacing | 480 tokens | 2023 tokens | 4.2x improvement |
| Erasure events | 23 | 3 | -87% |
| "Better prompt" searches | 15 | 3 | -80% |
| f_keep = -1.0 (cache miss) | 10+ occurrences | 1 occurrence | -90%+ |
| Prompt throughput | 715 tok/s | 893 tok/s | +25% |

### Erasure Events Detail

**Before:** 23 erasure events, including task 3880 erasing 8 checkpoints spanning ~12K tokens (reprocessing 8270 tokens in 8.86s).

**After:** Only 3 erasure events:
1. Task 2: erased 1 checkpoint at pos 15997 (initial request, expected)
2. Task 1830: erased 2 checkpoints at pos 61 and 30215 (slot reset, expected)

No cascade erasures.

### Checkpoint Spacing Distribution

**Before:** Clustered around 1-304 tokens (pathological):
```
1, 2, 7, 14, 15, 32, 39, 42, 60, 75, 77, 79, 83, 87, 91, 92, 105, 108, 110, 112, ...
```

**After:** All spacings above 512 (the relaxed floor), ranging 577-6355:
```
577, 765, 973, 1026, 1028, 1083, 1107, 1149, 1172, 1215, 1274, 1655, 2304, 2766, 2817, ...
```

## Deliverables

### Files Created/Modified
- `tools/server/server-context.cpp` -- fix patch (1 hunk, +7 lines / -1 line)
- `fix-is-last-user-message-bypass-of-checkpoint-min-step.diff` -- clean unified diff for maintainers
- `llama.cpp-checkpoint-erasure-issue-proposed-fix.txt` -- issue description with reproduction steps, root cause, and fix
- `llama.cpp-how-we-fixed-checkpoint-erasure-bug.md` -- this file
- `llama.cpp-ali0une-fix-checkpoints.md` -- full analysis document
- `llama.cpp-router-pre-fix.log` -- pre-fix log for reference

### Git Commit
`ee427b8f5` -- server : enforce relaxed spacing floor for last-user-message checkpoints (#25023)

## How It Started

The `-fit` sleep/wake fix taught us to look for patterns: state being overwritten by side effects, guards that cannot distinguish user intent from system behavior. This time, the pattern was in the checkpoint logs -- spacings of 2, 25, 75 tokens when the config said 1024 minimum. The `is_last_user_message` bypass looked reasonable in isolation (ensure a checkpoint at the last user message) but was pathological in practice (fire on every turn, collapse the window).

What made this fix straightforward was the lessons from `-fit`: small, targeted changes in the right place (`server-context.cpp`), preserving computed state rather than letting it be overwritten by side effects.

## Key Takeaways

- **Bypasses need floors:** An unconditional bypass of a spacing check is equivalent to no check at all. Even a relaxed floor prevents pathological behavior.
- **Gate chain analysis matters:** Tracing all 7 gates before the buggy line revealed that `near_prompt_end` already guarantees end-of-prompt checkpoints, making the `is_last_user_message` bypass less critical than it appeared.
- **Log evidence is gold:** The pre-fix log (3513 lines, 207 checkpoint positions) vs post-fix log (691 lines, 19 positions) tells the entire story quantitatively.
- **AI as assistive tool:** Bug discovery, investigation, and solution design were human-led. AI helped trace code flow, verify regression scenarios, and format documentation.

---

*Non-native English speaker, French*  
*Assisted-by: llama.cpp:local pi*