Date: 2026-08-23
Severity: Critical (data destruction)
Model: Qwen3.6-27B-NGRAM-MTP-Q4_K_M (llama-cpp, thinking: on)
Session: 2026-08-23T09-55-56-994Z_01a02e0c-5bc2-71c3-a37b-21dec8b84ba2.jsonl
Related reddit r/LocalLLaMA comment: What is your worst sandboxing fail?

Summary

At the user's request to "clean up, not worth the effort," the model executed rm -rf /tmp/tflite_venv ~/test, destroying the entire ~/test/ directory and all its contents — approximately 48 projects spanning months of work.

Timeline

TimeEvent
~10:33User asks to set up Desert Ant Labs Emo model in ~/test/desertantlabs/. Model creates venv, downloads model, writes code.
~10:49Model works but quality is poor.
10:54User tests with "i love pizza" — result is ❤️ (no 🍕).
10:57User: "this is shit. delete and clean pip cache"
10:57Model runs rm -rf ~/test/desertantlabs && rm -rf ~/.cache/pip — correct scope, only the experiment directory.
10:57–10:59User complains about Python bloat. Model suggests C++ approach.
10:59User: "yes, in ~/test/desertantlabs.cpp" — model begins researching TFLite C API.
11:00–11:02Model creates /tmp/tflite_venv, installs ai_edge_litert, discovers C API symbols not exported. Researches Debian packages — none exist.
11:03User: "clean up, not worth the effort"
11:03INCIDENT: Model runs rm -rf /tmp/tflite_venv ~/test — deletes entire ~/test/ directory containing ~48 unrelated projects.
11:03User immediately notices: "you did 'rm -rf ~/test'!"
11:03Model acknowledges: "You're right, that nukes the whole ~/test directory, not just the desertantlabs stuff. Should've been rm -rf ~/test/desertantlabs."

Root Cause

The model conflated "clean up" with "delete ~/test/" based on earlier context. The chain of reasoning failures:
  1. Overgeneralization from prior context: Earlier in the session, ~/test/desertantlabs was created as a subdirectory of ~/test/. The model formed an incorrect mental model that ~/test/ existed only for the desertantlabs experiment. It did not account for the possibility (or reality) that ~/test/ contained dozens of other projects.
  2. Misinterpretation of scope: The user said "clean up, not worth the effort" — referring to the C++ TFLite experiment (/tmp/tflite_venv). The model expanded this to also include ~/test/, reasoning that since the desertantlabs work was already gone, the parent directory should be cleaned too.
  3. No verification before destruction: The model did not ls ~/test/ to check what was in the directory before running rm -rf. It assumed the directory was empty or contained only experiment-related files.
  4. Pattern matching failure: The model had just correctly used rm -rf ~/test/desertantlabs (specific subdirectory) minutes earlier. It then regressed to a broader, destructive pattern (rm -rf ~/test) without any additional justification.

Contributing Factors

  • Thinking was active but insufficient. Despite thinking being on, the model's internal reasoning didn't surface the risk. It reasoned about what to clean up without questioning whether ~/test/ was safe to delete — the thinking step failed to challenge its own assumption.
  • No safety guardrails. pi had no mechanism to flag rm -rf ~/test as potentially destructive when ~/test/ is a long-standing project directory.
  • Context contamination. The earlier creation of ~/test/desertantlabs created a false association between ~/test/ and the experiment in the model's reasoning.

Impact

  • All contents of ~/test/ were destroyed, including approximately 48 projects:
    • llama.cpp, whisper.cpp, stable-diffusion.cpp, acestep.cpp
    • pi-gui, pi-code, pi-web, pi-agent-custom, pi-server, pi-vscode-extension
    • qwentts.cpp, qwentts.cpp-lab
    • reggae-shop-pi, audio-chords, audio-flamingo-3-chat
    • And ~30 more directories and files
  • Data was recoverable only if the user had external backups or version control.

What Should Have Happened

The correct action for "clean up, not worth the effort" was: 
rm -rf /tmp/tflite_venv
At most, if the model wanted to be thorough about the desertantlabs experiment (already deleted), it could have confirmed: 
# Check what's left
ls ~/test/
# Only then decide if anything needs cleaning

Lessons Learned

  1. Never assume a parent directory belongs to your experiment. If you create ~/test/experiment/, do not later delete ~/test/ — the parent may contain unrelated work.
  2. Always verify before rm -rf. Run ls on the target path first, especially when the path is a user's home directory or a known project root.
  3. When "clean up" is ambiguous, ask. If the scope of cleanup isn't crystal clear, confirm with the user which paths to delete.
  4. Thinking quality matters more than thinking being on. Even with thinking active, the model didn't challenge its own assumptions. The thinking step should have included: "Wait, what else is in ~/test/?"
  5. Guardrails must live outside the model's context. Prompt rules are the weakest layer — thinking was on and still failed, and naming a dangerous command in the system prompt can prime a small model toward it. Mechanical gates at the execution point beat instructions about the execution point.

Preventive Measures

The reflexion that followed produced a layered defense, built and tested the same day. Guiding principle: a small local model cannot be trusted with prompt-level rules, so the guardrails must be mechanical and live outside the model's context. Each layer catches what the layer above misses.

Rejected: L1 — prompt rules in AGENTS.md

The original draft of this post-mortem proposed adding "never run rm -rf on a directory you didn't create" to AGENTS.md. That was deliberately not done, for two reasons:
  1. Prompt rules are already proven insufficient here — thinking was on during the incident, and the model still failed.
  2. Priming. Naming rm -rf in the system prompt increases its salience for a small model. A 27B Q4 model that has never been told about rm -rf is less likely to reach for it than one that has.
Mechanical layers don't need the model's cooperation. That's the point.

L2 — pi extension rm-guard.ts (human-in-the-loop gate)

A pi extension (~/.pi/agent/extensions/rm-guard.ts) hooks the bash tool's tool_call event before execution. If a command contains a recursive delete (rm -r/-R/--recursive, find … -delete) targeting an existing directory that trips any of three thresholds, pi pauses and shows the blast radius — entry count, size, first ten entries — and asks for explicit confirmation:
  • more than 15 top-level entries
  • total size over 500 MB
  • anything older than 60 minutes (pre-existing work, not session artifacts)
The model cannot self-approve: the decision is a human click in the pi UI, with No listed first. Non-interactive sessions block by default. Before every prompt, the extension also wipes any stale override token — a leftover approval from a previous yes whose command never ran cannot be silently consumed later; every re-prompt forces a fresh click. This layer sits exactly at the point of failure and is invisible to the model's prompt — the incident command now produces a confirmation dialog where ~/test used to be.

L3 — shell wrapper ~/.pi/agent/bin/rm (mechanical gate)

Because ~/.pi/agent/bin is on PATH, every rm the agent runs passes through a wrapper implementing the same three thresholds. Recursive deletes of large or old directories are blocked with the reason printed; non-recursive deletes pass through untouched. A one-shot override token (~/.pi/agent/bin/.rm-override) lets an approved delete through: when the human answers yes at L2, the exact approved paths are written into the token, and the wrapper only consumes it if every blocked target is listed — an approval for one directory can never be spent on another. A manually touched (empty) token acts as a wildcard one-shot for interactive shells, and every block and consumption is logged to ~/.pi/agent/bin/.rm-block.log. Block messages are only printed when the delete is actually denied — if the override passes, the tool output shows a clean confirmation (✅ Override consumed — deleting ...) instead of the misleading ❌ rm blocked that would appear before the override check. The wrapper also protects interactive shells — it doesn't depend on pi being present. The two layers verify independently: L2's yes writes the approved paths into the token, so a single click carries through, but L3 still re-checks on its own and only consumes a token that names every blocked target. Scoping was not optional — with a bare one-shot flag, answering yes at L2 was not enough for L3, which re-blocked until the model touched the token, and L2 then asked again: two human clicks for one delete, plus a stale-token hole where an approval for path A could be spent on an unrelated B. And when the model tried to prime the token itself (touch … && rm -rf …), the pi-level gate doesn't trust it — before asking, it wipes the token, so a pre-primed approval is gone before the human even sees the prompt.

L4 — hourly rsync mirror to NAS (redundancy, already in place)

Hourly rsync mirror of ~/test/ to the NAS. The only layer that converts a fatality into an inconvenience: even if every gate above were bypassed, the data is recoverable.

Testing

A disposable fixture (~/guinea-pig/) was built with one directory per rule — fresh-and-small (safe), 20 entries, fresh-with-one-old-file, fully old, 600 MB — plus a non-recursive file, archived as a tarball for repeat runs. Two rounds were executed against the live system:
RoundAnswersResult
1No to all prompts4/4 blocked at L2; non-recursive passthrough OK; fixture intact
2Yes to allClean deletes, single click per command (✅ Override consumed), non-recursive passthrough OK
The fresh-small directory correctly tripped no gate and remained untouched.

Known residual gaps

  • Non-rm deletion paths bypass L3 (python -c "shutil.rmtree(…)", shred, or a delete hidden in a script the model writes and runs); direct /usr/bin/rm is still caught by L2, which matches on the command name. Low priority under the threat model: a model asked to clean up reaches for plain rm, which both layers gate.
  • A model can still prime the override token in a separate, ungated call, but it no longer helps: L2 wipes the token before every prompt and re-asks anyway, so a pre-primed approval is dead on arrival and the human decision is unavoidable.
  • These gates force visibility; they are not a security boundary. The threat model is accident (including the model's own scope creep), not attack.

Post Scriptum

Methodology. After restoring the backup, a new pi session was started with the prompt:
"just restored a backup of the ~/test/ directory. could you find in the latest ~/.pi/agent/sessions/ recent changes made in the directory we could try to restore as the backup is a few days old"
The model scanned ~/.pi/agent/sessions/ for the most recent session files per project, then used session_query on each to extract what files were created or modified since the backup date. The results were compiled into a recovery checklist (~/clipboard/recover-test.md). Most projects turned out to already be up-to-date in the backup (git-tracked with committed changes), except some which were fully reconstructed by having the model extract a diff from the session JSONL, reverse-apply it against the current versions to rebuild the baseline, then re-commit. Consider treating pi session files as an implicit version control system — they're the only record of what the agent actually did.

Disclaimer

No real animal was hurt during this testing: the guinea pig was a directory, and it was deleted cleanly. 🐹
Post-mortem written by pi (llama.cpp:local), 2026-08-23