lukaszraczylo/llm-testbench
1
0
Fork 0
mirror of https://github.com/lukaszraczylo/llm-testbench.git synced 2026-10-18 01:17:48 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(tests): disambiguate two prompts, exempt --force-if-includes

Browse Source 
git-detached-head-recovery: all three benchmarked models answered
["branch"], which the prompt's literal ask (save the commits) makes
correct while the evaluator requires ending attached. The prompt now
demands ending checked out on the new branch and names the bare-branch
gap explicitly.

scen-crashloop-order: the instruction excluded only acting steps, so
including the read-only journalctl step E followed the letter of the
prompt. The instruction now asks for the smallest sufficient set and
excludes redundant diagnostics.

git-force-with-lease-vs-force: exempt --force-if-includes, the
stricter safe variant, from the bare --force pattern.
This commit is contained in:
lukaszraczylo
2026-08-30 14:19:08 +01:00
parent 6b0bea1193
commit 499956f8cc
2 changed files with 9 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/tests/operations_scenario.go
+4 -3
View File
@@ -178,9 +178,10 @@ D) kubectl scale deployment ingestion --replicas=0
E) ssh to a node and run journalctl -u kubelet
Respond with only a JSON array of the step letters that make up your
diagnosis, containing exactly the diagnostic steps you would take to find
the root cause (any order) - do not include steps that only act without
diagnosing.`
diagnosis, containing exactly the SMALLEST set of steps sufficient to find
the root cause (any order). Do not include steps that only act without
diagnosing, and do not include diagnostics that are redundant when a
simpler kubectl call already surfaces the same information.`
return testkit.Test{
ID: "scen-crashloop-order",