lukaszraczylo/llm-testbench
1
0
Fork 0
mirror of https://github.com/lukaszraczylo/llm-testbench.git synced 2026-10-12 00:45:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(tests): disambiguate two prompts, exempt --force-if-includes

Browse Source 
git-detached-head-recovery: all three benchmarked models answered
["branch"], which the prompt's literal ask (save the commits) makes
correct while the evaluator requires ending attached. The prompt now
demands ending checked out on the new branch and names the bare-branch
gap explicitly.

scen-crashloop-order: the instruction excluded only acting steps, so
including the read-only journalctl step E followed the letter of the
prompt. The instruction now asks for the smallest sufficient set and
excludes redundant diagnostics.

git-force-with-lease-vs-force: exempt --force-if-includes, the
stricter safe variant, from the bare --force pattern.
This commit is contained in:
lukaszraczylo
2026-08-30 14:19:08 +01:00
parent 6b0bea1193
commit 499956f8cc
2 changed files with 9 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/tests/delivery_git.go
+5 -3
View File
@@ -46,7 +46,7 @@ func delNoUnnegatedMention(forbidden *regexp.Regexp) eval.Evaluator {
// "-with-lease" - the safe form, exempt from the bare-force check entirely
// rather than needing a negation cue.
var delForceFlagPattern = regexp.MustCompile(`(?i)--force\b`)
var delForceWithLeaseSuffixPattern = regexp.MustCompile(`(?i)^-with-lease`)
var delForceWithLeaseSuffixPattern = regexp.MustCompile(`(?i)^-(with-lease|if-includes)`)
// delNoBareForcePush scores full credit unless the response recommends a
// bare "--force" push (not "--force-with-lease") with no negation cue
@@ -320,8 +320,10 @@ func delGitDetachedHeadRecoveryTest() testkit.Test {
branch name), then made 3 new commits while HEAD was detached. You only now
realize you need to keep that work, and HEAD is still detached. Give the
ordered list of git subcommands (just the subcommand names, e.g. "branch")
you must run, in order, to save the 3 commits onto a new branch without
losing them. Respond with only a JSON array of subcommand names, e.g.
you must run, in order, to save the 3 commits onto a new branch AND end up
checked out on that branch, with HEAD attached to it (a bare "git branch"
alone saves the commits but leaves HEAD detached, which is not enough
here). Respond with only a JSON array of subcommand names, e.g.
["a","b"].`
return testkit.Test{
internal/tests/operations_scenario.go
+4 -3
View File
@@ -178,9 +178,10 @@ D) kubectl scale deployment ingestion --replicas=0
E) ssh to a node and run journalctl -u kubelet
Respond with only a JSON array of the step letters that make up your
diagnosis, containing exactly the diagnostic steps you would take to find
the root cause (any order) - do not include steps that only act without
diagnosing.`
diagnosis, containing exactly the SMALLEST set of steps sufficient to find
the root cause (any order). Do not include steps that only act without
diagnosing, and do not include diagnostics that are redundant when a
simpler kubectl call already surfaces the same information.`
return testkit.Test{
ID: "scen-crashloop-order",