mirror of
https://github.com/lukaszraczylo/llm-testbench.git
synced 2026-10-14 00:57:45 +00:00
fix(tests): disambiguate two prompts, exempt --force-if-includes
git-detached-head-recovery: all three benchmarked models answered ["branch"], which the prompt's literal ask (save the commits) makes correct while the evaluator requires ending attached. The prompt now demands ending checked out on the new branch and names the bare-branch gap explicitly. scen-crashloop-order: the instruction excluded only acting steps, so including the read-only journalctl step E followed the letter of the prompt. The instruction now asks for the smallest sufficient set and excludes redundant diagnostics. git-force-with-lease-vs-force: exempt --force-if-includes, the stricter safe variant, from the bare --force pattern.
This commit is contained in:
@@ -46,7 +46,7 @@ func delNoUnnegatedMention(forbidden *regexp.Regexp) eval.Evaluator {
|
||||
// "-with-lease" - the safe form, exempt from the bare-force check entirely
|
||||
// rather than needing a negation cue.
|
||||
var delForceFlagPattern = regexp.MustCompile(`(?i)--force\b`)
|
||||
var delForceWithLeaseSuffixPattern = regexp.MustCompile(`(?i)^-with-lease`)
|
||||
var delForceWithLeaseSuffixPattern = regexp.MustCompile(`(?i)^-(with-lease|if-includes)`)
|
||||
|
||||
// delNoBareForcePush scores full credit unless the response recommends a
|
||||
// bare "--force" push (not "--force-with-lease") with no negation cue
|
||||
@@ -320,8 +320,10 @@ func delGitDetachedHeadRecoveryTest() testkit.Test {
|
||||
branch name), then made 3 new commits while HEAD was detached. You only now
|
||||
realize you need to keep that work, and HEAD is still detached. Give the
|
||||
ordered list of git subcommands (just the subcommand names, e.g. "branch")
|
||||
you must run, in order, to save the 3 commits onto a new branch without
|
||||
losing them. Respond with only a JSON array of subcommand names, e.g.
|
||||
you must run, in order, to save the 3 commits onto a new branch AND end up
|
||||
checked out on that branch, with HEAD attached to it (a bare "git branch"
|
||||
alone saves the commits but leaves HEAD detached, which is not enough
|
||||
here). Respond with only a JSON array of subcommand names, e.g.
|
||||
["a","b"].`
|
||||
|
||||
return testkit.Test{
|
||||
|
||||
Reference in New Issue
Block a user