Limit concurrent compile and run subprocesses. Heavy load no longer pushes correct solutions past the timeout.
Build Go code before it runs, so a cold compile does not use the run timeout.
Cap captured output at 1 MiB. Set a wait delay so a child that holds a pipe cannot hang the run.
Report a timeout as "timed out after Ns".
Pin TZ, PYTHONHASHSEED, PYTHONUTF8 and GOTOOLCHAIN for repeatable output.
Framework runs a category/subcategory test catalog against multiple
LLMs through an OpenAI-compatible endpoint. Deterministic evaluators
score each response. Reports render as table, markdown, or JSON.
lukaszraczylo