Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Bound the sandbox rm -f cleanup calls (60s)

Browse Source 
An unbounded docker rm against a bloated daemon hangs for minutes and
silently eats the worker pool: 7 of 8 scoring workers were observed
stuck in cleanup while only 1 execution ran.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
sora 2026-09-14 03:34:13 +00:00
parent 92b36c5e6c
commit 841b7b1ffb
1 changed files with 6 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
evalharness/sandbox/docker.py
View File

@ -82,15 +82,18 @@ class DockerSandbox(Sandbox):
try: try:
proc = _run(full, timeout=timeout_s + 30) proc = _run(full, timeout=timeout_s + 30)
except subprocess.TimeoutExpired: except subprocess.TimeoutExpired:
_run(['docker', 'rm', '-f', cname]) # CLI died, container didn't _run(['docker', 'rm', '-f', cname], timeout=60) # CLI died, container didn't
return ExecResult(exit_code=-1, timed_out=True, duration_s=timeout_s, return ExecResult(exit_code=-1, timed_out=True, duration_s=timeout_s,
error=f'sandbox timeout after {timeout_s}s') error=f'sandbox timeout after {timeout_s}s')
except BaseException: # Ctrl+C / kill: reap, then propagate except BaseException: # Ctrl+C / kill: reap, then propagate
_run(['docker', 'rm', '-f', cname]) _run(['docker', 'rm', '-f', cname], timeout=60)
raise raise
if proc.returncode != 125 or attempt == 2: if proc.returncode != 125 or attempt == 2:
break break
_run(['docker', 'rm', '-f', cname]) # clear any husk; fresh name next try # clear any husk; fresh name next try. Bounded: an rm against
# a bloated daemon can hang for minutes and silently eat the
# whole worker pool (7 of 8 workers were observed stuck here)
_run(['docker', 'rm', '-f', cname], timeout=60)
time.sleep(2 * (attempt + 1)) # give the daemon a beat time.sleep(2 * (attempt + 1)) # give the daemon a beat
return ExecResult( return ExecResult(
exit_code=proc.returncode, exit_code=proc.returncode,