An unbounded docker rm against a bloated daemon hangs for minutes and
silently eats the worker pool: 7 of 8 scoring workers were observed
stuck in cleanup while only 1 execution ran.
Co-Authored-By: Claude <noreply@anthropic.com>
- docker exec: named containers; a timed-out/killed 'docker run' only
kills the CLI client while the container lives on (--rm fires on
EXIT) -- rm -f the name on timeout/interrupt so runs stop leaking
- exit 125 = daemon-side failure, not model failure: retry up to 2x
(a bloated daemon was turning healthy samples into pass=0)
- scoring milestones: first completion logs immediately, then every
~5% (10% was too sparse when docker is slow: minutes of silence
right after the 'scoring' phase starts, looks hung)
Co-Authored-By: Claude <noreply@anthropic.com>
