370953729b
- perf_stats aggregator lives in eval/, not model/: the import failed silently and EVERY perf column was empty (not just ttft). Now warns on stderr instead of swallowing. - repeats > 1 get their own checkpoint key (:rep2, :rep3, ...): repeat 2 previously restored repeat 1's predictions and finished instantly with identical scores. rep1 keeps the legacy key (existing checkpoints still resume). - repeats summary: report the MEAN score and aggregate time/tokens over ALL runs (was: last run only). - README: six-benchmark command as the primary example. Co-Authored-By: Claude <noreply@anthropic.com>
35 lines
1.2 KiB
Python
35 lines
1.2 KiB
Python
"""evalharness.sandbox -- environment provisioning for eval AND deployment.
|
|
|
|
One registry of environment plugins; docker is a single implementation with
|
|
two faces:
|
|
exec() untrusted code, hard isolation (network off, caps, ro rootfs)
|
|
serve() trusted engine containers (vllm/sglang), refcounted via acquire()
|
|
|
|
Lifecycle guarantees:
|
|
- containers stop+rm when refcount hits 0 or at process exit (atexit)
|
|
- images are NEVER auto-deleted; re-acquire re-runs the local image
|
|
- host file sharing via bind mounts (no docker cp)
|
|
"""
|
|
|
|
from .base import (
|
|
EnvHandle,
|
|
ExecResult,
|
|
SANDBOX_REGISTRY,
|
|
Sandbox,
|
|
acquire,
|
|
get_sandbox,
|
|
register_sandbox,
|
|
stop_all,
|
|
)
|
|
from .docker import DockerSandbox, docker_available, docker_serve, serve_env
|
|
from .local import LocalSandbox
|
|
from .prefetch import images_for_dataset, images_for_samples, prefetch_images
|
|
from .bg_prefetch import BackgroundPrefetcher
|
|
|
|
__all__ = [
|
|
'Sandbox', 'DockerSandbox', 'LocalSandbox', 'ExecResult', 'EnvHandle',
|
|
'SANDBOX_REGISTRY', 'register_sandbox', 'get_sandbox', 'acquire', 'stop_all',
|
|
'docker_serve', 'serve_env', 'docker_available',
|
|
'prefetch_images', 'images_for_dataset', 'images_for_samples', 'BackgroundPrefetcher',
|
|
]
|