370953729b
- perf_stats aggregator lives in eval/, not model/: the import failed silently and EVERY perf column was empty (not just ttft). Now warns on stderr instead of swallowing. - repeats > 1 get their own checkpoint key (:rep2, :rep3, ...): repeat 2 previously restored repeat 1's predictions and finished instantly with identical scores. rep1 keeps the legacy key (existing checkpoints still resume). - repeats summary: report the MEAN score and aggregate time/tokens over ALL runs (was: last run only). - README: six-benchmark command as the primary example. Co-Authored-By: Claude <noreply@anthropic.com>
46 lines
1.5 KiB
Python
46 lines
1.5 KiB
Python
"""evalharness.model -- calling and deploying models.
|
|
|
|
Two plugin families, deliberately separate lifecycles:
|
|
ModelAdapter HOW to call (protocol): openai_compatible, mock, ...
|
|
Deployer HOW to run (environment): vllm, sglang, external (docker-pinned)
|
|
|
|
Model spec grammar (strings everywhere, no config ceremony):
|
|
mock / mock:boxed / mock:tool offline
|
|
openai/http://host:8000/v1?model_id any OpenAI-protocol endpoint (vllm, sglang, cloud)
|
|
deploy:vllm/qwen3-8b Deployer resolves endpoint (models.yaml pins env)
|
|
|
|
All adapters are async and return structured ModelOutput(text, tool_calls,
|
|
usage) -- the hinge the future agent loops hang on.
|
|
"""
|
|
|
|
from .adapter import (
|
|
ADAPTER_REGISTRY,
|
|
ModelAdapter,
|
|
OpenAICompatible,
|
|
MockAdapter,
|
|
register_adapter,
|
|
resolve_adapter,
|
|
parse_model_spec,
|
|
)
|
|
from .deployer import (
|
|
DEPLOYER_REGISTRY,
|
|
Deployer,
|
|
VLLMDeployer,
|
|
SGLangDeployer,
|
|
External,
|
|
deploy,
|
|
load_model_config,
|
|
register_deployer,
|
|
stop_all,
|
|
)
|
|
from .output import ModelOutput, ToolCall, Usage
|
|
from .runner import generate_predictions, run_eval
|
|
|
|
__all__ = [
|
|
'ModelAdapter', 'OpenAICompatible', 'MockAdapter', 'register_adapter',
|
|
'resolve_adapter', 'parse_model_spec', 'ADAPTER_REGISTRY',
|
|
'Deployer', 'VLLMDeployer', 'SGLangDeployer', 'External',
|
|
'register_deployer', 'deploy', 'stop_all', 'load_model_config', 'DEPLOYER_REGISTRY',
|
|
'ModelOutput', 'ToolCall', 'Usage', 'generate_predictions', 'run_eval',
|
|
]
|