- Document the rule in docs/EXPERIMENT_GUIDE.md section 7
- Update all config.env SCENARIOS to follow the 5x rule
- Migrate dsv4_h200_sglang, dsv4_h200_vllm, dsv4_p800_sglang
from global NUM_PROMPTS to per-scenario num_prompts
- Keep long-context phase2 as exception (low counts documented)
- Add legacy 3-field fallback in run_bench.sh loops
- P800 INT8 cannot sustain >1 concurrent long-context request without OOM;
set all scenarios to concurrency=1.
- Keep max_running=2 because max_running=1 triggers req_to_token_pool
allocation failures in this image.
- Reduce 64k max_context_len to 72000 (65536 + 4096 + pad).
- Make run_bench.sh resilient to container death: detect dead container and
skip remaining scenarios in the group instead of crashing the orchestrator.
- Update README to document the concurrency limitation.
Without chunked prefill the server OOM/crashes (exit 137) during or
immediately after concurrent 64k-context scenarios. Speculative decoding
remains disabled as requested.
These were failed/intermediate smoke-test runs created while debugging the
managed server startup. Keep the repository clean; new runs can be reproduced
with experiments/dsv4_p800_sglang/run_bench.sh.
Add experiments/dsv4_p800_max_context_length/ to find the longest input
context that P800 SGLang INT8 can serve for DeepSeek-V4-Flash-INT8.
- Docker-based server start with dynamic --context-length.
- Single-request, single-output-token probes for each candidate length.
- Records success/failure, server args, and metrics per length.
- Generates results.json and report.md summarizing the max supported length.
- Smoke-tested at 4096 tokens successfully.
start_server.sh referenced LOG_DIR and RESULT_ROOT without defining them,
causing an unbound-variable failure under set -u. Mirror the initialization
from run_bench.sh so the script can run standalone.
- New experiments/dsv4_h200_max_context_length/
- start_sglang.sh / start_vllm.sh accept target length as argument
- run_bench.sh tests a ladder of input lengths (default 64k -> 1M)
with --random-range-ratio 1.0 for exact length
- extract_metrics.py + parse_results.py produce results.json + report.md
- README.md documents usage and control variables
- Root README updated with entry and quick command
- parse_backend.py for sglang_vs_vllm and dspark_vs_default:
- compute slo_status per scenario (TTFT P95 < 3000ms, TPOT mean < 50ms)
- append slo_status to results.json
- add SLO column to report.md
- BENCHMARK_WORKFLOW.md documents slo_status in schema and checklist
- Add rule 9: exact server command/args must be saved in config.server_args
- Update Final JSON Schema example to include server_args
- Update checklist and new-experiment instructions
- sglang_vs_vllm/run_bench.sh: add phase1/phase2 server_args per backend
- dspark_vs_default/run_bench.sh: add server_args for dspark/default
- READMEs document where to find the recorded server args
- Update .gitignore to handle results/<run_id>/<backend>/raw_outputs/ structure
- Remove in-progress SGLang vs vLLM raw jsonl and partial results from git index
- config.env: add SHAREGPT_DATASET, SHAREGPT_CONTEXT_LEN and SHAREGPT_SCENARIOS
- run_bench.sh: add run_sharegpt() and integrate it after Phase 1 for each backend
- parse_backend.py: support both phase{1|2} and sharegpt raw output filenames
- README.md: document ShareGPT scenarios
- New experiments/dsv4_h200_sglang_vs_vllm/ with TP=8 on all 8 H200 cards
- Phase1 short-context throughput + Phase2 long context up to 200k
- Unified scenario matrix, warmup, parsing, and side-by-side comparison report
- H200 SGLang vs vLLM entry added to root README
- Add experiments/dsv4_h200_sglang (config, server start, orchestrator, parser, README)
- Fix start_server.sh to wait for HTTP 200 via curl --fail instead of treating 503 as ready
- Add H200 SGLang baseline to root README and update .gitignore for per-run raw_outputs/logs
- Remove stale experiments/dsv4_h200_dspark/DSPARK_FIX_PR_PREP.md
