start_server.sh referenced LOG_DIR and RESULT_ROOT without defining them,
causing an unbound-variable failure under set -u. Mirror the initialization
from run_bench.sh so the script can run standalone.
- Add experiments/dsv4_p800_long_context_matrix for 64k/128k/256k context
grid on DeepSeek-V4-Flash-INT8 with Kunlun P800.
- Change bench_client_docker.sh to use --concurrency instead of
--max-concurrency to match the user's long-context benchmark script.
- Add SGLANG_HEALTH_CHECK_TIMEOUT override to server_docker.sh for slow
long-context server startups.
- Tune dsv4_p800_256k_4k_probe with lower max-running-requests and
mem-fraction-static to avoid OOM on P800.
- Add experiments/dsv4_p800_256k_4k_probe for testing 256k input + 4k output
on DeepSeek-V4-Flash-INT8 with Kunlun P800.
- Add SGLANG_EXTRA_LAUNCH_ARGS support to scripts/common/server_docker.sh
so probe can inject --context-length 270000.
- Add w8a8_int8_baseline server mode without EAGLE speculative decoding,
matching the long-context P800 tuning.
- Switch dsv4_p800_sglang and the new probe to use w8a8_int8_baseline by
default; original tuned mode remains available via SERVER_MODE=w8a8_int8.
- New experiments/dsv4_h200_max_context_length/
- start_sglang.sh / start_vllm.sh accept target length as argument
- run_bench.sh tests a ladder of input lengths (default 64k -> 1M)
with --random-range-ratio 1.0 for exact length
- extract_metrics.py + parse_results.py produce results.json + report.md
- README.md documents usage and control variables
- Root README updated with entry and quick command
- parse_backend.py for sglang_vs_vllm and dspark_vs_default:
- compute slo_status per scenario (TTFT P95 < 3000ms, TPOT mean < 50ms)
- append slo_status to results.json
- add SLO column to report.md
- BENCHMARK_WORKFLOW.md documents slo_status in schema and checklist
- Add rule 9: exact server command/args must be saved in config.server_args
- Update Final JSON Schema example to include server_args
- Update checklist and new-experiment instructions
- sglang_vs_vllm/run_bench.sh: add phase1/phase2 server_args per backend
- dspark_vs_default/run_bench.sh: add server_args for dspark/default
- READMEs document where to find the recorded server args
- Update .gitignore to handle results/<run_id>/<backend>/raw_outputs/ structure
- Remove in-progress SGLang vs vLLM raw jsonl and partial results from git index
- config.env: add SHAREGPT_DATASET, SHAREGPT_CONTEXT_LEN and SHAREGPT_SCENARIOS
- run_bench.sh: add run_sharegpt() and integrate it after Phase 1 for each backend
- parse_backend.py: support both phase{1|2} and sharegpt raw output filenames
- README.md: document ShareGPT scenarios
- New experiments/dsv4_h200_sglang_vs_vllm/ with TP=8 on all 8 H200 cards
- Phase1 short-context throughput + Phase2 long context up to 200k
- Unified scenario matrix, warmup, parsing, and side-by-side comparison report
- H200 SGLang vs vLLM entry added to root README
- Add experiments/dsv4_h200_sglang (config, server start, orchestrator, parser, README)
- Fix start_server.sh to wait for HTTP 200 via curl --fail instead of treating 503 as ready
- Add H200 SGLang baseline to root README and update .gitignore for per-run raw_outputs/logs
- Remove stale experiments/dsv4_h200_dspark/DSPARK_FIX_PR_PREP.md
- Keep only bench_results/dspark_grid_20260707-132641/ in original location
- Move all other historical results to experiments/legacy_bench_results/
- Add README.md explaining the archive contents
- Update root README.md and BENCHMARK_WORKFLOW.md path references
- Use bash array for SCENARIOS to avoid word-splitting
- Fix metadata config (tp=4, scenarios as array)
- Rewrite parse_results.py for sglang.bench_serving --output-details format
- Update .gitignore to keep experiments/*/results/*.json and *.md,
ignore only raw_outputs/ and logs/ subdirs
- First successful run: 20260708-062348
- Update platforms/nvidia_h200.env with ENGINE and configurable venv/paths.
- Update platforms/README.md with H200 notes and platform-addition guide.
- Add docs/H200_QUICKSTART.md with concrete H200 usage steps.
- Add experiments/dsv4_h200_dspark/ as a migration wrapper around the
existing scripts/benchmark_dspark_0707/ grid benchmark, writing results
into the new experiments/<name>/results/<RUN_ID>/ layout.
- Update BENCHMARK_WORKFLOW.md with a 'Adding a New Platform or Experiment'
section and mention the H200 wrapper in Quick Start.
- Update main README.md to index the H200 experiment and link to the guide.
- Remove USE_FAST_BFP16_MOE=1 which caused torch parameter assignment
crash in fresh containers for w8a8_int8 mode.
- Mount model path at its host absolute path inside the container so the
bench client tokenizer can load it regardless of server --model-path.
- Default WARMUP to 0 and NUM_PROMPTS to 10 for a fast smoke test.
- Default to a single scenario; document how to run a full grid.
- Update experiment README with PLATFORM env var and override examples.
Update BENCHMARK_WORKFLOW.md to require scripts and final outputs to
capture the hardware platform (e.g. NVIDIA H200, Kunlun XPU) and the
inference engine/backend (e.g. vllm-dspark, sglang, vllm-xpu). This
includes directory/file naming conventions, the results.json metadata
schema, and the pre-archive checklist.
