- Add envs/UV_ENV_SETUP.md with standard commands for creating
vLLM and SGLang virtual environments using uv.
- Configure UV_CACHE_DIR under envs/ to avoid polluting home directory.
- Include cu129-specific reinstall steps for SGLang kernel packages.
- Update envs/README.md to reference the new guide.
- Add RESUME_RUN_ID env var for breakpoint resume in adaptive benchmarks.
When set, the script reuses an existing result directory and skips
already-tested (TP, DP, ISL, OSL) shapes based on adaptive_shapes.jsonl.
- Fix RUN_ID unbound variable in resume mode.
- Fix jq query to use -s (slurp) for jsonl files.
- Add resume skip logic to DRY_RUN mode as well.
- Rename DSL -> OSL across all adaptive benchmark files for consistency:
- scripts/common/adaptive_bench_lib.sh
- scripts/common/adaptive_concurrency.py
- experiments/dsv4_h200_vllm_tp_dp_matrix/adaptive_config.env
- experiments/dsv4_h200_vllm_tp_dp_matrix/run_adaptive_concurrency.sh
- experiments/dsv4_h200_sglang_tp_dp_matrix/adaptive_config.env
- experiments/dsv4_h200_sglang_tp_dp_matrix/run_adaptive_concurrency.sh
- experiments/ADAPTIVE_CONCURRENCY_USAGE.md
- Implemented a bash script to run six benchmark experiments sequentially.
- Added features for fault tolerance, out-of-memory recovery, and checkpoint management.
- Included detailed logging for monitoring experiment progress and results.
- Integrated GPU memory cleanup checks before each experiment.
- Provided functionality to resume experiments from checkpoints.
- Run vLLM server inside vllm/vllm-openai:latest container on H200.
- Run benchmark client inside lmsysorg/sglang:latest using
sglang.bench_serving --backend vllm, matching the sglang reference.
- Use RUNTIME_BASE for self-contained logs/pids/tmp instead of hardcoded
/data/user1/yy paths.
- Update model path to /data3/hf_models/DeepSeek-V4-Flash.
- Sample only low/high concurrency endpoints to match sglang matrix.
- Fix docker 'invalid reference format' caused by DOCKER_CLIENT_IMAGE not
being exported to bash -c subshells used by run_bench_serving.
- Add offline dummy ShareGPT dataset for random benchmark mode.
- Create configuration file for the custom benchmark experiment.
- Implement result parsing script to handle JSONL outputs from the benchmark.
- Develop run script to orchestrate the benchmark execution, including server management and health checks.
- Add server start script to launch a single vLLM service on all available GPUs.
- Fix docker invalid reference format by exporting DOCKER_IMAGE and
USE_DOCKER_CLIENT to bash subshells (run_bench_serving)
- Add persistent container mode: start_sglang_container.sh +
run_sglang_in_container.sh for faster config switching
- Add README.md with setup and migration guide
export -f only exports the function definition, not the variables it
references. MODEL_PATH and RESULT_BASE were empty inside timeout bash -c
subshells, causing docker run -v ::ro invalid spec. Add explicit export
for all variables used by run_bench_serving().
export -f only exports the function definition, not the variables it
references. DATASET_PATH was empty inside timeout bash -c subshells,
causing docker run -v ::ro invalid spec even after the dynamic vol_args
fix. Add explicit export after sourcing config.env.
When DATASET_PATH is empty, -v ::ro is passed to docker run, which
produces 'invalid spec: ::ro: empty section between colons'. Build
the volume args dynamically and only add the dataset mount when the
path is non-empty.
timeout(1) cannot execute shell functions directly; it needs a standalone
command. Export run_bench_serving via export -f and wrap both warmup and
scenario runs in bash -c so the function is visible in the subshell.
Add rule 10: commit experiment code together with final results.json
and .md reports; do not commit logs/raw_outputs/gpu_logs. Also update
the pre-commit checklist accordingly.
Source code was already committed; this adds only the structured final
artifacts from run 20260709-152419_matrix. Raw jsonl/logs/gpu_logs remain
ignored per .gitignore.
- config.env: switch to /data/models/DeepSeek-V4-Flash, USE_DOCKER=1,
CONTEXT_LENGTH=1M, CONCURRENCY_SAMPLES=2, add DATASET_PATH.
- start_sglang_docker.sh: add --ipc host --shm-size 16g for NCCL.
- start_sglang_dp.sh: use --model-path for Docker CLI.
- run_bench.sh: mount dataset into client container and pass --dataset-path
to avoid HF download on offline nodes.
- Document the rule in docs/EXPERIMENT_GUIDE.md section 7
- Update all config.env SCENARIOS to follow the 5x rule
- Migrate dsv4_h200_sglang, dsv4_h200_vllm, dsv4_p800_sglang
from global NUM_PROMPTS to per-scenario num_prompts
- Keep long-context phase2 as exception (low counts documented)
- Add legacy 3-field fallback in run_bench.sh loops
- P800 INT8 cannot sustain >1 concurrent long-context request without OOM;
set all scenarios to concurrency=1.
- Keep max_running=2 because max_running=1 triggers req_to_token_pool
allocation failures in this image.
- Reduce 64k max_context_len to 72000 (65536 + 4096 + pad).
- Make run_bench.sh resilient to container death: detect dead container and
skip remaining scenarios in the group instead of crashing the orchestrator.
- Update README to document the concurrency limitation.
Without chunked prefill the server OOM/crashes (exit 137) during or
immediately after concurrent 64k-context scenarios. Speculative decoding
remains disabled as requested.
These were failed/intermediate smoke-test runs created while debugging the
managed server startup. Keep the repository clean; new runs can be reproduced
with experiments/dsv4_p800_sglang/run_bench.sh.
Add experiments/dsv4_p800_max_context_length/ to find the longest input
context that P800 SGLang INT8 can serve for DeepSeek-V4-Flash-INT8.
- Docker-based server start with dynamic --context-length.
- Single-request, single-output-token probes for each candidate length.
- Records success/failure, server args, and metrics per length.
- Generates results.json and report.md summarizing the max supported length.
- Smoke-tested at 4096 tokens successfully.
