cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution

RUN_ID="dsv4pro-phase2-$(date +%Y%m%d-%H%M%S)"
tmux new-session -d -s dsv4pro-phase2 \
  "RUN_ID=${RUN_ID} bash run_hardware_contention_attribution.sh all \
   2>&1 | tee /data/hzy/${RUN_ID}.log"

all internally calls the Phase 1 entry to start and stop the two-node TP16 service. Do not manually run Phase 1 start or stop in parallel.

Before using GPUs:

bash -n run_hardware_contention_attribution.sh
python3 -m unittest discover -s tests -v
DRY_RUN=1 RUN_ID=dsv4pro-phase2-dryrun \
  bash run_hardware_contention_attribution.sh all

Scope

The default run covers:

Workload	Purpose
`128K -> 1, C=1`	Long Prefill latency
`32K -> 1, C=16`	Concurrent Prefill throughput
`1K -> 1K, C=32`	Ordinary Decode
`1K -> 4K, C=16`	Sustained Decode and KV growth
`128K -> 1K, C=1`	Long-context Decode
`1K -> 1K, C=32` plus `128K -> 1`	Prefill/Decode contention

The service and benchmark implementation remains owned by Phase 1. Phase 2 adds only the monitoring window, case markers and hardware summaries.

Files

File	Purpose
`run_hardware_contention_attribution.sh`	Only shell entry; service, collectors, cases, cleanup
`config.env`	Phase 1 path, node addresses, cases and sampling policy
`hardware_contention_attribution.py`	Markers, manifest, validation and summaries
`tests/test_hardware_contention_attribution.py`	Parser and unit-conversion tests

Output

results/<RUN_ID>/
  manifest.json
  run.log
  markers.csv
  bench/<case-or-suite>/
  service/
    head_server_cmd.txt
    worker_server_cmd.txt
    head_server.log
    worker_server.log
  head/
    gpu_samples.csv
    dcgm_dmon.log
    mpstat.log
    pidstat.log
    sar_net.log
    perf_stat.log
    docker_top.log
    rdma.csv
    static_before.log
    static_after.log
    collector_commands/
  worker/
    ...
  bench_summary.csv
  gpu_summary.csv
  rdma_summary.csv
  case_windows.csv
  case_gpu_summary.csv
  case_rdma_summary.csv
  summary.json
  report.md

Raw collectors are diagnostic evidence. Final performance comparisons must still use a no-profiler benchmark such as Phase 1.