# vLLM Adaptive Concurrency Search

`run_adaptive_concurrency.sh` keeps the existing ISL/DSL and TP/DP matrix, but
searches concurrency at runtime instead of testing only low/high endpoints.

## Search rule

For every fixed `(TP, DP, ISL, DSL)`:

```text
C = 1, 2, 4, 8, ... up to SEARCH_MAX_CONCURRENCY
num_prompts = C * NUM_PROMPTS_MULTIPLIER
```

The search stops when Total TPS grows by less than `TPS_MIN_GAIN_PCT` for
`PLATEAU_PATIENCE` consecutive points. Defaults are 2% and two points.

- `saturation_concurrency`: first point in the final low-gain streak
- `best_tps_concurrency`: tested point with the highest Total TPS
- `stop_probe_concurrency`: last point tested to confirm the stop

OOM stops only the current shape. The service is restarted before the next
shape. A non-OOM failure is retried once after a service restart.

Before every measured point, the client sends `C` concurrent warm-up requests.
This keeps lazy kernel compilation and CUDA graph capture out of the measured
TTFT/TPS. `BENCH_WARMUP_MAX_REQUESTS=0` means no cap; use a positive value only
when deliberately limiting warm-up at very high concurrency.

## Dataset requirement

The default remains compatible with the fixed matrix baseline:

```text
--dataset-name random --dataset-path DATASET_PATH
--random-range-ratio 1.0
```

In the installed SGLang client, the upstream default `0.0` samples each length
uniformly from `1` to the requested maximum. This adaptive experiment sets
`1.0` explicitly so every point represents the fixed ISL/DSL shown in the
matrix. Override `RANDOM_RANGE_RATIO` only when a mixed-length workload is
intentional.

`DATASET_PATH` must have at least
`SEARCH_MAX_CONCURRENCY * NUM_PROMPTS_MULTIPLIER` usable two-turn
conversations. The current `runtime/dummy_sharegpt.json` has only two and is
not valid for a real adaptive run.

For setup smoke tests without ShareGPT, explicitly use:

```bash
BENCH_DATASET_NAME=random-ids
```

This sends generated token IDs with `--tokenize-prompt`; it is convenient for
smoke tests but changes the token distribution, so do not mix it with random
ShareGPT-backed baseline results.

## Commands

Print the plan without starting a server:

```bash
DRY_RUN=1 bash run_adaptive_concurrency.sh
```

Run one small search:

```bash
TP_LIST="8" \
ISL_LIST="1024" \
DSL_LIST="128" \
GRID_LIMIT=1 \
SEARCH_MAX_CONCURRENCY=8 \
DATASET_PATH=/path/to/ShareGPT.json \
bash run_adaptive_concurrency.sh
```

Run the full search in tmux:

```bash
tmux new -s vllm-adaptive
DATASET_PATH=/path/to/ShareGPT.json bash run_adaptive_concurrency.sh
```

Tune the stop rule when needed:

```bash
TPS_MIN_GAIN_PCT=3.0 \
PLATEAU_PATIENCE=2 \
SEARCH_MAX_CONCURRENCY=512 \
bash run_adaptive_concurrency.sh
```

## Outputs

Results are written to `adaptive_results/<run_id>/`:

```text
adaptive_points.jsonl
adaptive_points.csv
adaptive_summary.jsonl
adaptive_summary.csv
adaptive_summary.md
run_manifest.json
tp*/raw_outputs/
tp*/metrics/
tp*/logs/
tp*/gpu_logs/
```

Every successful point is accepted only when `completed == num_prompts` and
actual input/output lengths match the requested shape within configured
tolerances. Invalid workloads never participate in saturation decisions.
