96 worker threads racing transformers 5.x lazy imports on the FIRST
_get_tokenizer call raised ImportError and degraded that whole first
batch to the char approximation (the old single-threaded path never
raced). First load now holds a threading.Lock; the transformers
'>model_max_length' logging is silenced inside truncation (counting a
2M-token doc before trimming it is the point), and the per-sample
degradation print becomes a one-shot warning.
Co-Authored-By: Claude <noreply@anthropic.com>
