deps(annotations): drop hard vllm dependency to unblock CI torch/torchcodec resolution

Fast Pytest 'dataset' tier failed collecting tests/datasets/test_video_
decoder_cache.py with 'Could not load libtorchcodec ... undefined symbol:
torch_dtype_float4_e2m1fn_x2' — a torch/torchcodec ABI mismatch.

Root cause: the annotations extra's vllm hard-pins an older torch
(via xformers/xgrammar -> torch 2.8). uv resolves a SINGLE unified lock
across all extras, so vllm capped torch to 2.8 for every tier —
including dataset, whose torchcodec 0.11.1 needs torch 2.11. The
result was torch 2.8 + torchcodec 0.11.1 installed together -> ABI break.
(main has no vllm, so it resolves torch 2.11 + torchcodec 0.11.1 cleanly.)

Fix: remove vllm from the annotations extra. It is not needed by
the shipped workflow — examples/annotations/run_hf_job.py gets vllm from
the vllm/vllm-openai image and talks to it over the OpenAI-compatible
API (--vlm.backend=openai), and vlm_client._make_vllm_client imports vllm
lazily. For the in-process --vlm.backend=vllm path, install vllm
separately (the ImportError now says so).

After the fix uv resolves torch 2.11.0 + torchcodec 0.11.1 (matching
main); uv lock --check is clean. The annotations extra still provides
datasets / transformers / openai.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

pyproject.toml

@@ -231,7 +231,17 @@ annotations = [
     "lerobot[dataset]",
     "lerobot[transformers-dep]",
     "openai>=1.40,<2.0",
-    "vllm>=0.6.0,<1.0.0; sys_platform == 'linux'",
+    # NOTE: ``vllm`` is intentionally NOT a hard dependency here. vLLM
+    # hard-pins an older torch (via xformers/xgrammar), and because uv
+    # resolves a single unified lock across all extras, including it would
+    # cap ``torch`` for every other extra too (e.g. forcing torch 2.8 while
+    # ``torchcodec`` in the ``dataset`` extra needs torch 2.11 -> ABI break
+    # in CI). vLLM is also not needed by the shipped workflow: the HF Jobs
+    # launcher (``examples/annotations/run_hf_job.py``) gets it from the
+    # ``vllm/vllm-openai`` image and talks to it over the OpenAI-compatible
+    # API (``--vlm.backend=openai``), and ``vlm_client._make_vllm_client``
+    # imports vllm lazily with an actionable error. To use the in-process
+    # ``--vlm.backend=vllm`` locally, ``pip install vllm`` separately.
 ]
 
 # Development

src/lerobot/annotations/steerable_pipeline/vlm_client.py

@@ -216,7 +216,11 @@ def _make_vllm_client(config: VlmConfig) -> VlmClient:
         from vllm import LLM, SamplingParams  # type: ignore[import-not-found]
     except ImportError as exc:
         raise ImportError(
-            "vllm is required for backend='vllm'. Install with `pip install lerobot[annotations]`."
+            "vllm is required for backend='vllm'. Install it separately with "
+            "`pip install vllm` (it is not a hard dependency of the "
+            "``annotations`` extra because it pins an older torch). The HF "
+            "Jobs launcher uses the vllm/vllm-openai image + backend='openai' "
+            "instead."
         ) from exc
     # Workaround for cuDNN 9.x + torch 2.8 conv3d regression that surfaces
     # as CUDNN_STATUS_NOT_INITIALIZED in Qwen-VL vision-tower patch