admin/lerobot
1
0
Fork 0
mirror of https://github.com/huggingface/lerobot.git synced 2026-05-15 16:49:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(annotate): drop dataset-level `tools` parquet column

Browse Source 
PR 2 used to write a top-level ``tools`` column on every parquet shard
holding the JSON schema for the ``say`` tool, broadcast identically
across every row. That extends PR 1's schema for no real information
gain — the schema is a fixed code constant, parquet's RLE/dict encoding
collapses it on disk anyway, and HF/TRL chat-template consumers can
just import the constant directly.

PR 2 should fill in PR 1's existing schema, not add to it. So:

- ``writer.py``: stop emitting the ``tools`` column. Strip any legacy
  ``tools`` column from older shards on rerun so the schema converges to
  v3.1. ``SAY_TOOL_SCHEMA`` stays as a public constant (now joined by
  ``DEFAULT_TOOLS = [SAY_TOOL_SCHEMA]``); chat-template policies and the
  visualizer import them directly.
- ``test_writer.py``: replace the "tools column present" assertion with
  one that explicitly checks the column is absent, plus a new test
  asserting the constant's shape.
- ``test_pipeline_recipe_render.py``: drop the tools-column read; assert
  it's not present in the rewritten parquet.
- ``annotation_pipeline.mdx``: update the writer description to note the
  parquet stays small and the schema lives as a code constant.

If multi-tool-set support ever becomes real (datasets with different
tool inventories), the right home is ``meta/info.json["tools"]`` —
adding it later is non-breaking; ripping out a parquet column already
shipped is not.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pepijn
2026-04-30 15:54:37 +02:00
parent 0f6e3230df
commit b71e10da6b
4 changed files with 63 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/annotation_pipeline.mdx
+7 -2
View File
@@ -20,8 +20,13 @@ rewrites the data shards in place:
| speech tool-call atom (`style=null`, `say`) | `language_events` | Module 2 |
| `vqa` (user / assistant pair) | `language_events` | Module 3 |
The writer also adds a dataset-level `tools` column carrying the JSON schema
for the `say` tool call, and drops the legacy `subtask_index` column.
The writer drops the legacy `subtask_index` column. It does **not** add a
`tools` column to the parquet — the `say` tool's JSON schema is fixed and
lives as a code constant (`SAY_TOOL_SCHEMA` / `DEFAULT_TOOLS` in
`lerobot.annotations.steerable_pipeline.writer`), so the parquet stays
small and PR 2 doesn't extend PR 1's schema. Chat-template consumers
import the constant directly (e.g.
`apply_chat_template(messages, tools=DEFAULT_TOOLS)`).
## How to run it locally or on SLURM