mirror of
https://github.com/huggingface/lerobot.git
synced 2026-05-22 12:09:42 +00:00
refactor(annotate): drop dataset-level `tools` parquet column
PR 2 used to write a top-level ``tools`` column on every parquet shard holding the JSON schema for the ``say`` tool, broadcast identically across every row. That extends PR 1's schema for no real information gain — the schema is a fixed code constant, parquet's RLE/dict encoding collapses it on disk anyway, and HF/TRL chat-template consumers can just import the constant directly. PR 2 should fill in PR 1's existing schema, not add to it. So: - ``writer.py``: stop emitting the ``tools`` column. Strip any legacy ``tools`` column from older shards on rerun so the schema converges to v3.1. ``SAY_TOOL_SCHEMA`` stays as a public constant (now joined by ``DEFAULT_TOOLS = [SAY_TOOL_SCHEMA]``); chat-template policies and the visualizer import them directly. - ``test_writer.py``: replace the "tools column present" assertion with one that explicitly checks the column is absent, plus a new test asserting the constant's shape. - ``test_pipeline_recipe_render.py``: drop the tools-column read; assert it's not present in the rewritten parquet. - ``annotation_pipeline.mdx``: update the writer description to note the parquet stays small and the schema lives as a code constant. If multi-tool-set support ever becomes real (datasets with different tool inventories), the right home is ``meta/info.json["tools"]`` — adding it later is non-breaking; ripping out a parquet column already shipped is not. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -17,7 +17,6 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pyarrow.parquet as pq
|
||||
@@ -130,6 +129,7 @@ def test_pr1_canonical_recipe_renders_nonempty_from_pipeline_output(
|
||||
say = speech_rows[0]["tool_calls"][0]
|
||||
assert say["function"]["name"] == "say"
|
||||
assert isinstance(say["function"]["arguments"]["text"], str)
|
||||
# Tools column carries the say schema
|
||||
tools = json.loads(table.column("tools").to_pylist()[0])
|
||||
assert tools and tools[0]["function"]["name"] == "say"
|
||||
# PR 2 no longer writes a ``tools`` column — the say schema lives as a
|
||||
# constant (``SAY_TOOL_SCHEMA``) so PR 1's row struct is the single
|
||||
# source of truth for the v3.1 schema.
|
||||
assert "tools" not in table.column_names
|
||||
|
||||
Reference in New Issue
Block a user