When the data files and video files agree on episode count N but the metadata
lists a different count, rewrite meta/episodes.jsonl, meta/episodes_stats.jsonl
and info.json to N before the v2.1->v3.0 converter runs (which otherwise raises
'Number of episodes is not the same'). Only the safe direction is handled:
trimming metadata that lists MORE episodes than exist. Non-contiguous data,
data/video disagreement, or metadata missing episodes are left untouched.
A robot_type of so100/so101 is treated as wrong when the joint dim isn't a
multiple of 6, or (when names are present) the first 6 joints don't match the
canonical SO set. Such datasets are migrated structurally to v3.0 with joints
left untouched and robot_type relabeled 'unknown', instead of being skipped or
degrees-converted on a false assumption.
Only migrate datasets usable right away as a clean 6-DOF joint stack. SO datasets
whose action/observation.state dim isn't a multiple of 6 (extra bbox/EE columns
appended) are now skipped as out-of-scope, matching the end-effector skip, rather
than being relabeled '_nonstandard' and migrated structurally.
Some datasets store task-space end-effector pose (names like ee_x/ee_roll or
x/y/z) instead of joint angles; the degrees mapping is meaningless there. Detect
via feature names and skip them entirely (no conversion, no upload) rather than
migrating a mislabeled arm.
download_subfolder gains a repo= arg so it can source from any monorepo; the new
extract_dataset.py scoped-downloads one sub-dataset and re-uploads it at the root
of a new standalone dataset repo.
- download_subfolder: fetch only the target sub-dataset subtree instead of
enumerating the whole community_dataset_v3 monorepo tree (fixes apparent hang)
- normalized SO gripper (RANGE_0_100) left in native 0..100 frame, matching
degrees_new datasets, instead of remapping to +/-45deg
- uncalibrated normalized datasets: skip identity value rewrite, keep normalized
units and flag them APPROXIMATE on the dataset card
- remove --allow-uncalibrated flag and its CANON_IS_CALIBRATED side effect
CarolinePascal