Report datasets present in the manifest but absent from the destination repo,
grouped by their migration action (ERROR/skipped/...), to explain the src-vs-dst
count gap. Also de-duplicate manifest rows by root (resumed runs append).
Reads the run manifest(s), selects datasets that errored during migration and/or
were labelled SO but aren't a real 6-DOF SO arm, intersects with what's actually
present in the destination repo, and deletes their folders. Dry-run by default.
If the data files and any camera's video files (or two cameras) don't have the
same number of episodes, skip the dataset up front instead of letting the
converter raise 'All cams dont have same number of episodes' mid-run.
When the data files and video files agree on episode count N but the metadata
lists a different count, rewrite meta/episodes.jsonl, meta/episodes_stats.jsonl
and info.json to N before the v2.1->v3.0 converter runs (which otherwise raises
'Number of episodes is not the same'). Only the safe direction is handled:
trimming metadata that lists MORE episodes than exist. Non-contiguous data,
data/video disagreement, or metadata missing episodes are left untouched.
When the leading joint names match the canonical SO set, treat the dataset as a
genuine SO arm and convert those joints to degrees even if extra columns follow
(bbox, etc.), passing the trailing non-joint columns through untouched. Encoding
detection now looks only at the SO slice so appended columns can't skew it. Only
when no leading 6-DOF SO block can be substantiated is the dataset relabeled
'unknown' and migrated structurally.
A robot_type of so100/so101 is treated as wrong when the joint dim isn't a
multiple of 6, or (when names are present) the first 6 joints don't match the
canonical SO set. Such datasets are migrated structurally to v3.0 with joints
left untouched and robot_type relabeled 'unknown', instead of being skipped or
degrees-converted on a false assumption.
Only migrate datasets usable right away as a clean 6-DOF joint stack. SO datasets
whose action/observation.state dim isn't a multiple of 6 (extra bbox/EE columns
appended) are now skipped as out-of-scope, matching the end-effector skip, rather
than being relabeled '_nonstandard' and migrated structurally.
Preserve the original robot_type lineage (e.g. so100 -> so100_nonstandard) for
arms whose joint dim isn't a multiple of 6, rather than erasing it to 'unknown'.
Idempotent: won't re-append the suffix.
Some datasets store task-space end-effector pose (names like ee_x/ee_roll or
x/y/z) instead of joint angles; the degrees mapping is meaningless there. Detect
via feature names and skip them entirely (no conversion, no upload) rather than
migrating a mislabeled arm.
download_subfolder gains a repo= arg so it can source from any monorepo; the new
extract_dataset.py scoped-downloads one sub-dataset and re-uploads it at the root
of a new standalone dataset repo.
The manifest robot_type distribution has 'bi_so100_follower', which the trailing
underscore in 'bi_so_' missed. Drop it to 'bi_so' to catch all bimanual SO
variants (bi_so_follower, bi_so100_follower, ...) with no false positives.
Broaden SO_PREFIXES to (so100, so101, so_, bi_so_) so bimanual two-arm datasets
(robot_type 'bi_so_follower', 12-dim) are recognized as SO and get the degrees
conversion, instead of being skipped as non_so. so_ boundary avoids matching
stray names like 'sofa'.
Split classify() into reusable pieces: is_so_robot_type() for the robot_type
name test, and encoding_from_bounds() as the single source of truth for the
degrees_old/degrees_new/normalized/radians decision from per-joint min/max
(layout-agnostic, so v2.1 episodes_stats and v3.0 stats.json both feed it).
is_so is decided purely from the robot_type string, so datasets labeled
so100/so101 whose joint dim isn't a multiple of 6 carry a provably wrong label.
Rewrite meta/info.json robot_type to 'unknown' in that case so the v3.0 output
isn't misidentified as an SO arm.
Datasets whose action/state dim is not a multiple of 6 (e.g. 7-dim with an
appended EE pose, or 10-dim) are not a plain stack of SO arms, so the degrees
mapping doesn't apply. Migrate them structurally instead of aborting the whole
dataset with a ValueError.
- download_subfolder: fetch only the target sub-dataset subtree instead of
enumerating the whole community_dataset_v3 monorepo tree (fixes apparent hang)
- normalized SO gripper (RANGE_0_100) left in native 0..100 frame, matching
degrees_new datasets, instead of remapping to +/-45deg
- uncalibrated normalized datasets: skip identity value rewrite, keep normalized
units and flag them APPROXIMATE on the dataset card
- remove --allow-uncalibrated flag and its CANON_IS_CALIBRATED side effect
CarolinePascal