Matrix API Tutorial:从 25 个 LeRobot 数据集到训练 batch

目标:只用 Matrix API,从 25 个 LeRobot 数据集做出一份训练清单(tm)。然后在训练机上读出 batch、看 batch、测吞吐。

目的:用户不开网页,也能让 Claude(matrix skill)做完和网页相同的事。

依据:2026-10-10 实跑一遍。下文的 id、条数、耗时都是这次的结果。API 在集群内调用(身份 Developer)。带个人 km 令牌经公网调用 /api/user/me 和快照接口,在本机和集群内的 pod 里都返回本人身份(10-11 实测)。

  1. 编 dm
  2. scan
  3. container
  4. dc
  5. 预览并发布 tm
  6. 装 wheel、读 batch
  7. 看 batch
  8. 测吞吐

0. 准备

  1. 在 keymaker 左侧栏的 API Tokens 页建一个个人令牌,Bound app 填 matrix。不要用服务页上的 New token:那样建出来的是服务令牌,调用会返回 403。
  2. 设两个环境变量。
  3. 调 /api/user/me,确认返回的是你自己的名字。
bashexport MATRIX_URL=https://matrix.libra.sudoai.cc
export MATRIX_TOKEN=km.xxxx.xxxx          # 只填 km. 开头的这一串,不带 "Authorization: Bearer"
curl -s -H "Authorization: Bearer $MATRIX_TOKEN" "$MATRIX_URL/mx_gen/api/user/me"

在自己电脑上和在集群里自己的 pod 里,用法一样:都走公网地址。不要用集群内地址,那里不认 km 令牌,操作会记成 Developer。

下文每条 curl 都带这两个 header。删除、隐藏、覆盖这类接口,skill 不调用。

1. 编数据清单(dm)

25 个数据集各一组。每组最多 200 条,不足 200 条时全取。

bashcurl -s -X POST "$MATRIX_URL/mx_gen/api/compile/submit" -H "Authorization: Bearer $MATRIX_TOKEN" -H "Content-Type: application/json" -d '{
  "manifest_name": "tutorial-lr25-200each",
  "seed": 42,
  "groups": [
    {"group_name": "agibot_maniformer_g2a_0724", "datasets": ["agibot_maniformer_g2a_0724"], "target_count": 200},
    {"group_name": "internrobotics_agibot_a2d", "datasets": ["internrobotics_agibot_a2d"], "target_count": 200},
    {"group_name": "neoteaiembodied_ur5e", "datasets": ["neoteaiembodied_ur5e"], "target_count": 200},
    ...
  ]
}'
# 返回 job_id;每 5 秒查一次,直到 job.status 是 completed
curl -s -H "Authorization: Bearer $MATRIX_TOKEN" "$MATRIX_URL/mx_gen/api/compile/status/job_compile_..."

结果:manifest_3b7eae568211fb05,4856 条,编译 6.25 s。24 组各 200 条,agibot_maniformer_g2a_0724 只有 56 条。

依据:job_compile_7af48f6a9aee 的返回;逐条读成员,组名等于数据集名 4856/4856。

按名字选数据集,一律用 datasets: ["名字"]。

dataset_name 也按完整名字精确匹配,和 datasets 相同。

依据:src/eggy_matrix_gen/catalog/predicates.py:50-52(dataset_name = %s)。

2. scan:看每个数据集有什么

bashcurl -s -X POST "$MATRIX_URL/mx_gen/api/container/scan" -H "Authorization: Bearer $MATRIX_TOKEN" -H "Content-Type: application/json" \
  -d '{"manifest_id": "manifest_3b7eae568211fb05", "dataset_name": "agibot_g2a_imitationlearning", "n_episodes": 3}'

返回里要用 4 个字段:sources(源 key → 宽度或类型)、slots(每一维的名字)、data_hz、entity_ids。

结果:25/25 成功,每次中位 617 ms。帧率 5–30 Hz。只有 7 个数据集同时有相机内参 K 和外参 c2w。

25 个数据集的 scan 结果
数据集data_hz帧数rgb 路数深度路数K + c2w
agibot_maniformer_g2a_0724306313–920250有
internrobotics_agibot_a2d30492–70130—
neoteaiembodied_ur5e30733–534140—
neoteaiembodied_arx5_single30550–255740—
rdt_ft_agilex_cobot_magic25234–51733—
robocoin_tianqing_a2d_0130468–49580—
roboset_franka542–4240—
agibot_g2a_richinteraction301838–248461有
robocoin_galaxea_r1_lite30206–49330—
agibot_g2a_reinforcementlearning301244–241661有
agibot_g2a_imitationlearning301496–470261有
neoteaiembodied_aloha_dual30946–102270—
abc_130k_xdof_stereo304272–632840—
robocoin_galbot_g1301190–126130—
neoteaiembodied_flexiv_rizon4s_single302432–438240—
neoteaiembodied_ur5302023–370340—
robocoin_agilex_cobot_magic30453–110130—
robocoin_agilex_split_aloha30351–46530—
bridge_v2_widowx_250523–4910—
robomind_franka_dual_4rgb15105–28040—
neoteaiembodied_arx5_dual30585–394370—
robomind_franka_dual_6rgb15213–26366有
droid_franka15177–43130—
genrobot_das_finger30863–361280有
realman_new_0812_rs_02302270–337331有

3. container:训练样本的形状

3 路 rgb(224×224)、1 路深度、qpos 32 维、action 32 维 × 8 步、base_hz 15。

json{
  "name": "tutorial-lr25",
  "body": {
    "base_hz": 15,
    "rgbs": {
      "cam0": {
        "shape": [
          1,
          224,
          224
        ],
        "norm_group": 0
      },
      "cam1": {
        "shape": [
          1,
          224,
          224
        ],
        "norm_group": 0
      },
      "cam2": {
        "shape": [
          1,
          224,
          224
        ],
        "norm_group": 0
      }
    },
    "depths": {
      "dep0": {
        "shape": [
          1,
          224,
          224
        ]
      }
    },
    "segs": {},
    "state": {
      "T": 1,
      "slots": {
        "qpos": {
          "width": 32,
          "default": 0
        }
      }
    },
    "action": {
      "chunk_size": 8,
      "slots": {
        "action": {
          "width": 32,
          "default": 0
        }
      }
    },
    "meta": {
      "fields": {},
      "default": "N/A"
    }
  }
}
bashcurl -s -X POST "$MATRIX_URL/mx_gen/api/container/container-specs" -H "Authorization: Bearer $MATRIX_TOKEN" -H "Content-Type: application/json" -d @container.json

结果:一份草稿 cd_…。草稿只有你能看见,可以先在上面建 dc、取样本调试。

调好后发布:POST /mx_gen/api/container/container-specs/cd_…/publish,得到 cs_59f9a82579d0efa4。不用调试时,在 JSON 顶层加 "publish": true,一步发布。

发布后的 id 只由内容决定。已发布的 container 谁也不能改,要改就衍生一份新草稿。

4. dataset config(dc):每个数据集怎么连到 container

每个数据集一个 dc。顺序是 validate → sample → 保存。例子是 agibot,带了 K 和 c2w:

json{
  "dataset": "agibot_g2a_imitationlearning",
  "data_hz": 30,
  "rgbs": [
    {
      "src": "views/head.back/rgb",
      "slot": "cam0",
      "resize": "pad",
      "K": "views/head.back/K",
      "c2w": "views/head.back/c2w"
    },
    "… cam1、cam2 同样写法 …"
  ],
  "depths": [
    {
      "src": "views/head/depth",
      "slot": "dep0",
      "resize": "pad"
    }
  ],
  "segs": [],
  "state": [
    {
      "src": "observation.state.qpos.left_arm",
      "src_idx": [
        0,
        1,
        2,
        3,
        4,
        5,
        6
      ],
      "slot": "qpos",
      "dst_idx": [
        3,
        4,
        5,
        6,
        7,
        8,
        9
      ],
      "t": [
        0
      ]
    },
    "… 其余关节同样写法 …"
  ],
  "action": [
    {
      "src": "action.qpos.left_arm",
      "src_idx": [
        0,
        1,
        2,
        3,
        4,
        5,
        6
      ],
      "slot": "action",
      "dst_idx": [
        3,
        4,
        5,
        6,
        7,
        8,
        9
      ],
      "t": [
        0,
        1,
        2,
        3,
        4,
        5,
        6,
        7
      ]
    },
    "…"
  ]
}
bash# 1. 校验:sources 原样用 scan 的返回
curl -s -X POST "$MATRIX_URL/mx_gen/api/container/validate" -H "Authorization: Bearer $MATRIX_TOKEN" -H "Content-Type: application/json" \
  -d '{"container": <container body>, "config": <dc body>, "sources": <scan.sources>}'
# 2. 取一个真实样本,看 state / action 的数值和 mask
curl -s -X POST "$MATRIX_URL/mx_gen/api/container/sample" -H "Authorization: Bearer $MATRIX_TOKEN" -H "Content-Type: application/json" \
  -d '{"container": {...}, "config": {...}, "manifest_id": "manifest_3b7eae568211fb05", "dataset_name": "agibot_g2a_imitationlearning", "entity_id": "<scan.entity_ids[0]>"}'
# 3. 保存
curl -s -X POST "$MATRIX_URL/mx_gen/api/container/dataset-configs" -H "Authorization: Bearer $MATRIX_TOKEN" -H "Content-Type: application/json" \
  -d '{"name": "tutorial-lr25-agibot_g2a_imitationlearning", "container_id": "cs_59f9a82579d0efa4", "body": {...}}'

结果:validate 25/25,sample 25/25(中位 662 ms),保存 25/25,名字是 tutorial-lr25-<数据集>。

7 个数据集补上了 K / c2w:agibot_maniformer_g2a_0724, agibot_g2a_richinteraction, agibot_g2a_reinforcementlearning, agibot_g2a_imitationlearning, robomind_franka_dual_6rgb, genrobot_das_finger, realman_new_0812_rs_02。

rgb 的 K / c2w 只在 scan 的 sources 里同时有 <视角>/K 和 <视角>/c2w 时才写。没有就不写这两个字段,不能写空字符串。

5. 预览并发布训练清单(tm)

bash# 模板:train 和 val 各取一份
curl -s -H "Authorization: Bearer $MATRIX_TOKEN" "$MATRIX_URL/mx_gen/api/compile/recipe-templates/container_v3?split=train"
curl -s -H "Authorization: Bearer $MATRIX_TOKEN" "$MATRIX_URL/mx_gen/api/compile/recipe-templates/container_v3?split=val"
# 先校验一个池的配方(不编译)
curl -s -X POST "$MATRIX_URL/mx_gen/api/compile/validate-recipe" -H "Authorization: Bearer $MATRIX_TOKEN" -H "Content-Type: application/json" \
  -d '{"split": "train", "recipe": {...含 groups}, "runtime": {...}}'
# 预览:publish=false,等 job.status = preview_ready
curl -s -X POST "$MATRIX_URL/mx_gen/api/compile/training-preview" -H "Authorization: Bearer $MATRIX_TOKEN" -H "Content-Type: application/json" -d @tm.json
# 用户看过预览后发布这次预览
curl -s -X POST "$MATRIX_URL/mx_gen/api/compile/training-publish" -H "Authorization: Bearer $MATRIX_TOKEN" -H "Content-Type: application/json" \
  -d '{"preview_job_id": "job_training_...", "train_manifest_id": "tm_d10ab8c22b017de2", "val_manifest_id": "tm_4282f074ed57cc19"}'
tm.json(节选)
json{
  "data_manifest_id": "manifest_3b7eae568211fb05",
  "training_manifest_names": {
    "train": "tutorial-lr25",
    "val": "tutorial-lr25-val"
  },
  "split_plan": {
    "seed": 42,
    "sources": [
      {
        "dataset": "agibot_maniformer_g2a_0724",
        "train": {
          "kind": "rest"
        },
        "val": {
          "kind": "count",
          "value": 8
        }
      },
      "… 25 个数据集各一行 …"
    ]
  },
  "pools": {
    "train": {
      "container_id": "cs_59f9a82579d0efa4",
      "recipe": {
        "stream": "…模板…",
        "batching": "…模板…",
        "image": "…模板…",
        "groups": [
          {
            "group_id": "agibot_maniformer_g2a_0724",
            "weight": 0.04,
            "dataset_config_id": "dc_90d3eeb4feacbf00",
            "cut_idle_head": false,
            "cut_idle_head_threshold": 0.5,
            "cut_idle_tail": false,
            "cut_idle_tail_threshold": 0.5
          },
          "… 25 组,weight 都是 0.04 …"
        ]
      },
      "runtime": {
        "num_workers": 12,
        "prefetch_factor": 2,
        "pin_memory": true,
        "persistent_workers": true
      },
      "group_datasets": {
        "agibot_maniformer_g2a_0724": [
          "agibot_maniformer_g2a_0724"
        ],
        "…": "…"
      }
    },
    "val": "同 train,recipe 和 runtime 取 split=val 的模板"
  },
  "publish": false
}

结果:预览 39.18 s。train tm_d10ab8c22b017de2(4656 条),val tm_4282f074ed57cc19(200 条 = 25 × 8)。发布调用 3.7 s。

group 的 weight 加起来必须等于 1。第一次每组写 1.0(合计 25),validate-recipe 在提交前就报了错。

val 池用 split=val 的模板。val 不能带光度增强,也不能设 drop_last。

split_plan 要列出 dm 里的每个数据集。漏一个会被拒。

6. 在训练机上装 wheel,读 batch

训练机要有 torch 和 OSS 凭据(AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY)。

bashpython3 -m venv --system-site-packages ~/mx && . ~/mx/bin/activate     # 环境里已有 torch
curl -s -H "Authorization: Bearer $MATRIX_TOKEN" "$MATRIX_URL/mx_gen/api/compile/manifests/tm_d10ab8c22b017de2/snapshot" > snapshot.json
# install_guide.download_command:用 OSS 凭据下载 wheel
python3 -c "import json,os; os.system(json.load(open('snapshot.json'))['install_guide']['download_command'])"
pip install --force-reinstall --no-deps ./eggy_matrix_gen-0.1.0-py3-none-any.whl
pip install "pydantic>=2.0" "av>=12.0" "opencv-python-headless<4.11" "omegaconf>=2.3" "pillow>=10.0"   # install_guide.deps_command
pythonfrom eggy_matrix_gen.compile_.manifest_io import read_training_manifest
from eggy_matrix_gen.dataloader import ExecutionOverrides, MatrixDataloader

tm = read_training_manifest("s3://sudoai-sh-oss/matrix/manifests/tm_d10ab8c22b017de2.json").manifest
loader = MatrixDataloader(tm, rank=0, world_size=1, is_train=True,
                          execution=ExecutionOverrides(batch_size=16, num_workers=16))
batch = next(iter(loader))
for k, v in batch.items():
    print(k, tuple(v.shape) if hasattr(v, "shape") else type(v).__name__)
一个 batch 的形状(batch_size 16)
bashrgb_mask     [16, 3, 1]
K            [16, 3, 1, 3, 3]
c2w          [16, 3, 1, 4, 4]
cam_mask     [16, 3, 1]
depths       [16, 1, 1, 1, 224, 224]
depth_mask   [16, 1, 1]
segs         [16, 0, 1, 1, 1, 1]
seg_mask     [16, 0, 1]
state        [16, 1, 32]
state_mask   [16, 1, 32]
action       [16, 8, 32]
action_mask  [16, 8, 32]
ray_mask     [16, 3, 1]
t0           [16]
image_t      [16]

结果:40 个 batch,640 个样本,失败 0,25 组都读到。

cam_mask = 1 的正好是第 4 步补了 K / c2w 的 7 组。数据里没有 action 的 3 组,action_mask = 0:bridge_v2_widowx_250, genrobot_das_finger, internrobotics_agibot_a2d。

依据:H200 1 卡台架,venv(--system-site-packages)里只装 wheel,不用仓库代码。

第三条命令不能省。pip 带 --no-deps,不装依赖。10-10 实测缺 pydantic、av、cv2。

缺包时,loader 取第一个样本就报 ImportError,并写明要装的包。

每组的 mask 填充率(40 个 batch)
组样本rgbcamdepthstateactionaction NaN(mask=1)
abc_130k_xdof_stereo291.00.00.00.4380.4380
agibot_g2a_imitationlearning301.01.01.00.750.750
agibot_g2a_reinforcementlearning241.01.01.00.750.750
agibot_g2a_richinteraction241.01.01.00.750.750
agibot_maniformer_g2a_0724271.01.00.00.750.750
bridge_v2_widowx_250250.3330.00.00.0310.00
droid_franka261.00.00.00.250.2430
genrobot_das_finger231.01.00.00.0620.00
internrobotics_agibot_a2d221.00.00.00.8440.00
neoteaiembodied_aloha_dual201.00.00.00.4380.4380
neoteaiembodied_arx5_dual331.00.00.00.50.50
neoteaiembodied_arx5_single221.00.00.00.250.250
neoteaiembodied_flexiv_rizon4s_single201.00.00.00.250.2810
neoteaiembodied_ur5231.00.00.00.2190.2130
neoteaiembodied_ur5e251.00.00.00.2190.2190
rdt_ft_agilex_cobot_magic241.00.01.00.4380.4280
realman_new_0812_rs_02221.01.00.00.5310.5310
robocoin_agilex_cobot_magic241.00.00.00.4380.4380
robocoin_agilex_split_aloha251.00.00.00.4380.4380
robocoin_galaxea_r1_lite311.00.00.00.4380.4380
robocoin_galbot_g1261.00.00.00.8440.8440
robocoin_tianqing_a2d_01291.00.00.00.6250.6250
robomind_franka_dual_4rgb301.00.00.00.5620.5390
robomind_franka_dual_6rgb371.01.01.00.50.50
roboset_franka191.00.00.00.250.24232

state / action 的填充率低于 1,是因为 32 维里只有本体实际的维数有值。

7. 看 batch(batch viewer)

skill 自带转储和查看工具,在 ~/.claude/skills/matrix/tools/batch_viewer/。

bashT=~/.claude/skills/matrix/tools
python3 $T/batch_viewer/dump.py tm_d10ab8c22b017de2 --batches 50 --batch-size 8 --workers 16 --out ./batches
python3 $T/batch_viewer/server.py --root ./batches --port 8765
# 远程训练机:本地执行 ssh -L 8765:127.0.0.1:8765 <训练机>,再打开 http://127.0.0.1:8765/?tm=tm_d10ab8c22b017de2

结果:50 个 batch 转储速度 41.9 样本/s。页面按 batch 显示相机、深度、state、action 和每个数组的指纹。

batch viewer 显示 tm_d10ab8c22b017de2 第 0 个 batch:rdt_ft_agilex_cobot_magic 的 3 路相机、深度和 state

8. 测吞吐

bashT=~/.claude/skills/matrix/tools
for W in 8 16 24; do
  W=$W B=100 EPOCH=$((W / 8 + 2)) python3 $T/throughput_v3.py tm_d10ab8c22b017de2     # 每次换一个没读过的 epoch
done
worker 数启动(建 loader + 第一个 batch,s)稳定吞吐(样本/s)
89.4450.01
169.4993.59
2412.98158.18

依据:throughput_v3.py 输出,batch_size 32(清单默认值),每次 100 个 batch,冷数据(每次换新的 epoch)。台架是 H200 1 卡,cgroup 限 32 核,所以不代表 8 卡整机。

已知问题