Matrix API Tutorial:从 25 个 LeRobot 数据集到训练 batch
目标:只用 Matrix API,从 25 个 LeRobot 数据集做出一份训练清单(tm)。然后在训练机上读出 batch、看 batch、测吞吐。
目的:用户不开网页,也能让 Claude(matrix skill)做完和网页相同的事。
依据:2026-10-10 实跑一遍。下文的 id、条数、耗时都是这次的结果。API 在集群内调用(身份 Developer)。带个人 km 令牌经公网调用 /api/user/me 和快照接口,在本机和集群内的 pod 里都返回本人身份(10-11 实测)。
- 编 dm
- scan
- container
- dc
- 预览并发布 tm
- 装 wheel、读 batch
- 看 batch
- 测吞吐
0. 准备
- 在 keymaker 左侧栏的 API Tokens 页建一个个人令牌,Bound app 填
matrix。不要用服务页上的 New token:那样建出来的是服务令牌,调用会返回 403。 - 设两个环境变量。
- 调
/api/user/me,确认返回的是你自己的名字。
bashexport MATRIX_URL=https://matrix.libra.sudoai.cc
export MATRIX_TOKEN=km.xxxx.xxxx # 只填 km. 开头的这一串,不带 "Authorization: Bearer"
curl -s -H "Authorization: Bearer $MATRIX_TOKEN" "$MATRIX_URL/mx_gen/api/user/me"
在自己电脑上和在集群里自己的 pod 里,用法一样:都走公网地址。不要用集群内地址,那里不认 km 令牌,操作会记成 Developer。
下文每条 curl 都带这两个 header。删除、隐藏、覆盖这类接口,skill 不调用。
1. 编数据清单(dm)
25 个数据集各一组。每组最多 200 条,不足 200 条时全取。
bashcurl -s -X POST "$MATRIX_URL/mx_gen/api/compile/submit" -H "Authorization: Bearer $MATRIX_TOKEN" -H "Content-Type: application/json" -d '{
"manifest_name": "tutorial-lr25-200each",
"seed": 42,
"groups": [
{"group_name": "agibot_maniformer_g2a_0724", "datasets": ["agibot_maniformer_g2a_0724"], "target_count": 200},
{"group_name": "internrobotics_agibot_a2d", "datasets": ["internrobotics_agibot_a2d"], "target_count": 200},
{"group_name": "neoteaiembodied_ur5e", "datasets": ["neoteaiembodied_ur5e"], "target_count": 200},
...
]
}'
# 返回 job_id;每 5 秒查一次,直到 job.status 是 completed
curl -s -H "Authorization: Bearer $MATRIX_TOKEN" "$MATRIX_URL/mx_gen/api/compile/status/job_compile_..."
结果:manifest_3b7eae568211fb05,4856 条,编译 6.25 s。24 组各 200 条,agibot_maniformer_g2a_0724 只有 56 条。
依据:job_compile_7af48f6a9aee 的返回;逐条读成员,组名等于数据集名 4856/4856。
按名字选数据集,一律用 datasets: ["名字"]。
dataset_name 也按完整名字精确匹配,和 datasets 相同。
依据:src/eggy_matrix_gen/catalog/predicates.py:50-52(dataset_name = %s)。
2. scan:看每个数据集有什么
bashcurl -s -X POST "$MATRIX_URL/mx_gen/api/container/scan" -H "Authorization: Bearer $MATRIX_TOKEN" -H "Content-Type: application/json" \
-d '{"manifest_id": "manifest_3b7eae568211fb05", "dataset_name": "agibot_g2a_imitationlearning", "n_episodes": 3}'
返回里要用 4 个字段:sources(源 key → 宽度或类型)、slots(每一维的名字)、data_hz、entity_ids。
结果:25/25 成功,每次中位 617 ms。帧率 5–30 Hz。只有 7 个数据集同时有相机内参 K 和外参 c2w。
25 个数据集的 scan 结果
| 数据集 | data_hz | 帧数 | rgb 路数 | 深度路数 | K + c2w |
|---|---|---|---|---|---|
agibot_maniformer_g2a_0724 | 30 | 6313–9202 | 5 | 0 | 有 |
internrobotics_agibot_a2d | 30 | 492–701 | 3 | 0 | — |
neoteaiembodied_ur5e | 30 | 733–5341 | 4 | 0 | — |
neoteaiembodied_arx5_single | 30 | 550–2557 | 4 | 0 | — |
rdt_ft_agilex_cobot_magic | 25 | 234–517 | 3 | 3 | — |
robocoin_tianqing_a2d_01 | 30 | 468–495 | 8 | 0 | — |
roboset_franka | 5 | 42–42 | 4 | 0 | — |
agibot_g2a_richinteraction | 30 | 1838–2484 | 6 | 1 | 有 |
robocoin_galaxea_r1_lite | 30 | 206–493 | 3 | 0 | — |
agibot_g2a_reinforcementlearning | 30 | 1244–2416 | 6 | 1 | 有 |
agibot_g2a_imitationlearning | 30 | 1496–4702 | 6 | 1 | 有 |
neoteaiembodied_aloha_dual | 30 | 946–1022 | 7 | 0 | — |
abc_130k_xdof_stereo | 30 | 4272–6328 | 4 | 0 | — |
robocoin_galbot_g1 | 30 | 1190–1261 | 3 | 0 | — |
neoteaiembodied_flexiv_rizon4s_single | 30 | 2432–4382 | 4 | 0 | — |
neoteaiembodied_ur5 | 30 | 2023–3703 | 4 | 0 | — |
robocoin_agilex_cobot_magic | 30 | 453–1101 | 3 | 0 | — |
robocoin_agilex_split_aloha | 30 | 351–465 | 3 | 0 | — |
bridge_v2_widowx_250 | 5 | 23–49 | 1 | 0 | — |
robomind_franka_dual_4rgb | 15 | 105–280 | 4 | 0 | — |
neoteaiembodied_arx5_dual | 30 | 585–3943 | 7 | 0 | — |
robomind_franka_dual_6rgb | 15 | 213–263 | 6 | 6 | 有 |
droid_franka | 15 | 177–431 | 3 | 0 | — |
genrobot_das_finger | 30 | 863–3612 | 8 | 0 | 有 |
realman_new_0812_rs_02 | 30 | 2270–3373 | 3 | 1 | 有 |
3. container:训练样本的形状
3 路 rgb(224×224)、1 路深度、qpos 32 维、action 32 维 × 8 步、base_hz 15。
json{
"name": "tutorial-lr25",
"body": {
"base_hz": 15,
"rgbs": {
"cam0": {
"shape": [
1,
224,
224
],
"norm_group": 0
},
"cam1": {
"shape": [
1,
224,
224
],
"norm_group": 0
},
"cam2": {
"shape": [
1,
224,
224
],
"norm_group": 0
}
},
"depths": {
"dep0": {
"shape": [
1,
224,
224
]
}
},
"segs": {},
"state": {
"T": 1,
"slots": {
"qpos": {
"width": 32,
"default": 0
}
}
},
"action": {
"chunk_size": 8,
"slots": {
"action": {
"width": 32,
"default": 0
}
}
},
"meta": {
"fields": {},
"default": "N/A"
}
}
}
bashcurl -s -X POST "$MATRIX_URL/mx_gen/api/container/container-specs" -H "Authorization: Bearer $MATRIX_TOKEN" -H "Content-Type: application/json" -d @container.json
结果:一份草稿 cd_…。草稿只有你能看见,可以先在上面建 dc、取样本调试。
调好后发布:POST /mx_gen/api/container/container-specs/cd_…/publish,得到 cs_59f9a82579d0efa4。不用调试时,在 JSON 顶层加 "publish": true,一步发布。
发布后的 id 只由内容决定。已发布的 container 谁也不能改,要改就衍生一份新草稿。
4. dataset config(dc):每个数据集怎么连到 container
每个数据集一个 dc。顺序是 validate → sample → 保存。例子是 agibot,带了 K 和 c2w:
json{
"dataset": "agibot_g2a_imitationlearning",
"data_hz": 30,
"rgbs": [
{
"src": "views/head.back/rgb",
"slot": "cam0",
"resize": "pad",
"K": "views/head.back/K",
"c2w": "views/head.back/c2w"
},
"… cam1、cam2 同样写法 …"
],
"depths": [
{
"src": "views/head/depth",
"slot": "dep0",
"resize": "pad"
}
],
"segs": [],
"state": [
{
"src": "observation.state.qpos.left_arm",
"src_idx": [
0,
1,
2,
3,
4,
5,
6
],
"slot": "qpos",
"dst_idx": [
3,
4,
5,
6,
7,
8,
9
],
"t": [
0
]
},
"… 其余关节同样写法 …"
],
"action": [
{
"src": "action.qpos.left_arm",
"src_idx": [
0,
1,
2,
3,
4,
5,
6
],
"slot": "action",
"dst_idx": [
3,
4,
5,
6,
7,
8,
9
],
"t": [
0,
1,
2,
3,
4,
5,
6,
7
]
},
"…"
]
}
bash# 1. 校验:sources 原样用 scan 的返回
curl -s -X POST "$MATRIX_URL/mx_gen/api/container/validate" -H "Authorization: Bearer $MATRIX_TOKEN" -H "Content-Type: application/json" \
-d '{"container": <container body>, "config": <dc body>, "sources": <scan.sources>}'
# 2. 取一个真实样本,看 state / action 的数值和 mask
curl -s -X POST "$MATRIX_URL/mx_gen/api/container/sample" -H "Authorization: Bearer $MATRIX_TOKEN" -H "Content-Type: application/json" \
-d '{"container": {...}, "config": {...}, "manifest_id": "manifest_3b7eae568211fb05", "dataset_name": "agibot_g2a_imitationlearning", "entity_id": "<scan.entity_ids[0]>"}'
# 3. 保存
curl -s -X POST "$MATRIX_URL/mx_gen/api/container/dataset-configs" -H "Authorization: Bearer $MATRIX_TOKEN" -H "Content-Type: application/json" \
-d '{"name": "tutorial-lr25-agibot_g2a_imitationlearning", "container_id": "cs_59f9a82579d0efa4", "body": {...}}'
结果:validate 25/25,sample 25/25(中位 662 ms),保存 25/25,名字是 tutorial-lr25-<数据集>。
7 个数据集补上了 K / c2w:agibot_maniformer_g2a_0724, agibot_g2a_richinteraction, agibot_g2a_reinforcementlearning, agibot_g2a_imitationlearning, robomind_franka_dual_6rgb, genrobot_das_finger, realman_new_0812_rs_02。
rgb 的 K / c2w 只在 scan 的 sources 里同时有 <视角>/K 和 <视角>/c2w 时才写。没有就不写这两个字段,不能写空字符串。
5. 预览并发布训练清单(tm)
bash# 模板:train 和 val 各取一份
curl -s -H "Authorization: Bearer $MATRIX_TOKEN" "$MATRIX_URL/mx_gen/api/compile/recipe-templates/container_v3?split=train"
curl -s -H "Authorization: Bearer $MATRIX_TOKEN" "$MATRIX_URL/mx_gen/api/compile/recipe-templates/container_v3?split=val"
# 先校验一个池的配方(不编译)
curl -s -X POST "$MATRIX_URL/mx_gen/api/compile/validate-recipe" -H "Authorization: Bearer $MATRIX_TOKEN" -H "Content-Type: application/json" \
-d '{"split": "train", "recipe": {...含 groups}, "runtime": {...}}'
# 预览:publish=false,等 job.status = preview_ready
curl -s -X POST "$MATRIX_URL/mx_gen/api/compile/training-preview" -H "Authorization: Bearer $MATRIX_TOKEN" -H "Content-Type: application/json" -d @tm.json
# 用户看过预览后发布这次预览
curl -s -X POST "$MATRIX_URL/mx_gen/api/compile/training-publish" -H "Authorization: Bearer $MATRIX_TOKEN" -H "Content-Type: application/json" \
-d '{"preview_job_id": "job_training_...", "train_manifest_id": "tm_d10ab8c22b017de2", "val_manifest_id": "tm_4282f074ed57cc19"}'
tm.json(节选)
json{
"data_manifest_id": "manifest_3b7eae568211fb05",
"training_manifest_names": {
"train": "tutorial-lr25",
"val": "tutorial-lr25-val"
},
"split_plan": {
"seed": 42,
"sources": [
{
"dataset": "agibot_maniformer_g2a_0724",
"train": {
"kind": "rest"
},
"val": {
"kind": "count",
"value": 8
}
},
"… 25 个数据集各一行 …"
]
},
"pools": {
"train": {
"container_id": "cs_59f9a82579d0efa4",
"recipe": {
"stream": "…模板…",
"batching": "…模板…",
"image": "…模板…",
"groups": [
{
"group_id": "agibot_maniformer_g2a_0724",
"weight": 0.04,
"dataset_config_id": "dc_90d3eeb4feacbf00",
"cut_idle_head": false,
"cut_idle_head_threshold": 0.5,
"cut_idle_tail": false,
"cut_idle_tail_threshold": 0.5
},
"… 25 组,weight 都是 0.04 …"
]
},
"runtime": {
"num_workers": 12,
"prefetch_factor": 2,
"pin_memory": true,
"persistent_workers": true
},
"group_datasets": {
"agibot_maniformer_g2a_0724": [
"agibot_maniformer_g2a_0724"
],
"…": "…"
}
},
"val": "同 train,recipe 和 runtime 取 split=val 的模板"
},
"publish": false
}
结果:预览 39.18 s。train tm_d10ab8c22b017de2(4656 条),val tm_4282f074ed57cc19(200 条 = 25 × 8)。发布调用 3.7 s。
group 的 weight 加起来必须等于 1。第一次每组写 1.0(合计 25),validate-recipe 在提交前就报了错。
val 池用 split=val 的模板。val 不能带光度增强,也不能设 drop_last。
split_plan 要列出 dm 里的每个数据集。漏一个会被拒。
6. 在训练机上装 wheel,读 batch
训练机要有 torch 和 OSS 凭据(AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY)。
bashpython3 -m venv --system-site-packages ~/mx && . ~/mx/bin/activate # 环境里已有 torch
curl -s -H "Authorization: Bearer $MATRIX_TOKEN" "$MATRIX_URL/mx_gen/api/compile/manifests/tm_d10ab8c22b017de2/snapshot" > snapshot.json
# install_guide.download_command:用 OSS 凭据下载 wheel
python3 -c "import json,os; os.system(json.load(open('snapshot.json'))['install_guide']['download_command'])"
pip install --force-reinstall --no-deps ./eggy_matrix_gen-0.1.0-py3-none-any.whl
pip install "pydantic>=2.0" "av>=12.0" "opencv-python-headless<4.11" "omegaconf>=2.3" "pillow>=10.0" # install_guide.deps_command
pythonfrom eggy_matrix_gen.compile_.manifest_io import read_training_manifest
from eggy_matrix_gen.dataloader import ExecutionOverrides, MatrixDataloader
tm = read_training_manifest("s3://sudoai-sh-oss/matrix/manifests/tm_d10ab8c22b017de2.json").manifest
loader = MatrixDataloader(tm, rank=0, world_size=1, is_train=True,
execution=ExecutionOverrides(batch_size=16, num_workers=16))
batch = next(iter(loader))
for k, v in batch.items():
print(k, tuple(v.shape) if hasattr(v, "shape") else type(v).__name__)
一个 batch 的形状(batch_size 16)
bashrgb_mask [16, 3, 1]
K [16, 3, 1, 3, 3]
c2w [16, 3, 1, 4, 4]
cam_mask [16, 3, 1]
depths [16, 1, 1, 1, 224, 224]
depth_mask [16, 1, 1]
segs [16, 0, 1, 1, 1, 1]
seg_mask [16, 0, 1]
state [16, 1, 32]
state_mask [16, 1, 32]
action [16, 8, 32]
action_mask [16, 8, 32]
ray_mask [16, 3, 1]
t0 [16]
image_t [16]
结果:40 个 batch,640 个样本,失败 0,25 组都读到。
cam_mask = 1 的正好是第 4 步补了 K / c2w 的 7 组。数据里没有 action 的 3 组,action_mask = 0:bridge_v2_widowx_250, genrobot_das_finger, internrobotics_agibot_a2d。
依据:H200 1 卡台架,venv(--system-site-packages)里只装 wheel,不用仓库代码。
第三条命令不能省。pip 带 --no-deps,不装依赖。10-10 实测缺 pydantic、av、cv2。
缺包时,loader 取第一个样本就报 ImportError,并写明要装的包。
每组的 mask 填充率(40 个 batch)
| 组 | 样本 | rgb | cam | depth | state | action | action NaN(mask=1) |
|---|---|---|---|---|---|---|---|
abc_130k_xdof_stereo | 29 | 1.0 | 0.0 | 0.0 | 0.438 | 0.438 | 0 |
agibot_g2a_imitationlearning | 30 | 1.0 | 1.0 | 1.0 | 0.75 | 0.75 | 0 |
agibot_g2a_reinforcementlearning | 24 | 1.0 | 1.0 | 1.0 | 0.75 | 0.75 | 0 |
agibot_g2a_richinteraction | 24 | 1.0 | 1.0 | 1.0 | 0.75 | 0.75 | 0 |
agibot_maniformer_g2a_0724 | 27 | 1.0 | 1.0 | 0.0 | 0.75 | 0.75 | 0 |
bridge_v2_widowx_250 | 25 | 0.333 | 0.0 | 0.0 | 0.031 | 0.0 | 0 |
droid_franka | 26 | 1.0 | 0.0 | 0.0 | 0.25 | 0.243 | 0 |
genrobot_das_finger | 23 | 1.0 | 1.0 | 0.0 | 0.062 | 0.0 | 0 |
internrobotics_agibot_a2d | 22 | 1.0 | 0.0 | 0.0 | 0.844 | 0.0 | 0 |
neoteaiembodied_aloha_dual | 20 | 1.0 | 0.0 | 0.0 | 0.438 | 0.438 | 0 |
neoteaiembodied_arx5_dual | 33 | 1.0 | 0.0 | 0.0 | 0.5 | 0.5 | 0 |
neoteaiembodied_arx5_single | 22 | 1.0 | 0.0 | 0.0 | 0.25 | 0.25 | 0 |
neoteaiembodied_flexiv_rizon4s_single | 20 | 1.0 | 0.0 | 0.0 | 0.25 | 0.281 | 0 |
neoteaiembodied_ur5 | 23 | 1.0 | 0.0 | 0.0 | 0.219 | 0.213 | 0 |
neoteaiembodied_ur5e | 25 | 1.0 | 0.0 | 0.0 | 0.219 | 0.219 | 0 |
rdt_ft_agilex_cobot_magic | 24 | 1.0 | 0.0 | 1.0 | 0.438 | 0.428 | 0 |
realman_new_0812_rs_02 | 22 | 1.0 | 1.0 | 0.0 | 0.531 | 0.531 | 0 |
robocoin_agilex_cobot_magic | 24 | 1.0 | 0.0 | 0.0 | 0.438 | 0.438 | 0 |
robocoin_agilex_split_aloha | 25 | 1.0 | 0.0 | 0.0 | 0.438 | 0.438 | 0 |
robocoin_galaxea_r1_lite | 31 | 1.0 | 0.0 | 0.0 | 0.438 | 0.438 | 0 |
robocoin_galbot_g1 | 26 | 1.0 | 0.0 | 0.0 | 0.844 | 0.844 | 0 |
robocoin_tianqing_a2d_01 | 29 | 1.0 | 0.0 | 0.0 | 0.625 | 0.625 | 0 |
robomind_franka_dual_4rgb | 30 | 1.0 | 0.0 | 0.0 | 0.562 | 0.539 | 0 |
robomind_franka_dual_6rgb | 37 | 1.0 | 1.0 | 1.0 | 0.5 | 0.5 | 0 |
roboset_franka | 19 | 1.0 | 0.0 | 0.0 | 0.25 | 0.242 | 32 |
state / action 的填充率低于 1,是因为 32 维里只有本体实际的维数有值。
7. 看 batch(batch viewer)
skill 自带转储和查看工具,在 ~/.claude/skills/matrix/tools/batch_viewer/。
bashT=~/.claude/skills/matrix/tools
python3 $T/batch_viewer/dump.py tm_d10ab8c22b017de2 --batches 50 --batch-size 8 --workers 16 --out ./batches
python3 $T/batch_viewer/server.py --root ./batches --port 8765
# 远程训练机:本地执行 ssh -L 8765:127.0.0.1:8765 <训练机>,再打开 http://127.0.0.1:8765/?tm=tm_d10ab8c22b017de2
结果:50 个 batch 转储速度 41.9 样本/s。页面按 batch 显示相机、深度、state、action 和每个数组的指纹。
8. 测吞吐
bashT=~/.claude/skills/matrix/tools
for W in 8 16 24; do
W=$W B=100 EPOCH=$((W / 8 + 2)) python3 $T/throughput_v3.py tm_d10ab8c22b017de2 # 每次换一个没读过的 epoch
done
| worker 数 | 启动(建 loader + 第一个 batch,s) | 稳定吞吐(样本/s) |
|---|---|---|
| 8 | 9.44 | 50.01 |
| 16 | 9.49 | 93.59 |
| 24 | 12.98 | 158.18 |
依据:throughput_v3.py 输出,batch_size 32(清单默认值),每次 100 个 batch,冷数据(每次换新的 epoch)。台架是 H200 1 卡,cgroup 限 32 核,所以不代表 8 卡整机。
已知问题
roboset_franka:上游数据每条最后 2 帧 action 是 NaN,进了样本且 mask=1(40 个 batch 里 32 个值)。- 18 个数据集的 K 和 c2w 不全(多数两样都没有;
abc_130k_xdof_stereo只有 K,robocoin_agilex_split_aloha只有 c2w),cam_mask = 0。