pre-op Tutorial

eggy_matrix_gen.userop · 2026-10-09 · 三个例子都是 10-09 线上实测写的 op,代码和输出原样

目标:你自己写一个 pre-op,测试,然后上传到算子库。

目的:不改 matrix 代码,也能改变训练样本:选哪一帧、加哪些列、改哪些数值。

这 6 步都在训练 pod 里做,不需要 dc,也不需要 container。上传以后,在 WebUI 的 dc 里选用 op,本页不讲这部分。

  1. pre-op 在哪里运行
  2. 三个例子
  3. 准备
  4. 第 1 步 装 wheel
  5. 三种 op 的接口
  6. 加键、改键
  7. 第 2 步 看数据
  8. 第 3–6 步 写、测、上传

pre-op 在哪里运行

loader 取每个样本时,按下面的顺序运行。你写的是中间三段。

三个例子

三个例子都按本页的步骤写成,上传后在 WebUI 里用进 dc,发布成 tm,再用 MatrixDataloader 读回核对。

例子数据集op(种类)做什么训练读回核对(10-09)
1 一列 ×2334B(fd)norm_action_x2@1(step_op)加列 norm_action = action × 2tm_e82b2e56df7efc69:32 个样本,失败 0
2 两个 LeRobot 数据集agibot、galaxeaagibot_qpos_x2@1、galaxea_tcp_x2@1(step_op)加列 norm_qpos(宽 8)、norm_tcp(宽 9),都是原值 × 2co-train tm_4b00b2c4cb2bbc82:agibot 26/26、galaxea 22/22、334B 16/16
3 三种 op 各一个334B(fd)episode_op、step_sampler、step_op标帧号;只在帧号 % 7 == 3 的帧选 t;加列 action_re = action + 1tm_e254c93945b4f282:32 个样本,失败 0

依据:pod 上 check_norm_tm.py、check_cotrain_tm.py、check_tm.py 的输出。例子 3 上传时的名字是 e2e_mark_ep、e2e_mod_sampler、e2e_t_step。

准备

第 1 步 装 wheel

wheel 在私有 bucket 里。先用 OSS 凭据下载,再用 pip 安装。进 pod 以后直接粘贴:

python3 -c "import boto3,os;from botocore.config import Config;boto3.client('s3',endpoint_url=os.environ.get('AWS_ENDPOINT_URL_S3','https://oss-cn-shanghai.aliyuncs.com'),config=Config(s3={'addressing_style':'virtual'})).download_file('sudoai-sh-oss','matrix/dist/eggy_matrix_gen-0.1.0-py3-none-any.whl','eggy_matrix_gen-0.1.0-py3-none-any.whl')"
pip install --force-reinstall --no-deps ./eggy_matrix_gen-0.1.0-py3-none-any.whl

检查安装:

python3 -c "from eggy_matrix_gen.userop import EpisodeOp, StepSampler, StepOp, upload; print('ok')"

三种 op 的接口

基类你写的方法返回matrix 检查什么
EpisodeOpprocess(self, ep, ctx)Episode返回类型
StepSamplersample(self, ep, ctx)int,一个帧号 t返回类型,且 0 ≤ t < n_frames
StepOpprocess(self, ep, t, ctx)Step返回类型

注意:Step 填了 arrays,就整个替换 ep 的数值列。要保留原来的列,写 arrays={**ep.arrays, "新列": 数组}。三个例子都这样写。

加键、改键

op 改列只有一种方式:返回一份新的列字典。episode_op 返回新 Episode,step_op 返回 Step。下面 6 种写法都在本地跑过,结果见本节末尾。

要做什么写法WebUI 扫描后dc 里要做什么
1 加键Step(t=t, arrays={**ep.arrays, "speed": x})
episode_op:ep.with_arrays(speed=x)
新键标「pre-op 加的」把新键拖到槽上
2 改值,宽度不变ep.with_arrays(action=ep.arrays["action"] * 2)键名不变,不标「pre-op 加的」不用改。原来读 action 的路由读到新值
3 改值,宽度变了ep.with_arrays(qpos=ep.arrays["qpos"][:, :8])键名不变,宽度变成 8检查路由的 src_idx。超出新宽度时,校验报「src_idx … 超出 qpos 的维数 8」
4 改名arrays = dict(ep.arrays)
arrays["joint"] = arrays.pop("qpos")
Step(t=t, arrays=arrays)
joint 标「pre-op 加的」,qpos 不见了删掉读 qpos 的路由,把 joint 拖到槽上。不删时,校验报「数据集里没有 key qpos」
5 删键replace(ep, arrays={k: v for k, v in ep.arrays.items() if k != "qpos"})qpos 不见了删掉读 qpos 的路由
6 图像改名images = dict(ep.images)
images["head"] = images.pop("views/head/rgb")
Step(t=t, images=images)
head 是图像键图像路由的 src 改选 head

「WebUI 扫描后」一列依据本地 scan() 的输出和 matrix_adapter.py:292(只有新键名才标「pre-op 加的」)。两句校验报错依据 container/validate.py:115、123,没有在线上测。

规则:

6 种写法的完整代码和输出(本地实测,提交 42afe16)
"""keys_demo.py:加键、改键的六种写法,各跑一次,打印结果。python3 keys_demo.py"""
from dataclasses import replace

import numpy as np
from eggy_matrix_gen.container.episode import scan
from eggy_matrix_gen.userop import Ctx, Episode, EpisodeOp, Step, StepOp, run_local

n = 10
ep = Episode(n_frames=n, entity_id="fake",
             arrays={"action": np.ones((n, 8), np.float32), "qpos": np.ones((n, 36), np.float32)},
             images={"views/head/rgb": lambda t: np.full((4, 4, 3), t, np.uint8)})


class AddKey(StepOp):
    """1 加键:新列 speed,其他列原样带上。"""
    def process(self, ep, t, ctx):
        speed = np.linalg.norm(ep.arrays["action"], axis=1, keepdims=True).astype(np.float32)
        return Step(t=t, arrays={**ep.arrays, "speed": speed})


class Overwrite(EpisodeOp):
    """2 同名覆盖:action 改成 action × 2,宽度不变。"""
    def process(self, ep, ctx):
        return ep.with_arrays(action=ep.arrays["action"] * 2)


class Narrow(EpisodeOp):
    """3 同名覆盖并改宽度:qpos 36 维只留前 8 维。"""
    def process(self, ep, ctx):
        return ep.with_arrays(qpos=ep.arrays["qpos"][:, :8])


class Rename(StepOp):
    """4 改名:qpos 改名为 joint。"""
    def process(self, ep, t, ctx):
        arrays = dict(ep.arrays)
        arrays["joint"] = arrays.pop("qpos")
        return Step(t=t, arrays=arrays)


class Drop(EpisodeOp):
    """5 删键:去掉 qpos。"""
    def process(self, ep, ctx):
        return replace(ep, arrays={k: v for k, v in ep.arrays.items() if k != "qpos"})


class RenameImage(StepOp):
    """6 图像改名:views/head/rgb 改名为 head。"""
    def process(self, ep, t, ctx):
        images = dict(ep.images)
        images["head"] = images.pop("views/head/rgb")
        return Step(t=t, images=images)


def keys(run):
    return {k: v for k, v in scan(run.episode).keys.items()}


for op in (AddKey(), RenameImage()):
    print(type(op).__name__, keys(run_local(ep, Ctx(0, 0, 0), step_ops=[op])))
for op in (Overwrite(), Narrow(), Drop()):
    print(type(op).__name__, keys(run_local(ep, Ctx(0, 0, 0), episode_ops=[op])))
run = run_local(ep, Ctx(0, 0, 0), step_ops=[Rename()])
print("Rename", keys(run))
print("Overwrite 后 action[0,0] =", run_local(ep, Ctx(0, 0, 0), episode_ops=[Overwrite()]).episode.arrays["action"][0, 0],
      ";原 ep 的 action[0,0] =", ep.arrays["action"][0, 0])
print("RenameImage 后读 head 第 3 帧:", run_local(ep, Ctx(0, 0, 0), step_ops=[RenameImage()]).episode.images["head"](3)[0, 0, 0])


class Clash(StepOp):
    def process(self, ep, t, ctx):
        return Step(t=t, arrays={**ep.arrays, "views/head/rgb": np.zeros((ep.n_frames, 1), np.float32)})


class Flat(StepOp):
    def process(self, ep, t, ctx):
        return Step(t=t, arrays={**ep.arrays, "flat": np.zeros(ep.n_frames, np.float32)})


for op in (Clash(), Flat()):
    try:
        run_local(ep, Ctx(0, 0, 0), step_ops=[op])
        print(type(op).__name__, "没有报错")
    except Exception as e:
        print(type(op).__name__, "报错:", str(e)[:160])

输出:

AddKey {'action': 8, 'qpos': 36, 'speed': 1, 'views/head/rgb': 'image'}
RenameImage {'action': 8, 'qpos': 36, 'head': 'image'}
Overwrite {'action': 8, 'qpos': 36, 'views/head/rgb': 'image'}
Narrow {'action': 8, 'qpos': 8, 'views/head/rgb': 'image'}
Drop {'action': 8, 'views/head/rgb': 'image'}
Rename {'action': 8, 'joint': 36, 'views/head/rgb': 'image'}
Overwrite 后 action[0,0] = 2.0 ;原 ep 的 action[0,0] = 1.0
RenameImage 后读 head 第 3 帧: 3
Clash 报错: keys appear in more than one group: ['views/head/rgb']
Flat 报错: array 'flat' must be [N_frames=10, dim], got (10,)

第 2 步 看数据

op 的输入是一条 episode。写 op 之前,先用 load_dm 读几条真 episode,看清有哪些列、每列的形状。同一个意思的列,在不同数据集里名字不同。例子 2 就是先看列名,才知道 galaxea 没有叫 tcp 的列。

dm 是一张成员表,每行是一条 episode。一个 dm 可以包含多个数据集,fd / rd 和 LeRobot 的行可以混在一起。三个例子都读 manifest_8ab8c94622d4d724。

fd:334B

"""look.py:读一条 episode,打印它的内容。python3 look.py"""
import logging; logging.disable(logging.WARNING)
from eggy_matrix_gen.userop import load_dm

dm = load_dm("manifest_8ab8c94622d4d724")
print(dm.datasets)                                    # {数据集: 条数}

ep = next(dm.sample(1, dataset="334Broa008objfps16k12", seed=0))
print(ep.entity_id, ep.n_frames)
for k, v in ep.arrays.items():                        # 数值列:每列 [n_frames, dim]
    print(f"  {k:<40} {v.shape} {v.dtype}")
print("图像", sorted(ep.images))                       # 相机:调用 ep.images[键](帧号) 才读帧
print("文本", ep.texts)
print("meta 的键", sorted(ep.meta))                    # episode 信息
print("member 的键", sorted(ep.member))                # dm 成员表里这一行

输出的一部分(10-09 实测):

{'334Broa008objfps16k12': 910186, 'agibot_g1_swiftpicker': 158325, 'galaxea_r1_lite': 22459}
bath_55536.nfAjdERxUliH6qKIrII3Yg.NLSByiqQfGPVFXLLFfRKKQ.1 121
  timestep                                 (121, 1) float32
  tcp_pose                                 (121, 16) float32
  action                                   (121, 8) float32
  qpos                                     (121, 36) float32
  right_arm.target_qpos                    (121, 7) float32
  right_gripper.target_qpos                (121, 1) float32
  success.as_float                         (121, 1) float32
  views/chest.left/K                       (121, 9) float32
  views/chest.left/c2w                     (121, 16) float32
  ...                                      (一共 39 列)
图像 ['views/chest.left/rgb', 'views/chest.right/rgb', 'views/rhand.center/rgb', 'views/rhand.left/rgb', 'views/rhand.right/rgb']
文本 {'task_name': 'pick', 'embodiment_name': 'beta1.1_20260314_single_right', 'dataset_name': '334Broa008objfps16k12', 'family': 'fd', ...}
meta 的键 ['data_type', 'demeter_info', 'fd_version', 'iid_2_uuid', 'object_info', 'robot_info', 'task_info', 'world_info', ...]
member 的键 ['action_dim', 'control_freq_hz', 'dataset_name', 'embodiment_name', 'entity_id', 'family', 'is_success', 'layout', 'n_timesteps', 'qpos_dim', ...]

输出有删节,用 ... 表示。

LeRobot:agibot 和 galaxea

"""cols.py:两个 LeRobot 数据集各读一条,只列 qpos / tcp / pose / action 相关的列。"""
import logging; logging.disable(logging.WARNING)
from eggy_matrix_gen.userop import load_dm
dm = load_dm("manifest_8ab8c94622d4d724")
for ds in ("agibot_g1_swiftpicker", "galaxea_r1_lite"):
    ep = next(dm.sample(1, dataset=ds, seed=0))
    print("==", ds, ep.entity_id[:50], "n_frames", ep.n_frames, "data_hz", ep.data_hz)
    for k, a in ep.arrays.items():
        if any(w in k for w in ("qpos", "tcp", "eef", "pose")) or k.startswith("action"):
            print(f"  {k:55s} {a.shape} {a.dtype}")
    print("  images:", list(ep.images)[:6])

输出(10-09 实测):

== agibot_g1_swiftpicker agibot_g1_swiftpicker_ep_127998 n_frames 1616 data_hz 30.0
  observation.state.qpos.head                             (1616, 2) float32
  observation.state.qpos.torso                            (1616, 2) float32
  observation.state.qpos.left_arm                         (1616, 7) float32
  observation.state.qpos.right_arm                        (1616, 7) float32
  observation.state.qpos.left_gripper                     (1616, 1) float32
  observation.state.qpos.right_gripper                    (1616, 1) float32
  observation.state.pose.left_arm                         (1616, 9) float32
  observation.state.pose.right_arm                        (1616, 9) float32
  observation.state.pose.base                             (1616, 9) float32
  action.twist.base                                       (1616, 2) float32
  observation.state.pose.left_gripper                     (1616, 9) float32
  observation.state.pose.right_gripper                    (1616, 9) float32
  images: ['views/head/rgb', 'views/head.left/rgb', 'views/rear.left/rgb', 'views/head.right/rgb', 'views/rear.right/rgb', 'views/lhand.center/rgb']
== galaxea_r1_lite galaxea_r1_lite_ep_012576 n_frames 929 data_hz 15.0
  observation.state.qpos.torso                            (929, 3) float32
  observation.state.qpos.left_arm                         (929, 6) float32
  observation.state.qpos.right_arm                        (929, 6) float32
  observation.state.qpos.left_gripper                     (929, 1) float32
  observation.state.qpos.right_gripper                    (929, 1) float32
  observation.state.pose.base                             (929, 9) float32
  observation.state.pose.left_arm                         (929, 9) float32
  observation.state.pose.right_arm                        (929, 9) float32
  action.qpos.left_arm                                    (929, 6) float32
  action.qpos.right_arm                                   (929, 6) float32
  action.qpos.left_gripper                                (929, 1) float32
  action.qpos.right_gripper                               (929, 1) float32
  action.twist.base                                       (929, 6) float32
  action.twist.torso                                      (929, 6) float32
  observation.state.pose.left_gripper                     (929, 9) float32
  observation.state.pose.right_gripper                    (929, 9) float32
  action.pose.left_arm                                    (929, 9) float32
  action.pose.left_gripper                                (929, 9) float32
  action.pose.right_arm                                   (929, 9) float32
  action.pose.right_gripper                               (929, 9) float32
  images: ['views/head.left/rgb', 'views/head.right/rgb', 'views/lhand.center/rgb', 'views/rhand.center/rgb']

从这份输出能读出:

dm 的读取方法

用法返回说明
load_dm(dm_id)Dm读 dm 的头部。实测 0.3 s
dm.datasets{数据集: 条数}第一次较慢(实测 4.7 s),之后有缓存
len(dm)int成员总数
dm.entity_ids(数据集)[entity_id]按成员表顺序。不给数据集时返回全部
dm[entity_id]Episode按 id 读一条。不在 dm 里时报 KeyError
dm.sample(n, dataset, seed)Episode 迭代器不放回随机取。循环到哪条才读哪条
dm.member(行号){列: 值}成员表的一行,不读 episode。行号是整张表里的位置
dm.episode(行号)Episode按整张表的行号读

Episode 里有什么

字段类型说明
entity_idstr成员行的 entity_id
n_framesint帧数 N
data_hzfloat帧率
arrays{键: [N, dim] float32}数值列。op 主要读写这里
images{键: 帧号 → 像素}调用时才读帧。depths、segs 相同
texts{键: str}episode 级文本,例如 task_name
steps{键: [N 个值]}每帧的非数值列
meta{键: 值}fd / rd:整份 metadata.json,去掉轨迹。LeRobot:meta/ 下的文件,第一次访问某个键时才读;annotations 每集一个文件
member{键: 值}dm 成员表里这一行
常用信息怎么取(10-08 在 334B、galaxea、agibot 上实测)
信息两类通用fd / rdLeRobot
帧数ep.n_framesmeta["world_info"]["n_timesteps"]—
帧率ep.data_hz、ep.member["control_freq_hz"]1 / meta["world_info"]["timestep_interval"]meta["info"]["fps"]
任务ep.texts["task_name"]meta["task_info"]["task_name"]全表 meta["tasks"]
分段指令—没有meta["annotations"]["level0"] / ["level1"]:这一条自己的文件 meta/annotations/episode_NNNNNN.json(NNNNNN = 这一条的 episode_index)。每段的 start_frame / end_frame 是这一条的帧号,和 t、ep.arrays 的行号相同。level0 的 end_frame 等于帧数(不含);level1 及以下各数据集不统一,取帧前先截到 ep.n_frames - 1。没有标注的条是 None,先判断
是否成功—meta["task_info"]["success"];每帧 ep.steps["success"]annotations 每段的 is_success
机器人ep.member["embodiment_name"]meta["robot_info"]["robot_name"]meta["info"]["device_type"]
相机内外参ep.arrays["views/{相机}/K"]、["views/{相机}/c2w"]同左同左(galaxea 没有 K)
关节和维数ep.member["layout"]、action_dim、qpos_dim—meta["urdf"]["joint_bindings"]

第 3–6 步 写、测、上传

规则:一个 op 一个文件。上传的是这个类所在的整个文件。文件里不能 import 同目录的其他文件,因为训练时只拿到这一个文件。测试脚本可以 import op 文件,测试脚本不上传。

每个例子都按同样的四步做:写 op 文件 → 造数据单测 → 真 dm 试跑 → 上传。点下面的按钮切换例子。

需求:给 334B 加一列 norm_action = action × 2,原来的列不变。之后在 dc 里把 norm_action 接到 container 的 action 槽。

work/
  norm_action_op.py   class NormActionX2(StepOp)   加 norm_action 列
  test_norm.py        造数据单测
  try_norm.py         真 dm 试跑 + 测耗时
  upload_norm.py      上传

写 op:norm_action_op.py

import numpy as np
from eggy_matrix_gen.userop import Step, StepOp


class NormActionX2(StepOp):
    """加一列 norm_action = src 列 × scale(默认 action × 2)。原来的列不变。"""

    def __init__(self, src: str = "action", scale: float = 2.0):
        self.src = src
        self.scale = scale

    def process(self, ep, t, ctx):
        if self.src not in ep.arrays:
            raise KeyError(f"episode 没有列 {self.src!r},有的是 {sorted(ep.arrays)}")
        return Step(t=t, arrays={**ep.arrays, "norm_action": ep.arrays[self.src] * self.scale})
  • step_op 不改 t:原样返回 t=t。
  • 缺列时报错,并列出有的列。用在别的数据集上时,报错能直接看出原因。
  • src 和 scale 是参数,在 WebUI 里能改。

造数据单测:test_norm.py

"""单测:造一条 episode,不读真数据。"""
import numpy as np
from eggy_matrix_gen.userop import Ctx, Episode, run_local
from norm_action_op import NormActionX2

ep = Episode(n_frames=10, arrays={"action": np.arange(20, dtype=np.float32).reshape(10, 2)}, entity_id="fake")
run = run_local(ep, Ctx(0, 0, 0), step_ops=[NormActionX2()])
assert "norm_action" in run.episode.arrays
np.testing.assert_array_equal(run.episode.arrays["norm_action"], ep.arrays["action"] * 2)
np.testing.assert_array_equal(run.episode.arrays["action"], ep.arrays["action"])      # 原列不变
assert run.episode.arrays["norm_action"].dtype == np.float32
run3 = run_local(ep, Ctx(0, 0, 0), step_ops=[NormActionX2(scale=3.0)])
np.testing.assert_array_equal(run3.episode.arrays["norm_action"], ep.arrays["action"] * 3)
try:
    run_local(ep, Ctx(0, 0, 0), step_ops=[NormActionX2(src="nope")])
    raise SystemExit("缺列应该报错")
except Exception as e:
    assert "nope" in str(e), e
print("单测通过:norm_action = action × 2,原列不变,缺列报错")

输出:

单测通过:norm_action = action × 2,原列不变,缺列报错

真 dm 试跑:try_norm.py

"""用真 dm 自检:334B 取 3 条 episode,跑 step op,再测耗时。"""
import logging; logging.disable(logging.WARNING)
import numpy as np
from eggy_matrix_gen.userop import Ctx, load_dm, profile, run_local
from norm_action_op import NormActionX2

DM, DATASET = "manifest_8ab8c94622d4d724", "334Broa008objfps16k12"
dm = load_dm(DM)
for i, ep in enumerate(dm.sample(3, dataset=DATASET, seed=0)):
    run = run_local(ep, Ctx(0, 0, i), step_ops=[NormActionX2()])
    a, na = ep.arrays["action"], run.episode.arrays["norm_action"]
    assert na.shape == a.shape and np.array_equal(na, a * 2)
    print(f"{ep.entity_id[:40]} n_frames={ep.n_frames} action{a.shape} norm_action{na.shape} t={run.t} "
          f"action[t,:3]={np.round(a[run.t, :3], 4).tolist()} norm_action[t,:3]={np.round(na[run.t, :3], 4).tolist()}")
prof = profile(dm, dataset=DATASET, n=10, step_ops=[NormActionX2()])
print("profile", prof)

输出(10-09 实测):

bath_55536.nfAjdERxUliH6qKIrII3Yg.NLSByi n_frames=121 action(121, 8) norm_action(121, 8) t=47 action[t,:3]=[-0.9948999881744385, 0.5871000289916992, 0.020099999383091927] norm_action[t,:3]=[-1.98989999294281, 1.1741000413894653, 0.04019999876618385]
bath_65841.YIWB4RouVReUG30uINi-QQ.FUH9fa n_frames=93 action(93, 8) norm_action(93, 8) t=90 action[t,:3]=[0.04270000010728836, -0.01810000091791153, 0.039400000125169754] norm_action[t,:3]=[0.08529999852180481, -0.03620000183582306, 0.07890000194311142]
bath_77641.oEF7cjZrXlaFj0RrA6T_vA.bOx4Ek n_frames=152 action(152, 8) norm_action(152, 8) t=135 action[t,:3]=[0.24060000479221344, -0.7423999905586243, -0.2671000063419342] norm_action[t,:3]=[0.4810999929904938, -1.4846999645233154, -0.5343000292778015]
profile {'n': 10, 'dataset': '334Broa008objfps16k12', 'stages': {'episode_op': {'mean_ms': 0.002, 'max_ms': 0.003}, 'step_sampler': {'mean_ms': 0.141, 'max_ms': 0.231}, 'step_op': {'mean_ms': 0.065, 'max_ms': 0.094}}, 'added_ms': 0.208, 'throughput_estimate': {'before': 394.0, 'after': 393.7, 'basis': '394 x 325 / (325 + added_ms),估算,没实测'}}
  • step_op 的 mean_ms 0.065 是这个 op 每个样本的耗时。
  • 没给 sampler 时用内置 default_sampler@1,它的耗时算在 step_sampler 里。

上传:upload_norm.py

"""上传:带上 profile,记上传者。"""
import logging; logging.disable(logging.WARNING)
from eggy_matrix_gen.userop import load_dm, profile, upload
from norm_action_op import NormActionX2

prof = profile(load_dm("manifest_8ab8c94622d4d724"), dataset="334Broa008objfps16k12", n=10, step_ops=[NormActionX2()])
print(upload(NormActionX2, "norm_action_x2", profile=prof, uploader="e2e-norm-1009"))

输出:

norm_action_x2@1

实测时 uploader 写的是测试名 e2e-norm-1009。你上传时写自己的名字。

之后怎么用的

dc       dc_e7d79383ff5ec240   step_op norm_action_x2@1;norm_action[0..7] → action 槽
tm       tm_e82b2e56df7efc69   loader 读 32 个样本:action = 原 action × 2,失败 0

需求:co-train 里另外两个 LeRobot 数据集也各加一列,都是原值 × 2。agibot 改 qpos,galaxea 改末端位姿(tcp)。之后在各自的 dc 里把新列接到 container 的 qpos 槽。

两个数据集的列名不同,所以写两个 op。列名来自第 2 步 cols.py 的输出。

work/
  agibot_qpos_op.py      class AgibotQposX2(StepOp)   norm_qpos = [右臂 qpos 7, 右夹爪 1] × 2,宽 8
  galaxea_tcp_op.py      class GalaxeaTcpX2(StepOp)   norm_tcp = 右臂末端位姿 × 2,宽 9
  test_lerobot_ops.py    造数据单测(两个 op)
  try_lerobot_ops.py     真 dm 试跑 + 测耗时(两个数据集各 3 条)
  upload_lerobot_ops.py  上传(两个 op)

写 op:agibot_qpos_op.py

import numpy as np
from eggy_matrix_gen.userop import Step, StepOp

ARM, GRIP = "observation.state.qpos.right_arm", "observation.state.qpos.right_gripper"


class AgibotQposX2(StepOp):
    """agibot:加一列 norm_qpos = [右臂 qpos(7), 右夹爪(1)] × scale,宽 8。原来的列不变。"""

    def __init__(self, scale: float = 2.0):
        self.scale = scale

    def process(self, ep, t, ctx):
        missing = [k for k in (ARM, GRIP) if k not in ep.arrays]
        if missing:
            raise KeyError(f"episode 缺列 {missing}")
        q = np.concatenate([ep.arrays[ARM], ep.arrays[GRIP]], axis=1) * self.scale
        return Step(t=t, arrays={**ep.arrays, "norm_qpos": q.astype(np.float32)})

写 op:galaxea_tcp_op.py

import numpy as np
from eggy_matrix_gen.userop import Step, StepOp

TCP = "observation.state.pose.right_arm"


class GalaxeaTcpX2(StepOp):
    """galaxea:加一列 norm_tcp = 右臂末端位姿(tcp,位置 3 + 旋转 6)× scale,宽 9。原来的列不变。"""

    def __init__(self, scale: float = 2.0):
        self.scale = scale

    def process(self, ep, t, ctx):
        if TCP not in ep.arrays:
            raise KeyError(f"episode 缺列 {TCP!r}")
        return Step(t=t, arrays={**ep.arrays, "norm_tcp": (ep.arrays[TCP] * self.scale).astype(np.float32)})
  • 列名写成文件顶部的常量。测试脚本 import 这些常量,测试和 op 用同一个列名。
  • 拼接两列用 np.concatenate(..., axis=1),结果仍是 [n_frames, dim]。

造数据单测:test_lerobot_ops.py

"""单测:造 episode,不读真数据。"""
import numpy as np
from eggy_matrix_gen.userop import Ctx, Episode, run_local
from agibot_qpos_op import AgibotQposX2, ARM, GRIP
from galaxea_tcp_op import GalaxeaTcpX2, TCP

n = 12
arm, grip = np.random.rand(n, 7).astype(np.float32), np.random.rand(n, 1).astype(np.float32)
ep = Episode(n_frames=n, arrays={ARM: arm, GRIP: grip}, entity_id="fake_agibot")
q = run_local(ep, Ctx(0, 0, 0), step_ops=[AgibotQposX2()]).episode.arrays["norm_qpos"]
assert q.shape == (n, 8) and np.allclose(q, np.concatenate([arm, grip], 1) * 2)

pose = np.random.rand(n, 9).astype(np.float32)
ep = Episode(n_frames=n, arrays={TCP: pose}, entity_id="fake_galaxea")
c = run_local(ep, Ctx(0, 0, 0), step_ops=[GalaxeaTcpX2()]).episode.arrays["norm_tcp"]
assert c.shape == (n, 9) and np.allclose(c, pose * 2)

for op, arrays in ((AgibotQposX2(), {ARM: arm}), (GalaxeaTcpX2(), {"x": pose})):
    try:
        run_local(Episode(n_frames=n, arrays=arrays, entity_id="bad"), Ctx(0, 0, 0), step_ops=[op])
        raise SystemExit("缺列应该报错")
    except Exception as e:
        assert "缺列" in str(e), e
print("单测通过:norm_qpos 宽 8 = [arm, grip]×2;norm_tcp 宽 9 = pose×2;缺列报错")

输出:

单测通过:norm_qpos 宽 8 = [arm, grip]×2;norm_tcp 宽 9 = pose×2;缺列报错

真 dm 试跑:try_lerobot_ops.py

"""真 dm 自检:两个数据集各 3 条 episode,再测耗时。"""
import logging; logging.disable(logging.WARNING)
import numpy as np
from eggy_matrix_gen.userop import Ctx, load_dm, profile, run_local
from agibot_qpos_op import AgibotQposX2, ARM, GRIP
from galaxea_tcp_op import GalaxeaTcpX2, TCP

dm = load_dm("manifest_8ab8c94622d4d724")
cases = (("agibot_g1_swiftpicker", AgibotQposX2(), "norm_qpos", lambda e: np.concatenate([e.arrays[ARM], e.arrays[GRIP]], 1) * 2),
         ("galaxea_r1_lite", GalaxeaTcpX2(), "norm_tcp", lambda e: e.arrays[TCP] * 2))
for ds, op, col, want in cases:
    for i, ep in enumerate(dm.sample(3, dataset=ds, seed=0)):
        run = run_local(ep, Ctx(0, 0, i), step_ops=[op])
        got = run.episode.arrays[col]
        assert np.array_equal(got, want(ep).astype(np.float32)), (ds, ep.entity_id)
        print(f"{ds} {ep.entity_id[-12:]} n={ep.n_frames} {col}{got.shape} t={run.t} 等于原值×2")
    p = profile(dm, dataset=ds, n=10, step_ops=[op])
    print(ds, "step_op", p["stages"]["step_op"])

输出(10-09 实测):

agibot_g1_swiftpicker er_ep_127998 n=1616 norm_qpos(1616, 8) t=636 等于原值×2
agibot_g1_swiftpicker er_ep_045810 n=2116 norm_qpos(2116, 8) t=2048 等于原值×2
agibot_g1_swiftpicker er_ep_021493 n=3358 norm_qpos(3358, 8) t=2992 等于原值×2
agibot_g1_swiftpicker step_op {'mean_ms': 0.097, 'max_ms': 0.203}
galaxea_r1_lite te_ep_012576 n=929 norm_tcp(929, 9) t=366 等于原值×2
galaxea_r1_lite te_ep_013468 n=1102 norm_tcp(1102, 9) t=1066 等于原值×2
galaxea_r1_lite te_ep_011169 n=954 norm_tcp(954, 9) t=850 等于原值×2
galaxea_r1_lite step_op {'mean_ms': 0.054, 'max_ms': 0.066}

上传:upload_lerobot_ops.py

import logging; logging.disable(logging.WARNING)
from eggy_matrix_gen.userop import load_dm, profile, upload
from agibot_qpos_op import AgibotQposX2
from galaxea_tcp_op import GalaxeaTcpX2

dm = load_dm("manifest_8ab8c94622d4d724")
for cls, name, ds in ((AgibotQposX2, "agibot_qpos_x2", "agibot_g1_swiftpicker"), (GalaxeaTcpX2, "galaxea_tcp_x2", "galaxea_r1_lite")):
    prof = profile(dm, dataset=ds, n=10, step_ops=[cls()])
    print(upload(cls, name, profile=prof, uploader="e2e-norm-1009"))

输出:

agibot_qpos_x2@1
galaxea_tcp_x2@1

每个 op 用它自己的数据集测 profile。WebUI 选择弹层里显示的耗时:agibot_qpos_x2 0.226 ms,galaxea_tcp_x2 0.19 ms。

之后怎么用的

dc       dc_6f0556bbd01594fc   agibot:step_op agibot_qpos_x2@1;norm_qpos[0..7] → qpos 槽
dc       dc_838326142bddab6f   galaxea:step_op galaxea_tcp_x2@1;norm_tcp[0..7] → qpos 槽(宽 9,container 的 qpos 槽宽 8,只接前 8 维)
tm       tm_4b00b2c4cb2bbc82   co-train(334B + agibot + galaxea 各 1/3);loader:agibot 26/26、galaxea 22/22、334B 16/16

依据:check_cotrain_tm.py 在 PPU pod 上的输出,按 group_id 分数据集核对 state = 原值 × 2。

需求:三种 op 各写一个。给每帧标上帧号,只在帧号 % 7 == 3 的帧里选 t,再加一列 action_re = action + 1。训练读出的样本能直接核对 op 有没有生效:t0 % 7 == 3。

my_ops/
  mark_ep.py        class MarkEp(EpisodeOp)        加 frame_id 列
  mod_sampler.py    class ModSampler(StepSampler)  只在 frame_id % 7 == 3 的帧里选 t
  t_step.py         class TStep(StepOp)            加 e2e_t 列和 action_re 列
  test_ops.py       造数据单测
  try_ops.py        真 dm 试跑
  upload_ops.py     上传

写 op:mark_ep.py(episode_op)

"""episode_op:加 frame_id 列 = 帧号 0..n-1,形状 [n_frames, 1]。"""
import numpy as np
from eggy_matrix_gen.userop import EpisodeOp


class MarkEp(EpisodeOp):
    """加 frame_id 列 = 帧号 0..n-1。"""
    def process(self, ep, ctx):
        return ep.with_arrays(frame_id=np.arange(ep.n_frames, dtype=np.float32)[:, None])

写 op:mod_sampler.py(step_sampler)

"""step_sampler:只在 frame_id % mod == rem 的帧里选 t。要求 episode 已有 frame_id 列。"""
import numpy as np
from eggy_matrix_gen.userop import StepSampler


class ModSampler(StepSampler):
    """只在 frame_id % mod == rem 的帧里选 t。"""
    def __init__(self, mod: int = 7, rem: int = 3, col: str = "frame_id"):
        self.mod = mod
        self.rem = rem
        self.col = col

    def sample(self, ep, ctx):
        frame_id = ep.arrays[self.col][:, 0].astype(np.int64)
        frames = np.flatnonzero(frame_id % self.mod == self.rem)
        rng = np.random.default_rng((ctx.seed, ctx.epoch, ctx.index))
        return int(rng.choice(frames))

sampler 不 import MarkEp。它只读 frame_id 列。op 之间只通过 episode 的列传数据。

写 op:t_step.py(step_op)

"""step_op:加 e2e_t 列 = t(每帧相同),加 {src}_re 列 = src + offset。其他列原样带上。"""
import numpy as np
from eggy_matrix_gen.userop import Step, StepOp


class TStep(StepOp):
    """加 e2e_t = t 和 {src}_re = src + offset。"""
    def __init__(self, src: str = "action", offset: float = 1.0):
        self.src = src
        self.offset = offset

    def process(self, ep, t, ctx):
        a = ep.arrays[self.src]
        return Step(t=t, arrays={**ep.arrays,
                                 "e2e_t": np.full((ep.n_frames, 1), t, np.float32),
                                 f"{self.src}_re": (a + self.offset).astype(np.float32)})

造数据单测:test_ops.py

"""test_ops.py:每个 op 单独测,再三个串起来测。python3 test_ops.py"""
import numpy as np
from eggy_matrix_gen.container.episode import Episode
from eggy_matrix_gen.userop import Ctx, run_local

from mark_ep import MarkEp
from mod_sampler import ModSampler
from t_step import TStep

action = np.arange(40 * 8, dtype=np.float32).reshape(40, 8)
ep = Episode(n_frames=40, arrays={"action": action}, entity_id="fake")

# episode_op 单独跑
run = run_local(ep, Ctx(0, 0, 0), episode_ops=[MarkEp()])
assert (run.episode.arrays["frame_id"][:, 0] == np.arange(40)).all()

# 加上 sampler:200 个样本的 t 都满足 t % 7 == 3
ts = {run_local(ep, Ctx(0, 0, i), episode_ops=[MarkEp()], step_sampler=ModSampler()).t for i in range(200)}
assert ts and all(t % 7 == 3 for t in ts), ts

# 三个一起
run = run_local(ep, Ctx(0, 0, 5), episode_ops=[MarkEp()], step_sampler=ModSampler(), step_ops=[TStep()])
arrays = run.episode.arrays
assert run.t % 7 == 3
assert (arrays["e2e_t"] == run.t).all()
assert (arrays["action_re"] == action + 1).all() and (arrays["action"] == action).all()
print("单测通过:t 集合", sorted(ts))

输出:

单测通过:t 集合 [3, 10, 17, 24, 31, 38]
  • run_local 按 loader 的顺序跑你给的 op,返回 episode、t、image_t 和每段耗时 ms。
  • 只想测一个 op,也可以直接调方法,例如 MarkEp().process(ep, Ctx(0, 0, 0))。

真 dm 试跑:try_ops.py

"""try_ops.py:上传前检查 + 真 episode 跑三个 op + 测耗时。python3 try_ops.py"""
import logging; logging.disable(logging.WARNING)
import numpy as np
from eggy_matrix_gen.userop import Ctx, load_dm, profile, run_local
from eggy_matrix_gen.userop.library import check_source

from mark_ep import MarkEp
from mod_sampler import ModSampler
from t_step import TStep

# 上传时的检查:在新的 Python 进程里单独加载文件,读出种类和参数
for path, cls in (("mark_ep.py", "MarkEp"), ("mod_sampler.py", "ModSampler"), ("t_step.py", "TStep")):
    d = check_source(open(path).read(), cls)
    print("检查", path, d["kind"], {k: v["default"] for k, v in d["params"].items()})

DM, DATASET = "manifest_8ab8c94622d4d724", "334Broa008objfps16k12"
dm = load_dm(DM)
ops = dict(episode_ops=[MarkEp()], step_sampler=ModSampler(), step_ops=[TStep()])
for i, ep in enumerate(dm.sample(3, dataset=DATASET, seed=1)):
    run = run_local(ep, Ctx(0, 0, i), **ops)
    a = run.episode.arrays
    ok = run.t % 7 == 3 and np.array_equal(a["action_re"], ep.arrays["action"] + 1)
    print(ep.entity_id[:40], "n_frames", ep.n_frames, "t", run.t, "通过" if ok else "失败",
          "新列", sorted(set(a) - set(ep.arrays)))
print(profile(dm, dataset=DATASET, n=5, **ops))

输出(10-09 实测):

检查 mark_ep.py episode_op {}
检查 mod_sampler.py step_sampler {'mod': 7, 'rem': 3, 'col': 'frame_id'}
检查 t_step.py step_op {'src': 'action', 'offset': 1.0}
bed_69807.5TfHjDIuX6KA-1H5kSAVjA.EjusnxY n_frames 175 t 150 通过 新列 ['action_re', 'e2e_t', 'frame_id']
living_31373._sbaILRbXa6BilUrmPp32Q.aLtK n_frames 100 t 87 通过 新列 ['action_re', 'e2e_t', 'frame_id']
living_66848.kiacGmNDUDOehTULeKSGpw.z7FR n_frames 62 t 10 通过 新列 ['action_re', 'e2e_t', 'frame_id']
{'n': 5, ..., 'added_ms': 0.408, 'throughput_estimate': {'before': 394.0, 'after': 393.5, ...}}
  • check_source 做的是上传时的同一项检查,但不写 OSS。
  • added_ms 是三段 op 在每个样本上加的耗时。throughput_estimate 是估算,没有实测验证,只用来比较 op 的快慢。

上传:upload_ops.py

"""upload_ops.py:上传三个 op。同名再传得到下一个版本。"""
from eggy_matrix_gen.userop import upload

from mark_ep import MarkEp
from mod_sampler import ModSampler
from t_step import TStep

ME = "zhangsan"   # 换成你的名字
for cls, name in ((MarkEp, f"{ME}_mark_ep"), (ModSampler, f"{ME}_mod_sampler"), (TStep, f"{ME}_t_step")):
    print(upload(cls, name, uploader=ME))

输出是 名字@版本,例如:

zhangsan_mark_ep@1
zhangsan_mod_sampler@1
zhangsan_t_step@1

实测时名字前缀是 e2e,上传得到 e2e_mark_ep@1、e2e_mod_sampler@1、e2e_t_step@1。

之后怎么用的

dc       dc_e74e446bb47e6660   三个 op;action_re[0..7] → action 槽
tm       tm_e254c93945b4f282   loader 读 32 个样本:t0 % 7 == 3,action = 原 action + 1,失败 0

上传规则

完整文档:matrix 仓库 docs/training/USEROP.md。本页的代码和输出都来自 10-09 在 gigatron 镜像里的实测:例子 3 在 env -i 只留 OSS 凭据的环境里;例子 1、2 在 H200 pod 的 venv(/workspace-debug/user_sim2)里,装的是 42afe16 的 wheel。