2025-03-18

Paper Reading

---

OpenVLA

—

Related work

::: block VLM:

• bridge features from pretrained visual encoder(e.g. DINOv2, SigLIP) with pretrained LLM(e.g. Llama) Generalist Robot Policies:
• Octo: policy learning, compose pretrained component, learn to “stitch” them together.
• OpenVLA: end-to-end
  • more generalist
  • large Internet-scale dataset
  • generic architecture :::

—

VLM

::: block

• visual encoder: map image inputs to image patch embeddings
• projector: align image embeddings with word embeddings
• LLM backbone :::

—

OpenVLA

::: block

• concat SigLIP+DINOv2(helpful for improving spatial reasoning)
• projector: 2-layer MLP
• use Llama 2 as backbone
• map continuous action into discrete action token.
  • discretize each dimension of robot action separately into one of 256 bins.
  • each bin uniformly divided into $1^{st}$ to $99^{th}$ quantile
• Training Data: Open X-Embodiment dataset :::

---

RT-1

—

Preliminaries

::: block Robot learning:

• Aim to learn robot policy: $\pi (\cdot |i,x_i)$
• sample the action $a_t$ from learned distribution $\pi (\cdot |i,\{x_j{\}}_{j=0}^t)$
• target: maximize average reward(indicate complete or not) Transformer:
• sequence model
• map image&text to action sequence Imitation Learning:
• minimize the gap between ${\hat{a}}_t$ and $a_t^{\text{expert}}$
• refine $\pi$ by negative log-likelihood :::

—

System Overview

graph TB
a[Textural Instruction]-->|Universal Sentence Encoder|b[word embedding vector]
c[images]-->|ImageNet|d[features]
b-->|FiLM|e(affine transform)
d-->e
e-->|Tokenizer|f[Token]
f-->|Transformer|g[output Tokens]
g-->|Tokenizer Decode|h[action]

---

RT-2

—

RT-2

::: block Model:

• use CLIP to tokenize images and share embeddings with text
• use PaLI-x and PaLM-E as backbone of VLM
• decode output action token :::

—

::: block Co-Fine-tuning:

• combine datasets: to enhance more generalizing policies

Output Constraint:

• only sampling robot action when prompted with a robot-action task
• otherwise, answer natural language

chain of thought:

• an additional step: Plan Step. describes the purpose of the action that the robot is about to take in natural language first
• then followed by the actual action tokens. :::

---

RDT-1B

—

Related work

::: block DiT:

• combine diffusion and transformer VLA:
• Vision-Language-Action Model :::

—

Problem formulation

::: block ${\mathbf{o}}_t:=({\mathbf{X}}_{t-T_{img}+1:t+1},{\mathbf{z}}_t,c)$

• ${\mathbf{X}}_{t-T_{img}+1:t+1}=({\mathbf{X}}_{t-T_{img}+1},\cdots ,{\mathbf{X}}_t)$: RGB image history
• $z_t$: low-dimensional proprioception of robot
• $c$: control frequency
• ${\mathbf{a}}_t$: action, usually a subset of $z_{t+1}$ :::

—

Diffusion Model

::: block

1. ${\mathbf{a}}_t^k\sim \mathcal{N}(0,\mathbf{I})$
2. $${\mathbf{a}}_t^{k-1}=\frac{\sqrt{{\bar{\alpha }}^{k-1}}{\beta }^k}{1-{\bar{\alpha }}^k}{\mathbf{a}}_t^0+\frac{\sqrt{{\alpha }^k}(1-{\bar{\alpha }}^{k-1})}{1-{\bar{\alpha }}^k}{\mathbf{a}}_t^k+{\sigma }^k\mathbf{z}$$
   • ${\mathbf{a}}_t^0=f_{\theta }(l,{\mathbf{o}}^t,{\mathbf{a}}_t^k,k)$
   • $\mathcal{L}(\theta )=\text{MSE}({\mathbf{a}}_t,f_{\theta }(l,{\mathbf{o}}_t,\sqrt{{\bar{\alpha }}^k}{\mathbf{a}}_t^k+\sqrt{1-{\bar{\alpha }}^k}ϵ,k))$
   • ${\tilde{\mathbf{a}}}_t^k=\sqrt{{\bar{\alpha }}^k}{\mathbf{a}}_t^k+\sqrt{1-{\bar{\alpha }}^k}ϵ$
3. use action chunk to encourage time consistency and alleviate error accumulation over time :::

—

Encoding

::: block

• low-dimensional vectors represent physical quantities(proprioception, action chunk, control frequency)
  • use MLP with Fourier Features, capture the high-frequency changes
• image input: high-dimension
  • use image-text-aligned pretrained vision encoder: SigLIP
• language input:
  • pretrained T5-XXL :::

—

Network Structure

::: block

• QKNorm
• RMSNorm instead of LayerNorm
• MLP Decoder instead of linear decoder
• Alternative Condition Injection :::

—

Data

::: block ::: block Physically Interpretable Unified Action Space:

• $z_t$ and $a_t$
• unified space ::: :::

---

pi0

—

Related Work

::: block Flow Matching:

• denoise by conditional probability path $p_t(x|x_t)$
• loss: ${\mathcal{L}}_{CFM}(\theta )={\mathbb{E}}_{t,q(x_1),p_t(x|x_1)}\Vert v_t(x)-u_t(x|x_1){\Vert }_2^2$

Transfusion:

• train single transformer by multiple objectives
• loss: $\mathcal{L}={\mathcal{L}}_{LM}+\lambda {\mathcal{L}}_{DDPM}$ :::

—

${\pi }_0$ Model

::: block

• model data distribution: $p({\mathbf{A}}_t|{\mathbf{o}}_t)$
  • ${\mathbf{A}}_t=[{\mathbf{a}}_t,{\mathbf{a}}_{t+1},\cdots ,{\mathbf{a}}_{t+H-1}]$
  • ${\mathbf{o}}_t=[{\mathbf{I}}_t^1,\cdots ,{\mathbf{I}}_t^n,l_t,{\mathbf{q}}_t]$
• handle action by action expert, with CFM loss: $$L^{\tau }(\theta )={\mathbb{E}}_{p({\mathbf{A}}_t|{\mathbf{o}}_t),q({\mathbf{A}}_t^{\tau }|{\mathbf{A}}_t)}\Vert {\mathbf{v}}_{\theta }({\mathbf{A}}_t^{\tau },{\mathbf{o}}_t)-\mathbf{u}({\mathbf{A}}_t^{\tau }|{\mathbf{A}}_t){\Vert }_2$$

:::

—

Train Recipe

::: block

• first pretrain on big dataset
• then fine-tune with specific task :::