2025-04-09

Coding

—

ROCO Bench

• LLM generate path points
• using RRT

—

Humanoid Bench

• env
  • two agent mujoco model
  • reward, observation space, action space
• train
  • tdmpc2
  • ppo

---

Paper Reading

---

CoT

—

basic CoT

• manually add prompt
• make LLM thinking step by step

—

zero-shot CoT

• add prompt: "Let's think step by step"
• pros: simple, zero-shot
• cons: bad performance

—

AutoCoT

• use BERT, cluster
• kmeans
• example guided

—

ToT

• Thought Decomposition
• Thought Generator
  • Sample
  • Propose
• State Evaluator
  • Value
  • Vote

—

GoT

::: block

1. find unreasonable choices
2. analyze why unreasonable with context
3. find the best choice ::: —

Chain of Draft

::: block Think step by step, but only keep a minimum draft for each thinking step, with 5 words at most. Return the answer at the end of the response after a separator . :::

---

A-MEM

graph TB
a[Environment]
b[LLM Agent]
c[Agentic]
d[Memory]

a-->|Interaction|b
b-->|Interaction|a
b-->|Write|c
c-->|Read|b
c-->d
d-->c
c-->d
d-->c

—

Method

::: block

• Zettelkasten Method
  • $m_i=\{c_i,t_i,K_i,G_i,X_i,e_i,L_i\}$
• Link Generation
  • $s_{i,j}=\frac{e_i\cdot e_j}{|e_i||e_j|}$
  • ${\mathcal{M}}_{\text{near}}^n=\{m_j|\text{rank}(s_{i,j})\le k,m_j\in \mathcal{M}\}$
• Memory Evolution
  • $m_j^{\ast }\leftarrow \text{LLM}(m_n\Vert {\mathcal{M}}_{\text{near}}^n\backslash m_j\Vert m_j\Vert P_{s_3})$
• Retrieve Relative Memory
  • $e_q=f_{\text{enc}}[q]$
  • ${\mathcal{M}}_{retrieved}=\{m_j|\text{rank}(s_{i,j})\le k,m_i\in \mathcal{M}\}$

:::

---

Preserving and combining knowledge in robotic lifelong reinforcement learning

—

—

DPMM

• ${\theta }_k|\lambda \sim \mathcal{H}(\lambda )$
• $\pi |\alpha \sim \text{GEM}(\alpha )$
• $v_i|\pi \sim \text{Cat}(\pi )$
• $x_i|v_i\sim F({\theta }_{v_i})$

—

Variational Inference

$\text{ELBO}(q)={\sum }_{k=1}^K[{\mathbb{E}}_q[{\theta }_k{]}^{\top }{s}_k(x)-{\hat{N}}_k[a({\theta }_k)]+{\hat{N}}_k[\log {\pi }_k(\beta )]$ $-{\sum }_{n=1}^N{\hat{r}}_{nk}+{\mathbb{E}}_q\left[\log \frac{B({\beta }_k|1,\alpha )}{q({\beta }_k|{\hat{\alpha }}_{k_1},{\hat{\alpha }}_{k_0})}\right]+{\mathbb{E}}_q\left[\log \frac{\mathcal{H}({\theta }_k|\lambda )}{q({\theta }_k|{\hat{\lambda }}_k)}\right]]$

—

Metrics

• average success rate
  • $P(t)=\frac{1}{N}{\sum }_{i=1}^N{P}_i(t)\text{: 每个任务的成功率的平均}$
• forgetting
  • $F_i=(P_i(\Delta i)-P_i(T))\text{: 训练后的成功率减去最终成功率}$
  • $F=\frac{1}{N-1}{\sum }_{i=1}^N{F}_i$
• forward transfer
  • $F{T}_i=\frac{1}{i-1}{\sum }_{k=1}^{i-1}{P}_k(\Delta k)\text{ : 训练前i个任务的平均成功率}$
  • $FT=\frac{1}{N-1}{\sum }_{i=2}^{N}F{T}_i$
• Improvement of few-shot knowledge recall
  • $f=\frac{1}{T\times P_{\text{max}}}\left({\int }_{t_j}^{T_j}P(t)dt-{\int }_{t_i}^{T_i}P(t)dt\right)$