$f$-Policy

Paper

motivate: 通过与环境的交互来识别解决任务的优化行为. 能否通过一种方法为policy optimize提供更dense的shaping reward?

考虑最小化agent’s state visitation和goal distribution (假设每一个goal可以被represented成一个distribution. 最简单的是Dirac distribution), 以提供额外的learning signals.

Background

LEAP: 使用feasible vector确定sub-goal, 然后policy进行生成action. 但是本篇主要是针对one goal的情况, goal的选取不在本篇的考虑之内.

MDP: $⟨\mathcal{S},\mathcal{G},\mathcal{A},P,r,\gamma ,{\mu }_0,{\rho }_g⟩$. 其中, $\mathcal{S}$是状态空间, $\mathcal{A}$是动作空间, $\mathcal{G}\subset \mathcal{S}$是目标集合, $P:\mathcal{S}\times \mathcal{A}\mapsto \Delta (\mathcal{S})$是转移概率(其中, $\Delta (\cdot )$表示对集合的probability distribution), $\gamma \in [0,1)$是discount factor, ${\mu }_0$是初始状态分布, ${\rho }_g:\Delta (\mathcal{G})$是目标分布. 在一个episode开始时, 从分布${\mu }_0$和${\rho }_g$中采样初始状态$s_0$和目标$g$.

reward $r:\mathcal{S}\times \mathcal{A}\mapsto \mathbb{R}$基于agent的状态和goal的约束.

关注sparse的reward

Trajectory: 定义轨迹$\tau =(s_0,a_0,\cdots ,s_{T-1},a_{T-1},s_T)$. 累计reward $H_g(s):={\sum }_{t=0}^T[{\gamma }^{t}r(s_t+1,g)|s_0=s]$. 学习policy $\pi :\mathcal{S}\times \mathcal{G}\mapsto \Delta (\mathcal{A})$满足最大化预期reward ${\mathbb{E}}_{\pi ,s_0}[H_g(s_0)]$, 得到最优策略 ${\pi }^{\ast }={\mathrm{argmax}}_{{\pi }_{\theta }\in \Pi }{\mathbb{E}}_{\pi ,s_0}[H_g(s_0)]$

Distribution matching: 轨迹的概率定义为 $p_{\theta }(\tau ,g)=p(s_0){\prod }_{t=0}^{T}p(s_t|s_{t-1},a_{t-1}){\pi }_{\theta }(a_t|s_t,g)$. 定义状态$s$在轨迹$\tau$中被访问的次数${\eta }_{\tau }(s)$. 定义agent的goal-conditioned state visitation:

$$\begin{array}{rl}{p}_{\theta }(s,g)& =\frac{\int p_{\theta }(\tau ,g){\eta }_{\tau }(s)d\tau }{Z}\\ & =\frac{\int \prod p(s_{t+1}|s_t,a_t){\pi }_{\theta }(a_t|s_t,g){\eta }_{\tau }(s)d\tau }{\int \int \prod p(s_{t+1}|s_t,a_t){\pi }_{\theta }(a_t|s_t,g){\eta }_{\tau }(s)d\tau ds}\end{array}$$

定义f-divergences: $$D_f(P\Vert Q)={\int }_{P>0}P(x)f(\frac{Q(x)}{P(x)})dx-f^{\prime }(\infty )Q([P(x)=0])$$

f-divergence	$D_f(P\Vert Q)$	$f(u)$	$f^{\prime }(u)$	$f^{\prime }(\infty )$
FKL	$\int P(x)\log \frac{P(x)}{Q(x)}dx$	$u\log u$	$1+\log u$	Undefined
RKL	$\int Q(x)\log \frac{Q(x)}{P(x)}dx$	$-\log u$	$-\frac{1}{u}$	0
JS	$\frac{1}{2}\int P(x)\log \frac{2P(x)}{P(x)+Q(x)}+Q(x)\log \frac{2Q(x)}{P(x)+Q(x)}dx$	$u\log u-(1+u)\log \frac{1+u}{2}$	$\log \frac{2u}{1+u}$	$\log 2$
${\chi }^2$	$\frac{1}{2}\int Q(x)(\frac{P(x)}{Q(x)}-1{)}^{2}dx$	$\frac{1}{2}(1-u{)}^2$	$u$	Undefined

$f$-Policy Gradient

通过最小化goal和policy导向的最终state distribution的散度, 来学习一个policy:

$$J(\theta )=D_f(p_{\theta }(s)\Vert p_g(s))$$ $${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{\tau \sim p_{\theta }(\tau )}\left[[\sum _{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)][\sum _{t=1}^T{f}^{\prime }(\frac{p_{\theta }(s_t)}{p_g(s_t)})]\right]$$

有类似reinforce的问题, 上述梯度的计算完全依赖于on-policy updates. 使用类似proximal policy optimization的clipped objective:

$${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{s_t,a_t\sim p_{{\theta }^{\prime }}(s_t,a_t)}\left[\min (r_{\theta }(s_t)F_{{\theta }^{\prime }}(s_t),clip(r_{\theta }(s_t),1-ϵ,1+ϵ)F_{{\theta }^{\prime }}(s_t))\right]$$

其中, $r_{\theta }(t)=\frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{{\theta }^{\prime }}(a_t|s_t)}$, $F_{{\theta }^{\prime }}(s_t)={\sum }_{t^{\prime }=t}^T{\gamma }^{t^{\prime }}{f}^{\prime }(\frac{p_{{\theta }^{\prime }}(s_t)}{p_g(s_t)})$.

1. for $i=1\to num\_iter$:
   1. $B\leftarrow []$
   2. for $j=1\to num\_traj\_pre\_iter$:
      1. sample $g$, set $p_g(s)$
      2. 收集goal-conditioned trajectories $\tau :g$
      3. 使用$\tau$上的KDE进行Fit $p_{\theta }(s)$
      4. 对$\tau$里面的每一个state $s$, 存储$f^{\prime }\left(\frac{p_{\theta }(s)}{p_g(s)}\right)$
      5. $B\leftarrow B+\tau :g$
   3. for $j=1\to num\_policy\_updates$
      1. $\theta \leftarrow \theta -\alpha {\nabla }_{\theta }J(\theta )$