Skip to content

Commit 96d2a4f

Browse files
committed
revised vae, first draft flow, gan
1 parent bb66343 commit 96d2a4f

1 file changed

Lines changed: 4 additions & 4 deletions

File tree

‎vae/index.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -254,15 +254,15 @@ Amortized Variational Inference
254254
A noticable limitation of black-box variational inference is that **Step 1** executes an optimization subroutine that is computationally expensive. Recall that the goal of the **Step 1** is to find
255255
{% math %}
256256
\begin{align}
257-
\lambda^* = \argmax_{\lambda} \ELBO(\bx; \theta, \lambda).
257+
\lambda^* = \argmax_{\lambda\in \Lambda} \ELBO(\bx; \theta, \lambda).
258258
\end{align}
259259
{% endmath %}
260-
For a given choice of $$\theta$$, there is a well-defined mapping from $$\bx \mapsto \lambda^\ast$$. A key realization is that this mapping can be *learned*. In particular, one can train an encoding function (parameterized by $$\phi$$) $$f_\phi: \X \to \Lambda$$
260+
For a given choice of $$\theta$$, there is a well-defined mapping from $$\bx \mapsto \lambda^\ast$$. A key realization is that this mapping can be *learned*. In particular, one can train an encoding function (parameterized by $$\phi \in \Phi$$) $$f_\phi: \X \to \Lambda$$
261261
(where $$\Lambda$$ is the space of $$\lambda$$ parameters)
262262
on the following objective
263263
{% math %}
264264
\begin{align}
265-
\max_{\phi} \sum_{\bx \in \D} \ELBO(\bx; \theta, f_\phi(\bx)).
265+
\max_{\phi \in \Phi} \sum_{\bx \in \D} \ELBO(\bx; \theta, f_\phi(\bx)).
266266
\end{align}
267267
{% endmath %}
268268
It is worth noting at this point that $$f_\phi(\bx)$$ can be interpreted as defining the conditional distribution $$q_\phi(\bz \giv \bx)$$. With a slight abuse of notation, we define
@@ -274,7 +274,7 @@ It is worth noting at this point that $$f_\phi(\bx)$$ can be interpreted as defi
274274
and rewrite the optimization problem as
275275
{% math %}
276276
\begin{align}
277-
\max_{\phi} \sum_{\bx \in \D} \ELBO(\bx; \theta, \phi).
277+
\max_{\phi \in \Phi} \sum_{\bx \in \D} \ELBO(\bx; \theta, \phi).
278278
\end{align}
279279
{% endmath %}
280280
It is also worth noting that optimizing $$\phi$$ over the entire dataset as a *subroutine* everytime we sample a new mini-batch is clearly not reasonable. However, if we believe that $$f_\phi$$ is capable of quickly adapting to a close-enough approximation of $$\lambda^\ast$$ given the current choice of $$\theta$$, then we can interleave the optimization $$\phi$$ and $$\theta$$. The yields the following procedure, where for each mini-batch $$\M = \set{\bx^{(1)}, \ldots, \bx^{(m)}}$$, we perform the following two updates jointly

0 commit comments

Comments
 (0)