In factor analysis, the latent variableprior is just the unit variance Gaussian
$$ h ∼ N(h; 0, I) $$
The role of the latent variables is to capture the dependencies between different observed variables.
To cast PCA in a probabilistic framework, we can make a slight modification to the factor analysis model
It is an approach to modeling linear factors that seeks to separate an observed signal into many underlying signals that are scaled and added together to formthe observed data. These signals are intended to be fully independent, rather than merely decorrelated from each other
The idea is that the important characteristics of scenes change very slowly compared to the individual measurements that make up a description of a scene.
In general, we can apply the slowness principle to any differentiable model trained with gradient descent. The slowness principle may be introduced by adding a term to the cost function of the form
$$ λ\sum_tL(f(x^{(t+1)}), f(x^{(t)})) $$
One example - SFA algorithm:
$$ \min_θE_t(f(x^{(t+1)})_i− f (x^{(t)})_i)^2 $$
SFA is typically used to learn nonlinear features by applying a nonlinear basis expansion to x before running SFA.
Use linear decoder plus noise to obtain reconstruction of x.
Sparse coding, like other linear factor models, often produces poor samples. This happens even when the model is able to reconstructthe data well and provide useful features for a classifier. The reason is that each individual feature may be learned well, but the factorial prior on the hidden code results in the model including random subsets of all the features in each generated sample. This motivates the development of deeper models that can impose a nonfactorial distribution on the deepest code layer, as well as the development ofmore sophisticated shallow models.