Generative Modelling

Generative model is defined as a stochastic function $g : Z \times Y \to X$ (contrary to a discriminative model (e.g. classifier) $f : X \to Y$ ), where $z \in Z$ is the latent variable, and $y \in Y$ is the label/description. Output of the generator $g$ can be varied using the randomised latent variable.

z x \sim p (Z) = g (z, y)

Latent variable z is a multi-dimensional vector that describes the features of the output not mentioned in label to the model. In an image generation model, it could be the setting of the environment, colour depth, pose, background.

Process:

Learner: Given output data for a discriminative: ${y^{(i)}}_{i = 1}^{N}$ , learner gets fed and outputs a generator function $g$ .
Generator: Sampling the generator with a randomised vector $z$ produces output ${\overset{x}{^}^{(i)}}_{i = 1}^{N}$ .

Unconditional vs Conditional generative models: Model may be conditioned on inputs or some other variables c of the form $p (X ∣ Y)$ .

Objective: How to measure the quality of the output by a generative model?

Output synthetic data that matches original data on certain marginal statistics. For example, it has the same mean colour as real photos or same colour variance.

How to find the statistics for other modalities like text/molecules? Number of words/sentences?
Output synthetic data that has high probability under a density model fit to the real data, i.e. $\overset{x}{^} \sim p_{D}$ where $p_{D}$ is the true process that produces original data.

Approaches: How to form data generators?

Direct approach: learn the generator function $G : Z \times X \to Y$ directly. GANs or Diffusion models work based on direct approach.
Indirect approach: learn a score function $E : X \to R$ and generate samples that scores highly under this function. Density models and Energy based models work under indirect approach.