Questions about soft-min-snr loss

[hmicrobe]

Really nice work! When reading through the paper, I have some questions about the proposed soft-min-snr loss. Would appreciate your feedback on this.

1. In eq (5) of the hourglass diffusion transformers, it's mentioned that c_out^{-2}(\sigma) is incorporated, however, based on the definition of c_out, eq (5) should be

min(SNR, \gamma) * (\sigma_data^2 + \sigma^2) / (\sigma_data^2 * \sigma^2).

2. In the implementation:
   

   k-diffusion/k_diffusion/layers.py

   Lines 64 to 65 in 6ab5146

   	def _weighting_soft_min_snr(self, sigma):
   	return (sigma * self.sigma_data) ** 2 / (sigma ** 2 + self.sigma_data ** 2) ** 2

The \gamma=4 or 5 proposed in the paper doesn't seem to be used. Am I missing anything here?

[stefan-baumann]

In this code, gamma is hardcoded to depend on sigma_data, with gamma being chosen as gamma = sigma_data^-2. This, combined with the preconditioner compensation, leads to the formula you're seeing.