Gradient Penalty

[HCA97]

Hi,

I have a question regarding gradient penalty. I believe the L2-norm calculation of the gradient is incorrect. It should be sum across all axes except the batch axis because the norm of a matrix should be a scalar value. However, your L2-norm calculation returns a matrix not a scalar value. Because you are summing across the channel axis only. Is there a special reason for that?

vagan-code/vagan/model_vagan.py

Line 134 in 6be0b51

slopes = tf.sqrt(tf.reduce_sum(tf.square(grads), axis=1))

Best Regards,
Cem

[rmukh]

@HCA97, I guess you are right. It should be smth like:

slopes = tf.sqrt(tf.reduce_sum(tf.square(grads), axis=[1, 2, 3]))
penalty = self.exp_config.scale * tf.reduce_mean(tf.square(slopes - 1.0))