About the gradient representation.

[panmianzhi]

I'm not very familiar with the gradient representation of a sample (Eq. 1). And I have 2 questions about it:
(1) what is the advantage of using gradient representation compared with the penultimate layer embedding? can I use that from a fixed, pre-trained encoder?
(2) If my task is regression rather than classification. Does the gradient representation still work well? Or I need to use another representation?
Thank you very much!

[JordanAsh]

Hi,

The gradient representation could be thought of as a rank-1 approximation of the Fisher, which is a common object to consider for decision making. I believe we mention this somewhere in the paper, but you could refer to this paper for a more thorough discussion. If your task is regression, then the Fisher (if we're just considering the last layer) corresponds to using the penultimate-layer representation. There's some information on that in the linked paper as well.

I don't see any issue with using a pretrained model in either case.

Best,
Jordan

[asaran]

Hi,

For your first question, I will add points already noted in our paper:

• Using the hallucinated gradient representation with respect to the layer of the network is akin to uncertainty+diversity sampling whereas using the penultimate layer embeddings is akin to diversity sampling.
• A pretrained encoder's gradient representation will also capture uncertainty information with respect to the data distribution the model was originally trained on.

Thanks,
Akanksha