[QUESTION] Pretraining dinov2_vitb14 on 200k Custom Images — Loss Curve Evaluation and Optimization Suggestions

[changcongxun]

🤔 What’s your question?

Hi, I pretrained dinov2_vitb14 on a custom dataset containing 200,000 RGB images.
Here are my training settings:
Total epochs: 3000
Batch size: 512
GPUs: 8 × NVIDIA A800-SXM4-80GB
Below is the loss curve recorded by TensorBoard.

Could you please help me analyze whether the training process and the loss behavior look correct? Are there any potential improvements or optimization suggestions for better convergence or performance?

Thanks!

📌 Related code or context (if any)

[stegmuel]

Hi,
I would say that it did look decent until ~250k steps, afterwards things get a bit suspicious.
The default number of training steps is 125k. Did you try with the default setting first?
In general we found that using a number of steps close to the default helps a lot with stability. The capacity of the training can also be modulated with a larger batch size or larger models.