Configuration for object detection

[foreverzzx]

My idea is to use only the pretrained weights of the DINOv3 backbone, without loading the pretrained weights of the entire model. Below is my configuration, but it seems that the DINOv3 pretrained weights are not taking effect. How should I configure this correctly?

    lightly_train.train_object_detection(
        overwrite=True,
        devices='auto',
        batch_size=batch_size,  # 增加批量大小以提高训练稳定性
        out=exp_path,  # 新的输出目录，避免覆盖之前的结果
        model="dinov3/vitt16-ltdetr-coco",#"weights/dinov3_convnext_base_ltdetr_coco_251113_a310a474.pt", # 指定用哪个模型把对应的路径下载放到weights下
        model_args={
            "backbone_weights": "./weights/dinov3_vitl16_pretrain_sat493m-eadcf0ff.pth"
        },
        checkpoint=pretrained_weight_path,
        data={
            "path": "./",
            "train": train_img_path,
            "val": val_img_path,
            "names": classes,
        },
        transform_args={
            "image_size": (768, 768), 
            "val": {    # Override validation parameters
                "image_size": (768, 768), # (height, width)
            }
        },
        steps=steps,  # Small number of steps for demonstration, default is 90_000.
    )

[guarin]

Hi, you can use the following code for this:

lightly_train.train_object_detection(
    model="dinov3/vitt16-ltdetr",
    ... 

This will initialize the detection model with a pretrained DINOv3 backbone and random weights for the head.

If you want custom weights for the backbone you can pass them with the backbone_weights argument:

lightly_train.train_object_detection(
    model="dinov3/vitl16-ltdetr",
    model_args={"backbone_weights": "./weights/dinov3_vitl16_pretrain_sat493m-eadcf0ff.pth"}
    ... 

But note that in the example you posted above you set the model to a vitt16 (tiny ViT) and the backbone_weights to a vitl16 (large ViT) which won't work. The model and the backbone require the same architecture.

As there are no vitt16 weights available pretrained on satellite images you have two options:

1. Train a larger object detector (see my example above)
2. Distill the dinov3_vitl16_pretrain_sat493m-eadcf0ff.pth weights into a smaller DINOv3 model and then use that as backbone weights

I would recommend to go with 1) if possible or use the normal DINOv3 weights (not satellite images). If you really need a tiny satellite image backbone you can distill it like this:

lightly_train.pretrain(
    out="out/pretrain",
    data="<path directory with unlabeled images>",
    model="dinov3/vitt16",
    method="distillationv1",
    method_args={
        "teacher": "dinov3/vitl16-sat493m",
    },
    epochs=1000,
    batch_size=1024,
    ... 

You might have to tune the learning rate or other paraemters a bit as we never tried distilling a large satellite model into a tiny ViT.

Once the distillation is complete you can pass the exported model path from the pretraining to the train_object_detection command:

lightly_train.train_object_detection(
    model="dinov3/vitt16-ltdetr",
    model_args={"backbone_weights": "out/pretrain/exported_models/exported_model.pt"},
    ...