Some problems to run evaluate.py

[nviolante25]

Hello,

Thank you for open-sourcing your project!

I'm trying to run the evaluation script using the pretrained models 2doc.pt and 3doc.pt from Huggingface) , which I placed under the folders configs/2d and configs/3d respectively.
To run the 2D model I'm doing

python evaluate.py -m configs/2d

I encountered some errors at this point, and here are the changes I made to fix them.
First, I changed the name of the .yml file in the folder configs from dtc_multilight_2d.yml to config.yml and adapted this line of code to read that config file instead of the required .py. With this change, I can load the model configuration.
Second, to load the 2doc.pt model I changed these lines into this:

model_ckpt = [step for step in os.listdir(model_path) if "pt" in step][0]
ckpt_path = os.path.join(model_path, model_ckpt)

So in the end ckpt_path=configs/2d/2doc.pt , and I can load the pretrained weights.

However, after doing this, the load_state_dict operation fails due to mismatch in the shapes of the parameters (see part of the error message below):

RuntimeError: Error(s) in loading state_dict for SwinT2D:
        size mismatch for conv_first.0.weight: copying a param with shape torch.Size([96, 7, 3, 3]) from checkpoint, the shape in current model is torch.Size([192, 7, 3, 3]).
        size mismatch for conv_first.0.bias: copying a param with shape torch.Size([96]) from checkpoint, the shape in current model is torch.Size([192]).
        size mismatch for conv_first.1.weight: copying a param with shape torch.Size([96, 96, 3, 3]) from checkpoint, the shape in current model is torch.Size([192, 192, 3, 3]).
        size mismatch for conv_first.1.bias: copying a param with shape torch.Size([96]) from checkpoint, the shape in current model is torch.Size([192]).
        size mismatch for conv_first.2.weight: copying a param with shape torch.Size([96, 96, 3, 3]) from checkpoint, the shape in current model is torch.Size([192, 192, 3, 3]).
        size mismatch for conv_first.2.bias: copying a param with shape torch.Size([96]) from checkpoint, the shape in current model is torch.Size([192]).
        size mismatch for conv_skip.weight: copying a param with shape torch.Size([96, 96, 3, 3]) from checkpoint, the shape in current model is torch.Size([192, 192, 3, 3]).
        size mismatch for conv_skip.bias: copying a param with shape torch.Size([96]) from checkpoint, the shape in current model is torch.Size([192]).
        size mismatch for conv_end.weight: copying a param with shape torch.Size([3, 96, 3, 3]) from checkpoint, the shape in current model is torch.Size([3, 192, 3, 3]).
        size mismatch for patch_embed.norm.weight: copying a param with shape torch.Size([96]) from checkpoint, the shape in current model is torch.Size([192]).
        size mismatch for patch_embed.norm.bias: copying a param with shape torch.Size([96]) from checkpoint, the shape in current model is torch.Size([192]).
        size mismatch for layers.0.residual_group.blocks.0.norm1.weight: copying a param with shape torch.Size([96]) from checkpoint, the shape in current model is torch.Size([192]).

[cskrren]

I update the config for 2D object compositing model.

[nviolante25]

Hello @cskrren ,
Thank you for your answer. With that fix I was able to run the 2D and the 3D models on the DTCMultiLight dataset.

My next question is what do I need to do to apply the 3D model to my own scenes. I have a couple of scenes with composited objects I'd like to run with MV-Colight, but I'm not sure about the procedure for that. Do I need to provide my own 3DGS results as a gs.npy for it and use proj_custom_scene.py? (This also seem to require a hilbert3d_order9.npy which I don't know where to download).