bug? only 3 of 4 of Adapter features are used

[amandaluof]

Hi! Thanks for the great work, but I guess there may be a bug. The length of adapter feature list is 4. But only the top3 are added to the unet(unet_2d_condition.py line582).

Best wishes!

[nyp0x]

Can confirm that using all 4 down blocks improves quality (first row: only CrossAttnDownBlock2D, second row: CrossAttnDownBlock2D and DownBlock2D)

However, it seems the use of adapter features in samplers is also necessary to further improve quality