Question about long-time training

[leeruibin]

When I followed the training instructions to reproduce the results, I encountered some issues during replication. For example, as described in the paper, I fine-tuned the model for 3K steps, but the fine-tuned model’s performance was noticeably worse than that of the provided checkpoint.

This is the result from the reproduced 3K ckpt.

A.cinematic.scene.from.a.classic.western.movie.featuring.a.rugged.man.riding.a.powerful.horse.throu-0.7.mp4

The motion can become very slow.

And when I extended the training to around 15K steps, the model began to show strange camera movements and repetitive content.

A.cinematic.scene.from.a.classic.western.movie.featuring.a.rugged.man.riding.a.powerful.horse.throu-0.10.mp4

And this is the inference result from the provided ckpt.

A.cinematic.scene.from.a.classic.western.movie.featuring.a.rugged.man.riding.a.powerful.horse.throu-0.9.mp4

The motion seems to be normal.

Do you have any ideas about why this happen?

[mingzhe11]

How about your inference speed?

[Fangzhou-Ai]

The inference speed is the same as claimed by the author, but the quality of the video trained following the README is much more degregated compared with the ckpt offered by the author.

[bliblioscar]

I have the same question with you! The motion become very slow in my reproduction. @leeruibin

[SARIHUST]

Hi, I also ran into similar issues when reproducing the results.

In my case, the degradation appears even earlier — at 3K iterations, it already appears. The model begins to show noticeable camera drift / unintended camera movements, and the main subject sometimes disappears entirely. Then, in order to “follow” the prompt, the subject reappears suddenly and continues the motion, but the overall temporal consistency becomes quite strange.
This behavior looks very similar to what others reported at around 15K steps, just happening earlier for me.

Would love to know if anyone has insights into why this happens or if there are recommended hyperparameters/configs to stabilize the training.

A.close-up.3D.animated.scene.of.a.short.fluffy.monster.kneeling.beside.a.melting.red.candle.The.mo-0.mp4

A.close-up.shot.of.a.Victoria.crowned.pigeon.in.a.naturalistic.wildlife.photography.style.showcasin-0.mp4

A.stylish.woman.strolls.down.a.bustling.Tokyo.street.the.warm.glow.of.neon.lights.and.animated.city-0.mp4