Code for Training MiniMoE

[ojus1]

Can you please release code for "upcycling" LLMs to make MoEs? I have a use-case for multi-lingual LLMs where this would be incredibly helpful!

[GeneZC]

We may update the code later after the intermediate checkpoints of our MoE model are verified to be effective.