[submission] lion with optimal hyperparameters (excl ogbg)

[tfaod]

New Submission

Submission Information

submission_name: "lion"
submission_folder: "submissions/self_tuning/lion"
submission authors:
  * authors: "Alice Yang"  # List authors separated by commas
  * affiliations: "Meta Superintelligence Labs, FAIR Team "
algorithm authors:
  * authors: "Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, Quoc V. Le"
  * affiliations: "Google, UCLA"
version: "1.0"
ruleset: "self-tuning"
framework: "PyTorch"
description: "lion optimizer with optimal hparams"

Evidence for the Submission's Performance

• See results from two Lion sweeps, compared to baseline sfadamw_v2 and nadamw submissions

Sweep Details

• Lion Sweep 1 across wd, lr
  • sweep range:
    • wd: [1e-2, 10.0]
    • lr: [1e-5 to 5e-1]
  • fixed values from lion paper
    • beta1: 0.9
    • beta2: 0.99
• Lion Sweep 2 across betas for top wd, lr values
  • sweep range:
    • beta1: [0.9, 0.999]
    • beta2: [0.9, 0.999]
  • fixed values:
    • wd: {0.2, 0.5, 1.0}
    • lr: [1e-4, 5e-4]
• Final Top Value:
  • all_lion_lr_wd_betas_sweep_lr0.0002-wd0.5-one_minus_beta1_0.05-beta2_0.98

...

Comments

• These sweep results do not include the ogbg workload. Those runs are in-progress.
• After ogbg runs finish, I will incorporate them into the leaderboard and update the optimal hparams.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[tfaod]

Test 1 is failing because I don't have permission to write to the submissions_algorithms repo and I cannot create a new branch