zhaoyingjun / Tiny-R2 Public

Notifications You must be signed in to change notification settings
Fork 0
Star 1

A better combination: DSA, mHC, and DSMoE similar to DeepSeek R2.

1 star 0 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 32 Commits
data		data
hyper_connections		hyper_connections
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
Tiny_R2_journey.ipynb		Tiny_R2_journey.ipynb
config.py		config.py
model.py		model.py
requerment		requerment
sampler.py		sampler.py
train.py		train.py
value_residual.py		value_residual.py

Repository files navigation

Tiny-R2

A better combination: DSA, mHC, and DSMoE similar to DeepSeek R2. https://wandb.ai/yingjun-xuda/Tiny-R2/reports/train-loss-26-01-20-21-52-23---VmlldzoxNTY4NzA2Nw

About

A better combination: DSA, mHC, and DSMoE similar to DeepSeek R2.

Report repository

Releases

No releases published

Packages

No packages published

Languages