Project: Advanced Transformers

[jinmang2]

목적

Efficient Transformer family를 fairseq + huggingface style로 개발한다.

참고할 Libraries

• https://github.com/huggingface/transformers
• https://github.com/pytorch/fairseq
• https://github.com/beir-cellar/beir
• https://github.com/UKPLab/sentence-transformers
• https://github.com/deepset-ai/haystack

TODO

• past key value
• efficient transformers
• dynamic padding
• gradient checkpoint
• various positional embedding
• reversible residual connection
• chunk feed forward network
• 3d, 4d multi-head scaled dot product attention
• various heads
• generation mixin
• parrallelism mixin
• pushtohub mixin
• how to make tokenization script?
• trainer

후보

• memorized transformer
• compressive transformer
• transformer-xl
• retrieval enhanced transformer
• LoRA
• p-tuning
• Bi-encoder, Cross-encoder, Poly-encoder, ColBERT

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB