A Unified View of Long-Sequence Models towards Modeling Million-Scale Dependencies
Paper
• 2302.06218 • Published
• 1
Distributed
-
-
ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
Paper • 2306.10209 • Published • 2 -
SE-MoE: A Scalable and Efficient Mixture-of-Experts Distributed Training and Inference System
Paper • 2205.10034 • Published • 1 -
A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training
Paper • 2303.06318 • Published • 1 -
Ultra-Long Sequence Distributed Transformer
Paper • 2311.02382 • Published • 6 -
HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System
Paper • 2203.14685 • Published • 1 -
Pathways: Asynchronous Distributed Dataflow for ML
Paper • 2203.12533 • Published • 2 -
Distributed Pruning Towards Tiny Neural Networks in Federated Learning
Paper • 2212.01977 • Published • 1 -
Punica: Multi-Tenant LoRA Serving
Paper • 2310.18547 • Published • 2 -
Distributed Inference and Fine-tuning of Large Language Models Over The Internet
Paper • 2312.08361 • Published • 27 -
Training Transformers Together
Paper • 2207.03481 • Published • 6