Max Ryabinin

Max Ryabinin is VP of Research & Development, Model Shaping, at Together AI, where he works on efficient deep learning, long-context training infrastructure, and making large-scale AI systems more accessible. He is also known for research on decentralized deep learning and as an author of Hivemind, a PyTorch library for training neural networks over heterogeneous distributed hardware.

Untied Ulysses Pushes Llama-3-8B Training to 5 Million Tokens

Together AI’s Max Ryabinin argues that training transformers at multi-million-token context lengths is chiefly a memory-scheduling problem, not a matter of applying a single long-context technique. Using a Llama 3-8B run on an 8xH100 node as the example, he shows how fully sharded data parallelism, DeepSpeed Ulysses, activation checkpointing, CPU offloading and chunked sequence training each remove one bottleneck and expose the next. His proposed addition, Untied Ulysses, chunks attention heads and reuses context-parallelism buffers, with the presented results claiming scaling to 5 million tokens with limited throughput loss.

AI EngineerJun 8, 202611 min read