BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/Denver
X-LIC-LOCATION:America/Denver
BEGIN:DAYLIGHT
TZOFFSETFROM:-0700
TZOFFSETTO:-0600
TZNAME:MDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0600
TZOFFSETTO:-0700
TZNAME:MST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20260422T000714Z
LOCATION:401-402
DTSTART;TZID=America/Denver:20231115T140000
DTEND;TZID=America/Denver:20231115T143000
UID:submissions.supercomputing.org_SC23_sess168_pap352@linklings.com
SUMMARY:Hanayo: Harnessing Wave-Like Pipeline Parallelism for Enhanced Lar
 ge Model Training Efficiency
DESCRIPTION:Ziming Liu, Shenggan Cheng, Haotian Zhou, and Yang You (Nation
 al University of Singapore)\n\nLarge-scale language models have become inc
 reasingly challenging and expensive to train. Among various methods addres
 sing this issue, Pipeline Parallelism has been widely employed to accommod
 ate massive model weights within limited GPU memory. This paper introduces
  Hanayo, a wave-like pipeline parallelism strategy that boasts a concise s
 tructure and practical applicability, alongside a high-performance pipelin
 e execution runtime to tackle the challenges of pipeline strategy implemen
 tation. Hanayo mitigates the issues of pipeline bubbles and excessive memo
 ry consumption prevalent in existing schemes, without resorting to model d
 uplicates as in Chimera. Our evaluation, conducted on four distinct comput
 ing clusters and involving both GPT-like and BERT-like architectures with 
 up to 32 GPUs, demonstrates up to a 30.4% increase in throughput compared 
 to the state-of-the-art approach.\n\nTag: Artificial Intelligence/Machine 
 Learning\n\nRegistration Category: Tech Program Reg Pass\n\nReproducibilit
 y Badges: Artifact Available, Artifact Functional, Results Reproduced\n\nS
 ession Chair: Shuaiwen Leon Song (Microsoft Corporation)\n\n
END:VEVENT
END:VCALENDAR