BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/Chicago
X-LIC-LOCATION:America/Chicago
BEGIN:DAYLIGHT
TZOFFSETFROM:-0600
TZOFFSETTO:-0500
TZNAME:CDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0500
TZOFFSETTO:-0600
TZNAME:CST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20230124T171521Z
LOCATION:C1-2-3
DTSTART;TZID=America/Chicago:20221117T083000
DTEND;TZID=America/Chicago:20221117T170000
UID:submissions.supercomputing.org_SC22_sess226_spostg117@linklings.com
SUMMARY:Optimizing Communication in Parallel Deep Learning via Parameter P
 runing
DESCRIPTION:ACM Student Research Competition: Graduate Poster, ACM Student
  Research Competition: Undergraduate Poster, Posters\n\nOptimizing Communi
 cation in Parallel Deep Learning via Parameter Pruning\n\nSingh\n\nLarge s
 cale neural network training is challenging due to the high ratio of commu
 nication to computation. Recent work has shown that these large networks c
 ontain sparse subnetworks consisting of 10-20% of the parameters, which wh
 en trained in isolation reach comparable accuracy to the larger network. I
 n this work, we propose a novel approach that exploits the existence of th
 ese sparse subnetworks to dramatically improve the efficiency of large sca
 le neural network training. By storing in sparse and computing in dense, w
 e are able to reduce the number of parameters drastically while matching t
 he compute efficiency of the original network. We exploit this reduced par
 ameter set to optimize the communication time of AxoNN, a state-of-the-art
  framework for parallel deep learning. Our approach yields a significant s
 peedup of 17% when training a 2.7 billion parameter transformer model on 3
 84 GPUs.\n\nRegistration Category: Tech Program Reg Pass
END:VEVENT
END:VCALENDAR