Invited Talk
Speaker: Didem Unat, Koç University, Turkey
Title: A Dataflow-Graph Partitioning Method for Training Large Deep Learning Models
Abstract:
Large Deep Neural Networks (DNNs) models have substantial memory requirements to store the model parameters and intermediate results. As a result, the limited device memory becomes a bottleneck when training those models. We propose a deterministic, generic, and efficient partitioning strategy for DNNs that are represented as computational graphs. The proposed partitioning algorithm decides a placement of DNN’s underlying computational graph operations across multiple accelerators, so that the memory constraints of the devices are met and the training time is minimized. To the best of our knowledge, the strategy in this work is the first that has absolute independence of the structure and operation types in DNN models. Therefore, it guarantees future compatibility and can be used with any type of emerging model, even if it has zero resemblance to the existing ones in terms of structure or even the nature of the learning process and its operations. In this talk, I will be presenting the details of the method along with some performance data and comparison with related work.
